The goal is to have a base class, which has a method called “init_done()”.  I want this method to be called after the actual object is initialized.  That doesnt sound to bad, but try doing it and still have it be called after all calls to __init__ are done once you start subclassing (you only want it called once at the end, not after every parent calss’ call to __init__)!

So here it goes (I’m also adding a nifty print statement that one can exchang for e.g.  autoregister all your subclasses in a factory or something like that ).

class MyType(type):
    def __new__(cls, name, bases, attrs):
        init = attrs['__init__']
        def call_me_after_init(self,*args,**kwargs):
            init(self,*args,**kwargs)
            if type(self) == eval(name):
                print "AFTER init!!",   #or attrs['init']
        attrs['__init__'] = call_me_after_init
        return super(MyType, cls).__new__(cls, name, bases, attrs)

    def __init__(self, name, bases, attrs):
        super(MyType, self).__init__(name, bases, attrs)
        print "Register me in factory!  name:", name, "self:", self

class A(object):
    __metaclass__ = MyType
    def __init__(self, a):
        print "init A", a

class B(A):
    def __init__(self, a):
        print "init B start", a
        super(B, self).__init__(a)
        print "init B end", a

test = B("test")

This is the output:

Register me in factory! name: A self:
Register me in factory! name: B self:
init B start test
init A test
init B end test
AFTER init?!


AWESOME!!

If you read the rest of the blog post, I’ll show you what I mean about context, and why e.g.  Windows Touch makes life difficult if you want to program multi-touch.  I’ll show you how to rewrite a windows touch example project (5 C# source files and > 400 lines of code) in Python using PyMT (1 source file with 12 lines of code).  Yes 12 lines, you read correctly (and then there is the whole thing about it just running on Linux or OSX as well…but we’ll leave that for another blog post).

The Problem

Here is the problem:  When you start programming with multiple simultaneous input cursors (e.g. multi-touch) things very quickly turn very funky and very different from programming for e.g. a single mouse.  There is a lot of added complexity just in handling the state for multiple inputs without even beginning to talk about interpreting them in relation to each other.  To make things easier, I think we need APIs and SDKs that provide a context in which multi-touch input makes sense.  Instead, Windows (and other APIs) try to simplify things by giving you a more abstract view by default.  That can make things simpler, but these approaches also limit the kinds of things you can do with multi-touch tremendously! 

Windows 7 for example doesn’t give you the raw multi-touch information by default.  Touch or stylus input generates mouse events, and your application can choose to handle some gesture events that windows has pre-defined (like ‘pan’ or ‘zoom’).  Now I think that’s great if that’s all you want.  Especially for legacy support (which being Windows is obviously a priority).  It makes sense to do this, so that old applications have some way of working with the new input devices. 

But let’s say you actually want to write an application that takes full advantage of multi-touch input. There are some videos over at channel9 that talk about the Windows Touch API in Windows 7, and what you have to do if you want to provide the full multi-touch experience.  There is also a nice article about it here.  But if you opt for true multi-touch, it litterally gives you rawinformation.  Even if you are pretty familiar with the Windows API’s and know how to  handle the messaging system and all that jazz, multi-touch programming isn’t going to be much fun with this API.

Let me show you what I’m talking about:

Windows Touch ScratchPad Example App

The Windows 7 SDK comes with am example application called MTScratchPad.  You can read about it here or download it here to take a closer look, if you don’t have the SDK installed.  The application draws a line for every touch on the display while the finger is down.  Here is a screenshot (from the msdn documentation, I hope they don’t mind me using their picture)

Screenshot form teh ScratchPad Example in teh Windows SDK

If you take a look at the documentation or code, you will see that you have to do a lot of things as the developer in order to make the application do what you want.  you have to overwrite the WndProc method of your window to catch WM_MESSAGES, you have to decode the touch structures for each message, define your event handlers, call them yourself based on what WM_TOUCH message you got, and keep track of all the strokes yourself in a collection.  The example includes a class to represent a stroke and draws those lines (>170 lines of code).  The main source file “WMTouchForm” has over 250 lines of code (that’s with comments and blank lines removed mind you), and of course visual studio generates a whole bunch of code and some other files.  In PyMT I can make that entire application in 12 lines of code.

PyMT ScratchPad

Here is the PyMT version and a video explaining the code as well as showing a little bit more interesting example:

PyMT Video Tutorial 1 from Thomas Hansen on Vimeo.

from pymt import *
class Tracer(MTWidget):
   
    def on_touch_down(self, touch):
        touch.userdata['line'] = list(touch.pos)
       
    def on_touch_move(self, touch):
        touch.userdata['line'].extend(touch.pos)
           
    def draw(self):
        for touch in getAvailableTouches():
            drawLine(touch.userdata['line'])
   
w = MTWindow()
w.add_widget(Tracer())
runTouchApp()

Other SDK’s and multi-touch programming challenges

PyMT tries to give you a context in which multi-touch programming makes sense.  This involves making it easy to handle events, and learn more about them, providing functionality to interpret and deal with those events and offering a collection of classes that encapsulate useful interactions and event processing.  Now, if I’m honest, it’s unfair to compare it to the Windows Touch API, given that Windows Touch has to be part of and fit in with the overall Windows API.  But I think the point I’m trying to make is that we need to rethink the way we program for multi-touch and other novel input devices (just like we changed programming paradigms when we went from text to gui interfaces).  There is some other multi-touch SDK’s out there, that are probably more akin to what PyMT tries to do.  Some are proprietary (e.g. Snowflake), and others open source (e.g. MultiTouchVista).  I haven’t used or explored any of them, but I would welcome any feedback in how they approach some of the things I’m talking about here.

I have to say a little bit about the Surface SDK.  It looks much better than the other multi-touch API’s I’ve gotten to look at. But it works only for the $15,000 surface, and its proprietary, so I can’t really use or test it.  It does seem to have a great collection of surface/multi-touch controls.  I haven’t had a chance to check out how it stacks up if you want to experiment with your own interaction ideas, or more radical interface ideas.  (see e.g. http://channel9.msdn.com/pdc2008/PC17/ ).  From the video it seems very much like they are providing ready made controls for developers in an attempt to unify the end user experience a lot. PyMT has a whole bunch of controls(widgets) too, but they are aimed more at giving the developer something to work with, rather than unifying the experience.

I do think unifying the interaction is important at some point, but I think it’s a little early for that.  I wish we would focus a little bit more on providing developers with tools and API’s to let them be as creative as possible, instead of giving them tools to build the same kind of applications.  Let’s see what we can come up with first, before deciding what the controls need to be and how they work.  I don’t think we are ready for unifying either the SDK/API approach nor the interactions/controls for multi-touch or surface computing.  The interaction paradigm is so revolutionary, I think we need to adopt our development tools more to it and explore the interaction space.  Instead I think people are jumping the gun on trying to standardize the interface while using the development paradigms we used for the GUI.

I was invited about a month ago to attend IDC by “the nuigroup team”. For those that aren’t familiar with it, nuigroup is an online community of people interested in “natural” user interfaces. Last summer, I made some money participating as a student in google summer of code under the nuigroup organization. This year, I am lucky to share a mentoring position along side Mathew Virbel for a very talented student, Sharath Patali on his GSOC project related to pymt.  The groups feedback and sharing of ideas has been an invaluable resource for the software and studies I am working on at school.  Members of the group share knowledge, references, thoughts, ideas, and open source code for building and developing novel user interface devices and software.  At least thats how I had interpreted it so far.

I’m not sure I have one good word for conveying the entirety of my experiences from the Interactive Display Conference (IDC) in San Jose,CA this last week. It was certainly an interesting event.  I got to talk (albeit briefly) to both Andy Wilson and Jeff Han. Both of whom produce very inspiring and world class UI/HCI research on a consistent basis.  Further, it was a great honor to meet some of the other members of the nuigroup community in person, some of whom where inspiring not only because of their amazing intellect and artistic talents, but especially due to their friendliness, benevolence, and maturity.

Some concerning developments..

However, I am somewhat troubled by various apparent misconceptions and misunderstandings concerning the nuigroup community (a related blog post).  Many industry attendees where confused or even upset about the difference/associations between the nuigroup community and Natual User Interface A.B. of Sweden.   As if this matter wasn’t already confusing enough, other “nuigroup commercialization” efforts have apparently been underway by some of the individuals running the nuigroup.com forums and website.

Dont get me wrong.  I have no problem whatsoever with commercial ventures or proprietary software and business models.  I do however have a problem with what I think has been gross misconduct from some of the business entities involved with or arising out of the nuigroup.

Even more troubling to me personally, is the extent to which some of these issues have been seemingly intentionally kept under the table by individuals that consider themselves important members of the community.  An open community cannot function without transparent and honest discussions concerning its identity and purpose.

nuigroup needs to talk!

Of course, I can only speak for myself as a single member participating in the nuigroup community.  That however is the nature of communities, whether they be online research or hobbyist communities, academic communities, a small group of friends or entire countries. Here is a claim I make confidently without the need for a reference or argument: Organizing, maintaining, and governing communities is  a very hard problem.  And while many of us desire especially open, free, and democratic communities, these aspect make the organization and governance anything but easier.

I feel very strongly  that the community is entitled to an open and transparent discourse about these issues and its identity.  Whether the impact of the community is significant in industry or academia or not, the 5000 individuals contributing their thoughts and ideas have the right to know with whom they are sharing and under what conditions they are doing so.

I have been under the impression that many other community members have understood nuigroup in very similar terms as myself (at least most of the other members at IDC seemed to think so as well). Maybe that was ignorant and based on bad assumptions.  Either way, lets talk openly about what the nuigroup has been, is, and should be.  Here is what I think:

What I think:

I think I am way to busy with school work and trying to think about user interfaces to be able to worry about these issues.

I think teh community deserves an open dialog about nuigroup’s purpose, mission statement, and organization and that it is past due.  Maybe we have been to gullible or busy solving the technical problems we have been collaborating on. Maybe these issues have arisen only after the community has grown to a significant size.  Maybe ulterior motives have tainted the direction and caused the lack of appropriate steps along the way.  Maybe I am the only one who has been going along perceiving a very different community.  Maybe I am getting concerned about insignificant things.

To my understanding, the community itself has no direct affiliation with Natural User Interface A.B. of Sweden (NUI).  NUI was founded by early members of the nuigroup community and uses the same logo.  While NUI is free to contribute to the community like anyone else, the community cannot be responsible for the actions taken by NUI or endorse the company in any way shape or form.  The logo thing causes way to much confusion.  While I like the logo (and to my understanding nuigroup.com used it first), i could care less.  There is more important things in life to worry about than a freaking logo or name (unless your only objective is to make money through branding). Let’s get back to talking about novel UI’s.

The same things apply to NUIINC.  This is another  commercial entity founded in the US, by the individuals running the nuigroup.com webserver.  Apparently they are in the business of “creating communities and user interfaces”. It is my understanding that the community at large has not even been aware of the existence of this incorporation. At least I was’nt until last week.

Again, I think it is perfectly fine for a commercial enterprise to take interest in, support, or collaborate as part of the nuigroup community.  But you cannot claim to own, represent, or be anymore affiliated with the community than its other members.  Especially not if those members are lead to belive to be part of a truly open initiative.

It’s probably time to agree on some rules of conduct.  There are many ways to go here, but lets make sure everyone knows what is going on.  There is plenty of good research and examples of other online communities from which nuigroup can learn about how to organize itself (e.g Sugar Labs just went though some discussion last year.   John Hull pointed out this book:  ).  To what level it needs to establish rules for making collective decisions (e.g. hosting costs, gsoc, and other resources) needs to be established by the community as a whole.

At IDC, there was some talk about a foundation being formed to act as a legal entity and protect the communities collective resources.  The NUINC people apparently have already established such a foundation, much of this information was not layed out very clearly in my opinion though.  If a foundation or other legal entity needs to be formed, this needs to happen from within the community and based one some consensus.

Why the nuigroup community is a good thing

Despite these negative issues and circumstances, I do believe that the community has a very valid reason and purpose to exist.

For one I think that making the technology and knowledge we discuss more accessible is a good thing.  We do need to be more careful about respecting intellectual property and copyright!   Obviously no one should start a business using a patented technology after learning about it in the community without proper licensing.  We also cannot make available copies of copyrighted material.  We are free to discuss the knowledge, information and implications described in patents and copyrighted material.  If one disagrees with the laws governing these concepts, the appropriate venue or action to be taken has to be political or legal in nature.

What we can do is far more powerfull however, making the information and technology more accessible and easier to understand to people outside industry or academic communities only benefits us all.  Patent documents and conference papers can be hard to find or be very expensive.  No one can keep us from sharing the knowledge we gain from reading them. Neglecting the ideas of newcomers or those that aren’t privileged with direct acces to these materials would not only be arrogant, it would be counterproductive.  Especially in a field where so much of what we do is tainted by prior experiences and expectations about computer interfaces and their past limitations.

The assimilation and filtering of information and knowledge from various backgrounds is invaluable to members of all participating communities.  One nice aspect of IDC was the variety of backgrounds reflected in the speakers and attendees.  Yet, it came nowhere close to the unique and diverse backgrounds and demographics represented inside the nuigroup community.

A truly open online community about user interfaces and its collective intelligence can be a breeding ground for new ideas .  It can provide real time feedback and collaboration for our projects and work.  It spreads excitement about pushing user interfaces beyond our expectations.

An open and widely distributed community can provide valuable resources and novel evaluation methods for UI and HCI research.  Collaborative evaluation of user interfaces could provide many benefits besides just scalability and decentralization (we’re not all that good at recreating or confirming the user studies and experiemnts of our peers in the HCI field).

What do you think?

check it out:  pymt.googlecode.com

I have been working on integrating the gpu accelerated tracking into tbeta–a new tracker that is being developed by Seth Sandler.  I’ve managed to add all the gpu based pre processing to the current OpenCV based filter pipeline and added some new glsl filters along the way.  For now I am just reading back the result from the preprocessing using glReadPixels (which is really slow) and feeding it to the OpenCV contourFilter and tracker.  Still I seem to be getting a speedup from 170fps to 270fps (although I am not sure if those numbers are accurate).

I’ve got some bugs to fix when using video instead of the camera, and also still need to hook up the sliders to the gpu filters.  The parameters are still in the xml format I had them before.  And of course I still need to fix the rest of the pipeline to avoid the costly readback.

I have been playing around with a couple ideas of how to cope with this.  Some of which use alternative algorithm to label the individual components  Here is a screenshot of a really naive approach that actually works better than i thought it would.  It marks the horizontal and vertical middle pixel red and green respectively for each row/column.  The center point becomes yellow.  The idea is that if i have an image with just single pixels representing the center of each blob I can find them quickly on the gpu by processing e.g. each 10×10 block by a separate geometry shader thread.  Then I could easily produce variable length output to store the coordinates and only transfer back to the cpu what is really needed.

I’m pretty convinced that this approach is too naiive to work well, but it kind of looked cool.  We’ll see what else we can come up with.

I’ll add some documentation in the days to come. I need to also make the user interface for filter chain configuration better and get the blob detection and tracking working.

I have started coding and am working on adding camera support for a couple of different camera types. USB and ieee1394 work great on windows through DSLib and the CMU driver. I have gotten firewire support to work on OSX and Linux as well through libdc1394 but I still have quite some work to do with the cameras. I will see how far I can get this weekend. I am getting eager to write some glsl code for the image processing.

I will upload some code to svn after playing a bit more with the various camera interfaces so that people can actually use it with the hardware they have.