Now, I am obviously a bit partial on this topic, having worked in pyMT for quite a while and all. But I think doing so, we’ve made some valuable observations about what is involved when you program for multi-touch input. If you want to code multi-touch interactions or applications context becomes very important.
If you read the rest of the blog post, I’ll show you what I mean about context, and why e.g. Windows Touch makes life difficult if you want to program multi-touch. I’ll show you how to rewrite a windows touch example project (5 C# source files and > 400 lines of code) in Python using PyMT (1 source file with 12 lines of code). Yes 12 lines, you read correctly (and then there is the whole thing about it just running on Linux or OSX as well…but we’ll leave that for another blog post).
The Problem
Here is the problem: When you start programming with multiple simultaneous input cursors (e.g. multi-touch) things very quickly turn very funky and very different from programming for e.g. a single mouse. There is a lot of added complexity just in handling the state for multiple inputs without even beginning to talk about interpreting them in relation to each other. To make things easier, I think we need APIs and SDKs that provide a context in which multi-touch input makes sense. Instead, Windows (and other APIs) try to simplify things by giving you a more abstract view by default. That can make things simpler, but these approaches also limit the kinds of things you can do with multi-touch tremendously!
Windows 7 for example doesn’t give you the raw multi-touch information by default. Touch or stylus input generates mouse events, and your application can choose to handle some gesture events that windows has pre-defined (like ‘pan’ or ‘zoom’). Now I think that’s great if that’s all you want. Especially for legacy support (which being Windows is obviously a priority). It makes sense to do this, so that old applications have some way of working with the new input devices.
But let’s say you actually want to write an application that takes full advantage of multi-touch input. There are some videos over at channel9 that talk about the Windows Touch API in Windows 7, and what you have to do if you want to provide the full multi-touch experience. There is also a nice article about it here. But if you opt for true multi-touch, it litterally gives you rawinformation. Even if you are pretty familiar with the Windows API’s and know how to handle the messaging system and all that jazz, multi-touch programming isn’t going to be much fun with this API.
Let me show you what I’m talking about:
Windows Touch ScratchPad Example App
The Windows 7 SDK comes with am example application called MTScratchPad. You can read about it here or download it here to take a closer look, if you don’t have the SDK installed. The application draws a line for every touch on the display while the finger is down. Here is a screenshot (from the msdn documentation, I hope they don’t mind me using their picture)
Screenshot form teh ScratchPad Example in teh Windows SDK
If you take a look at the documentation or code, you will see that you have to do a lot of things as the developer in order to make the application do what you want. you have to overwrite the WndProc method of your window to catch WM_MESSAGES, you have to decode the touch structures for each message, define your event handlers, call them yourself based on what WM_TOUCH message you got, and keep track of all the strokes yourself in a collection. The example includes a class to represent a stroke and draws those lines (>170 lines of code). The main source file “WMTouchForm” has over 250 lines of code (that’s with comments and blank lines removed mind you), and of course visual studio generates a whole bunch of code and some other files. In PyMT I can make that entire application in 12 lines of code.
PyMT ScratchPad
Here is the PyMT version and a video explaining the code as well as showing a little bit more interesting example:
PyMT Video Tutorial 1 from Thomas Hansen on Vimeo.
from pymt import * class Tracer(MTWidget): def on_touch_down(self, touch): touch.userdata['line'] = list(touch.pos) def on_touch_move(self, touch): touch.userdata['line'].extend(touch.pos) def draw(self): for touch in getAvailableTouches(): drawLine(touch.userdata['line']) w = MTWindow() w.add_widget(Tracer()) runTouchApp()
Other SDK’s and multi-touch programming challenges
PyMT tries to give you a context in which multi-touch programming makes sense. This involves making it easy to handle events, and learn more about them, providing functionality to interpret and deal with those events and offering a collection of classes that encapsulate useful interactions and event processing. Now, if I’m honest, it’s unfair to compare it to the Windows Touch API, given that Windows Touch has to be part of and fit in with the overall Windows API. But I think the point I’m trying to make is that we need to rethink the way we program for multi-touch and other novel input devices (just like we changed programming paradigms when we went from text to gui interfaces). There is some other multi-touch SDK’s out there, that are probably more akin to what PyMT tries to do. Some are proprietary (e.g. Snowflake), and others open source (e.g. MultiTouchVista). I haven’t used or explored any of them, but I would welcome any feedback in how they approach some of the things I’m talking about here.
I have to say a little bit about the Surface SDK. It looks much better than the other multi-touch API’s I’ve gotten to look at. But it works only for the $15,000 surface, and its proprietary, so I can’t really use or test it. It does seem to have a great collection of surface/multi-touch controls. I haven’t had a chance to check out how it stacks up if you want to experiment with your own interaction ideas, or more radical interface ideas. (see e.g. http://channel9.msdn.com/pdc2008/PC17/ ). From the video it seems very much like they are providing ready made controls for developers in an attempt to unify the end user experience a lot. PyMT has a whole bunch of controls(widgets) too, but they are aimed more at giving the developer something to work with, rather than unifying the experience.
I do think unifying the interaction is important at some point, but I think it’s a little early for that. I wish we would focus a little bit more on providing developers with tools and API’s to let them be as creative as possible, instead of giving them tools to build the same kind of applications. Let’s see what we can come up with first, before deciding what the controls need to be and how they work. I don’t think we are ready for unifying either the SDK/API approach nor the interactions/controls for multi-touch or surface computing. The interaction paradigm is so revolutionary, I think we need to adopt our development tools more to it and explore the interaction space. Instead I think people are jumping the gun on trying to standardize the interface while using the development paradigms we used for the GUI.
. check back soon, im working on some updates to the website and stuff like that.
Python Meta Classes ROCK!!
I’ve been trying to do this for a long time, but could never find a way. Today I actually spent some time reading up on how metclasses work, and bam!
The goal is to have a base class, which has a method called “init_done()”. I want this method to be called after the actual object is initialized. That doesnt sound to bad, but try doing it and still have it be called after all calls to __init__ are done once you start subclassing (you only want it called once at the end, not after every parent calss’ call to __init__)!
So here it goes (I’m also adding a nifty print statement that one can exchang for e.g. autoregister all your subclasses in a factory or something like that ).
class MyType(type): def __new__(cls, name, bases, attrs): init = attrs['__init__'] def call_me_after_init(self,*args,**kwargs): init(self,*args,**kwargs) if type(self) == eval(name): print "AFTER init!!", #or attrs['init'] attrs['__init__'] = call_me_after_init return super(MyType, cls).__new__(cls, name, bases, attrs) def __init__(self, name, bases, attrs): super(MyType, self).__init__(name, bases, attrs) print "Register me in factory! name:", name, "self:", self class A(object): __metaclass__ = MyType def __init__(self, a): print "init A", a class B(A): def __init__(self, a): print "init B start", a super(B, self).__init__(a) print "init B end", a test = B("test")This is the output:
Register me in factory! name: A self:
Register me in factory! name: B self:
init B start test
init A test
init B end test
AFTER init?!
AWESOME!!