What is Gesture-Based Computing?


It is already common to interact with a new class of devices entirely by using natural gestures. The Microsoft Surface, the iPhone and iPod Touch, the Nintendo Wii, and other gesture-based systems accept input in the form of taps, swipes, and other ways of touching, hand and arm motions, or body movement. These are the first in a growing array of alternative input devices that allow computers to recognize and interpret natural physical gestures as a means of control. We are seeing a gradual shift towards interfaces that adapt to — or are built for — humans and human movements. Gesture-based computing allows users to engage in virtual activities with motion and movement similar to what they would use in the real world, manipulating content intuitively. The idea that natural, comfortable motions can be used to control computers is opening the way to a host of input devices that look and feel very different from the keyboard and mouse — and that enable our devices to infer meaning from the movements and gestures we make.

INSTRUCTIONS: Enter your responses to the questions below. This is most easily done by moving your cursor to the end of the last item and pressing RETURN to create a new bullet point. Please include URLs whenever you can (full URLs will automatically be turned into hyperlinks; please type them out rather than using the linking tools in the toolbar).

Please "sign" your contributions by marking with the code of 4 tildes (~) in a row so that we can follow up with you if we need additional information or leads to examples- this produces a signature when the page is updated, like this: - alan alan Jan 27, 2010

(1) How might this technology be relevant to the museums you know best?

  • There is no better way of learning that 'inscribing' knowledge on one's own body – and interacting with museum displays offer excellent opportunities for experiment and creativity. This is especially true of children who are still unpretentious enough to be able skip or dodge around a display without compromising their 'street cred' – even though there are plenty of adults who would love to be able to let their hair down a bit if they felt that there was no one around looking at them.
Other kinds of less antic-provoking gestures are these that are closer to the body – the iPhone and iPhone wanabees. Once you use an iPhone, iPad etc. it is very demeaning to go back to those mouse-driven or clunky keyboard navigated platforms. Clearly gesture navigation is here to stay and as long as it doesn’t actually endanger the art (or the public) it is an extremely undeveloped area for the museum with good potential for future development.

(2) What themes are missing from the above description that you think are important?

  • In addition to the platforms mentioned above: The Microsoft Surface, the iPhone and iPod Touch, the Nintendo Wii, there are also those companies who see their customer-base as the museum community. See for example: Ideum and http://www.artcom.de and many other companies around the world who are developing their wares with the museum visitor in focus.
Ideum (http://www.ideum.com) is an interactive design firm specializing in the development of multitouch-enabled computer exhibits, rich-internet applications, and social networking sites. Over the last decade, the firm has collaborated with top museums and cultural institutions across North America. Ideum has worked on over sixty interactive media projects covering a variety of topics - in the fields of art, history, music, science, and technology. Along with custom development work, Ideum sells multitouch tables and has developed GestureWorks, a multitouch software framework for Adobe Flash.
  • According to the website, self-promotion of ART+COM - The origins of ART+COM lie in a vision to research and develop interactive media solutions. From the very beginning the company has been committed to the future of the new media. Today we are among the leaders of the field, working on projects for industry, culture and the research sector. Actually – I agree with them – they have built some pretty impressive gesture-driven interactives.
  • - susan.hazan susan.hazan May 1, 2010
  • While we generally think of gesture as physical body motion of arms, hands, legs, fingers, or even whole body, an important, vital subset is eye tracking. Eye tracking can be used as a means of both interaction/triggering and evaluation using eye motion, duration of glance, pupil dilation and other factors. This has great potential for interaction with gallery interactives, web sites, and enabling the physically disabled to benefit from these opportunities. Among those at the forefront of research and design of eye tracking systems for museums is Slavko Milekic who recently presented his paper "Gaze-Tracking and Museums:Current Research and Implicaitions at the Museums and Web 2010 conference.- len.steinbach len.steinbach May 4, 2010
  • This piece in wired, which includes video and additional links describes the future of eye tracking and gesture based computing and the big companies (including MS) who are diving right in. Eye tracking tablets and the promise of Text 2.0: - len.steinbach len.steinbach May 4, 2010

(3) What do you see as the potential impact of this technology on education and interpretation in museums?

    • Museum visitors now come to the museum hoping (or incorrectly expecting) to be able to engage with the collection perhaps in ways they have become used to in their (online) social networks. Nina Simon deals with this in her Museum 2.0 blog where she explore the ways that the philosophies of Web 2.0 can be applied in museums to make them more engaging, community-based, vital elements of society http://museumtwo.blogspot.com/2006/12/what-is-museum-20.html.
Several years ago, Zehava Doering, presented a paper 'Strangers, Guests or Clients? Visitor Experiences in Museums' outlining the paradigm changes that have taken place in the relationship between the (historically embedded) museum voice of authority that is the 'keeper of knowledge' and the (more recent) pro-active visitor that expects to be able to do more that to passively consume an exhibition. Gesture navigation - as more compelling extensions of previous push-button interactivities - breach this gulf with simple and intuitive solutions for the visitor to engage in the gallery experience.
    • - susan.hazan susan.hazan May 1, 2010
    • Insofar as research is intrinsic to the development of educational and interpretive programs, eye- or gaze- tracking can help curators and educators understand what visitors are looking at when they look at objects, and better understand how to inform or direct their experience. Eye-tracking as part of interface design for otherwise standard web and gallery interactives will afford a new level of accessibility to the physically impaired, as well as to those who may just choose to use it. Eye tracking can trigger information resources such as adjacent screens, augmented reality devices, etc.- len.steinbach len.steinbach May 4, 2010
    • Museums are always challenged by the nature of the interfaces they use with any interactive --- not only by its technical and human usability, but also by its distraction from the content or object it is trying to control. Gesture based interfaces can mitigate the technical intrusion of educational and interpretive devices in the object experience, reduce the need for device supply and maintenance, and engage the body in a manner that makes for a more multi-sensory and therefore deeper learning experienceb- len.steinbach len.steinbach May 4, 2010

(4) Do you have or know of a project working in this area?

There are many, many successful examples I have seen in museums over the years – the list is extensive. I can only draw from my own practice.
The Israel Museum has extensive history of gesture-driven interactives in its educational galleries. See for example - Ariel Almos, Virtual Playground, 2003. Almos’s Virtual Playground invited visitors to don coloured hats which are individually ‘read’ and enable players, each in their turn, to hit a moving ball with their own coloured bat – much like the classical screen game Pong, only this time using their body (and hat) rather than their hands. The game is set against the background of a sandy beach where other visitors can casually relax against the background of a video seascape. http://musesphere.com/regeneration/ariel.jpg.
With Romy Achituv's Falling Rain, visitors are mesmerised by the falling text which settles like raindrops on their reflection on the screen, and as they reach out to catch a handful, or an armful of falling letters they try to make out what they are saying. Text Rain in fact gently reveals a poem by Evan Zimroth, Talk, You and as more letters are harvested, so words begin to come together and phrases of the poem appear. Visitors not only interact with the installation but also with one another as they endeavour to accumulate enough ‘rain’ to be able to read the text. Romy Achituv (www.gavaligai.com) is an Israeli artist currently living, working and teaching in Seoul http://musesphere.com/regeneration/rain.jpg.


- james.lin james.lin May 1, 2010Last year the National Palace Museum had a speicl exhibition on the Qing dynasty emperor Shizong (better known by his reign name, Yongzheng) http://www.npm.gov.tw/exh98/yongzheng/en01.htm. In this exhibition, we used Microsoft SurfaceComputing with gesture-based technology to share various interpretive material in the gallery. This year in our Future Museum project located at the Taoyuan international airport at Taiwan several interactive kiosk using gesture-based technologies is now showing. http://www.npm.gov.tw/exh99/npm_lohas/

The Cleveland Museum of Art had been developing a Chinese Scroll Project which used gesture to move through and explore a scroll. With major museum expansion and construction, I am not certain if it is on display yet. However, a working version has been demonstrated at Museums and the Web (2007?)- len.steinbach len.steinbach May 4, 2010




Please share information about related projects in our Horizon Project sharing form.