NOVEL INTERACTIVE TECHNOLOGY

Novel Interactive Technology



Novel Interactive Technology

Overview

Interactive Display is a novel interactive display technology developed by Microsoft Research. The outputs of two video cameras are combined behind a transparent projection display to produce an image of objects on the display surface. An otherwise normal sheet of acrylic plastic is transformed into a high-bandwidth input/output surface suitable for gesture-based interaction.

With two cameras the system can determine if a given object is on the display surface or above it. Interactive Display's image processing acts as a filter that only displays objects on the display surface, which allows for high-resolution touch images. For example, a high-resolution image of a paper document can be captured using a high-resolution still or CMOS video cameras. Interactive Display's capabilities are illustrated in this photograph of an interactive drawing program adding strokes derived from the touch image to a drawing image while using a cycling color map:

CopyCat is designed both as a platform to collect gesture data for our ASL recognition system and as a practical application which helps deaf children develop working memory and language skills while they play the game.

The system uses a video camera and wrist mounted accelerometers as the primary sensors. In CopyCat, the children use ASL to communicate to the heroine of the game, Iris the cat. For example, the child will sign to Iris, "ALLIGATOR ON CHAIR" (glossed from ASL). If the child signs poorly, Iris looks puzzled, and the child is encouraged to attempt the phrase again. If the child signs clearly, Iris "poofs" the villain and continues on her way. If the child cannot remember the correct phrase to direct Iris, she can click on a "help" button. The system then shows a short video with a signer demonstrating the correct ASL phrase. The child can then mimic the signer to communicate with Iris. This is similar to the help a mother provides a child when the child is not sure what to say in a sitiuation. Gesture-based interaction expands the possibilities for deaf educational technology by allowing signing children to interact with the computer using their gesture-based language. An initial goal of the system, suggested by our partners at the Atlanta Area School for the Deaf, is to elicit phrases which involve three and four signs from children who normally sign in phrases with one or two signs. This task encourages more complex sign construction and helps develop working memory for language.

The initial version of CopyCat had used a "Wizard of Oz" approach where an interpreter simulates the computer recognizer. This method allows research into the development of an appropriate game interface as well as data collection to train our hidden Markov model (HMM) based ASL recognition system. The current version utilizes computer reecognition to determine the corectness of the children's signing. The recognizer is accurate 85% of the time in correctly assesing the correctness of the children's signing. This level of accuracy did not cause the children any undue frustration and also alowed the children to ...