Simon Says (2D gesture recognition)

With Pengyu Hong, Beckman Institute, University of Illinois at Urbana-Champaign

(Coming soon) A paper on this work has been accepted for the IEEE International Conference on Face and Gesture Recognition, Grenoble, France, 2000.

This work uses a state based approach to gesture learning and recognition. Using spatial clustering and temporal alignment, each gesture is defined to be an ordered sequence of states in spatial-temporal space. The 2D image positions of the centers of the head and both hands of the user are used as features; these are located by a color based tracking method. From training data of a given gesture, we first learn the spatial information without doing data segmentation and alignment, and then group the data into segments that are automatically associated with information for temporal alignment. The temporal information is further integrated to build a Finite State Machine (FSM) recognizer. Each gesture has a FSM corresponding to it. The computational efficiency of the FSM recognizers allows us to achieve real-time on-line performance. We apply the proposed technique to build an experimental system that plays a game of “Simon Says” with the user.