Chun Zhu

Background

In recent years we have seen a revitalized interest in robots. As a matter of fact, some robots have come into our lives  already. An important problem that needs to be addressed is - how should we human interact with robots?

Human-robot interaction (HRI) is a very important issue in the design of assistive robotics, especially for elderly people, who usually suffer from problems with speech, or have difficulty in learning new computer skills. It is desirable to make the robot able to not only understand explicit human intentions from gestures, but also recognize the human daily activities, from which implicit human intentions may be inferred. Such a robot capability is called considerate intelligence.
 

Goal

We are developing a smart assisted living (SAIL) system to provide support to elderly people in their houses or apartments. The companion robot can understand explicit human intentions through hand gestures and detect implicit human intentions by recognizing human daily activities. Moreover, it is necessary to detect anomalies in daily activities and living patterns to alert the elderly and even provide help when he/she is helpless or in unconscious.

Achievement

We focus on daily human activity recognition in the SAIL system using wearable inertial sensors. We have developed a new motion sensor node using a commercial VN-100, as shown in Figure 2 (a, b, c). The node sends data through Zigbee to a receiver on the PC for processing. ZigBee is one type of short-distance wireless communication standards, which is used for wireless networking range of 10-20 meters mainly in home and offices. Each motion sensor node has an ID to be distinguished from others. Therefore, multi-person activity can also be tracked in this system. In this project, we have three channels of sensor measurements: body motion data, hand motion data, and the location information. Limited number of sensors will increase the difficulty of distinguishing the basic daily activities due to the inherited ambiguity. Therefore, we use a minimum setup of the wearable sensor system, which includes three inertial sensors, to recognize both hand gestures and body activities. Three wireless inertial sensors are attached on the right thigh, waist and the right wrist of the human subject to collect body and hand motion data.

We have solved the following problems:
1. We have developed two different versions of hardware setups for motion data collection. One is based on a wired motion sensor and a PDA to collect motion data. The other is a new motion sensor node using a VN-100 module and a Zigbee wireless communication module. The minimum number of sensors can significantly reduce the obtrusiveness of the system for motion data collection.
2. We presented three approaches to hand gesture recognition using a motion sensor. Individual gestures are recognized by the lower level HMMs using the training data from multiple users. The sequential constraints are modeled by a hierarchical hidden Markov model (HHMM) in the higher level. A neural network is used for segmentation of a gestures from daily non-gesture movements, so that the computational cost mainly caused by the HMM-based recognition algorithm can be reduced.
3. We introduced three approaches to human body activity recognition using different numbers of wearable sensor nodes. First, a sensor fusion-based algorithm is used for activity recognition in an office building. The algorithm combines neural networks and hidden Markov models to enhance the efficiency because HMM is only applied on selected segments of motion data by the neural networks. Second, a single motion sensor is used for online human daily activity recognition in an apartment. The constraints in the sequence of activities are modeled by an HMM and the modified short-time Viterbi algorithm is used for online body activity recognition. This approach has the advantage of reducing the obtrusiveness to the minimum. Third, motion data from the inertial sensor and location information from a motion capture system are fused for body activity recognition. The activities are first recognized using only the motion data from the inertial sensor and then Bayes’ theorem is used to integrate the location information to refine the recognition results. This approach has the advantage of reducing the obtrusiveness and the complexity of vision processing, while maintaining high accuracy of activity recognition.

4. We developed a dynamic Bayesian network-based approach to recognize human complex daily activities (body activities and hand gestures simultaneously) in a mock apartment. Three wireless motion sensors are worn on the right thigh, the waist, and the right hand of the human subject to provide motion data; while an optical motion capture system is used to obtain his/her location information. A three-level dynamic Bayesian network is implemented to model the intra-temporal and inter-temporal constraints among the location information, body activities and hand gestures. The body activity and hand gesture are estimated online using the short-time Viterbi algorithm. This approach has the advantage of reducing the obtrusiveness and the complexity of vision processing, while maintaining high accuracy of activity recognition.

5. We proposed a coherent framework to detect multiple types of anomalies in human’s daily life. Four types of abnormal behaviors: spatial anomaly, timing anomaly, duration anomaly and sequence anomaly, can be detected in realtime. The anomaly detection module can be integrated into the assisted living system, in which complex activities can be recognized and multiple types of anomalies can be detected.