Computer Vision and Signal Processing for Smart Homes

Computer Vision and processing of multimodal sensor data is very important to take Smart Homes to the next level. An intelligent everyday environment should be aware of its residents. It should understand their actions and ideally even be able to predict their behavior. In the KogniHome project, we are developing computer vision and signal processing methods for three demonstrators: (1) KogniChef, a cognitive cooking assistant, (2) KogniMirror, a human aware smart mirror and (3) KogniDoor, an intelligent entrance door.

KogniChef

KogniChef is a kitchen equipped with a variety of sensors and actors that are used to guide the user through the cooking of recipes by providing helpful assistance modules. These modules are either implicitly executed by the recipe state engine or they can be manually triggered by the user. The kitchen detects and tracks ingredients and tools on the worktop and recognizes which objects are grasped or moved. With multi-camera fusion (RGB, Kinect depth and thermal), the position, 3D shape, class, as well as temperature of the objects are known. Using integrated weighing scales the fill-rate of cooking containers and even stirring patterns can be recognized. KogniChef's hob-control enables the user to specify a container's desired temperature, which is then automatically reached and maintained by the system.

KogniMirror

KogniMirror is a smart mirror equipped with multiple RGBD cameras. It recognizes people, personalizes their information and provides useful mirror-related functions such as color-correction for color-blind people, face zooming or even a 360 degree view. The mirror is designed to minimize the need of interaction, but also provides gesture-based control. Enabled by multiple cameras and a display that is mounted behind a semi-transparent mirror, it can render an augmented and congruent view of the user.

KogniDoor

KogniDoor is an intelligent entrance door, which detects and recognizes the residents and can automatically opens the door for them. It also provides context related information on a screen that is attached on the inside of the door as well as through an additional E-paper display on the outside. For the automatic opening and closing of the door, we developed algorithms to avoid situations in which the door might injure people by unintentionally closing on their fingers or by colliding during opening.