Do you prefer audio books over printed ones, and if so, what is the advantage? Why is video conferencing so awkward? Why do you rely on SMS messages as a means of communication in some situations but on e-mailing in others? Did you ever notice that you raise your eyebrows when asking people a question – even on the telephone? And did you ever wonder how to make your point in a discussion most convincingly?
What insights can we gain from psychological measurements of biomechanical parameters and subjective judgements of manual actions (like object grasping) about the structures of the underlying cognitive representations? In this project, we will bring together statistical methods (like structure dimensional and principal components analysis) with
connectionist approaches employing artificial neural networks to test different hypotheses about the cognitive structure of manual actions. A major goal will be to emulate and control grasping behavior for a broad range of objects in kinematic simulations and - as a longer term objective - in real physics on a robot platform. read more »
For a long time Brain-Computer Interfaces (BCI) had been destined to act as pur spelling devices which enabled paralyzed people to communicate by mere thought. Our current projects aim to extend the scope of these devices and develop novel techniques for brain-robot interaction. A successful application of BCIs to robotic devices will have the tremendous advantage that the users will not be limited to pure communication tasks but also be able to manipulate their surrounding directly by only imagining actions.
In the last decades, many researchers have used evolutionary algorithms to adapt the topology and connection weights of recurrent neural networks for various control tasks. This has become a useful machine learning technique. Because handling large genomes is difficult, however, these neural networks typically contain only a few neurons. If the genome contains a recipe for construction of the network instead of the network itself, it can be much smaller. We have developed a method than can exactly do this, and performs very well on a number of different problems.
Priming of relevant motor degrees of freedom to achieve rapid alignment of motor actions can be conceptualised as the rapid selection of low-dimensional action manifolds that capture the essential motor degrees of freedom. The present project investigates the construction of such manifolds from training data and how observed action trajectories can be decomposed into traversals of manifolds from a previously acquired repertoire. To this end we focus on manual actions of an anthropomorphic hand and combine Unsupervised Kernel Regression (UKR, a recent statistical learning method) with Competitive Layer Models (CLM, a recurrent neural network architecture) to solve the tasks of manifold construction and dynamic action segmentation.
While language provides us with a concise code capturing much of the movement complexity of our mouth, we still lack a comparable representation for the movement of our hands. This project aims to create a database of human hand interaction patterns from a variety of multimodal data sources. An associated goal is to develop methods for the clustering of captured trajectory data into physics-based models of manual interaction. We hope that the resulting database can make a contribution towards a better grounding of control strategies for anthropomorphic robot hands and develop for robotics a similar utility as the WordNet database has for linguistics.
The Control Basis Framework (by Grupen et. al. 1998) is a powerful approach to closed loop control. This project aims at providing a library implementing the Control Basis Framework idea and possibly extending it to concurrent execution. Additional research is planned to investigate how to make machines learn to utilize the control affordances provided by synthesized controllers. read more »
We gradually increase our manual competence by exploring manual interaction spaces for many different kinds of objects. This is an active process that is very different from passive perception of "samples". The availability of humanoid robot hands offers the opportunity to investigate different strategies for such active exploration in realistic settings. In the present project, the investigation of such strategies shall be pursued from the perspective of „multimodal proprioception:“ correlating joint angles, partial contact information from touch sensors and joint torques as well as visual information about changes in finger and object position in such a way as to make predictions about "useful aspects" for shaping the ongoing interaction.
Unlike most existing approaches to the grasp selection task for anthropomorphic robot hands, this vision-based project aims for a solution, which does not depend on an a-priori known 3D shape of the object. Instead it uses a decomposition of the object view (obtained from mono or stereo cameras) into local, grasping-relevant shape primitives, whose optimal grasp type and approach direction are known or learned beforehand. Based on this decomposition a list of possible grasps can be generated and ordered according to the anticipated overall grasp quality.
Alignment is not restricted to support language, dialog functions and the execution of speech production. Similar mechanisms support the cooperative execution of more general actions. Besides the extension of alignment into the action domain, we hypothese that the formation of alignment is adaptive: Repetitions of actions facilitate the alignment and its adaptation is important for acquiring team expertise. In our opinion, adaptation and adaptive alignment paves the way for smooth and effective common acting.
Manipulation of paper is a rich domain of manual intelligence that we encounter in many daily tasks. The present project attempts to analyse and implement the "web" of visuo-motor coordination skills to endow an anthropomorphic robot hand with the ability to manipulate paper (and paper-like objects) in a variety of situations of increasing complexity. This will include aspects such as modeling interaction with compliant objects, action based representation as well as bimanual coordination to enable object transformations such as tearing and folding.
The spatio-temporal contact pattern during manipulation is a valuable source of information about object identity and object state, especially in uncertain environments. Using a bimanual robot manipulator setup with two 256 "pixel" touch sensor arrays, the present project is creating a "haptic pattern database" and investigates machine learning techniques to analyse the information contents of different haptic features and to extract identity and state information from haptic patterns. A closely connected goal are dynamic control strategies for contact movements with deformable or plastic objects, such as clay.
The goal of this project is to investigate the principles underlying co-evolution of a body shape and its neural controller. As a specific model system, we consider a robot hand that is controlled by a neural network. In contrast to existing work, we focus on the genetic regulation of neural circuits and morphological development. Our interest is directed at a better understanding of the facilitatory potential of co-evolution for the emergence of complex new functions, the interplay between development and evolution, the response of different genetic architectures to changing environments, as well as the role of important boundary constraints, such as wiring and tissue costs.
In the near future, more and more people will need assistance in everyday tasks while they still want to maintain a high degree of self-reliance. Cognitive robot servants will fulfill their individual needs. One promising way develop robots with a sufficiently high adaptability is to equip cognitive robots with task learning abilities, that lets them learn a task from demonstrations of naive (non-expert) users. This paradigm is widely known as Programming by Demonstration (PbD) or Imitation Learning.
What principles enable rapid and adaptive alignment in coordination?
This project investigates Gestalt principles and their generalization from the perceptual into the action/cooperation domain for modeling adaptive alignment and its functional replication in human-robot cooperation. Departing from learning algorithms for dynamic Gestalt formation in layered recurrent networks (Competitive Layer Model CLM), we develop a hybrid, hierarchical architecture for adaptive alignment in cooperation that integrates elements from connectionist and symbol-based representations. We evaluate its performance in a human-robot cooperation scenario involving two anthropomorphic hands mounted on a bimanual robot platform.
Motivated by the need of complex anthropomorphic robots to manage sophisticated spatial relationships between parts of their body and the environment, together with recent findings from the neurosciences about how the brain solves this challenge by means of a highly adaptable “body schema”, the present proposal pursues the goal of
Acquiring a profound knowledge about the cognitive processes underlying human-robot interaction is essential to better exploit the measurable components for brain-robot interfaces. The better the processes are understood, the better the EEG components originating from these processes can be used. A systematic evaluation of these components in connection with human-robot interaction is missing until today. Hence, it appears to be worthwhile to take a closer and impartial look at what is really happening on the cognitive level, as far as determinable by EEG signals.
MONARCA will develop and validate solutions for multi-parametric, long term monitoring of behavioural and physiological information relevant to bipolar disorder.
We explore a new BCI design for the control of robotic devices. Specifically, we show the first use of a code-modulating, Visually-Evoked Potential (cVEP)-based BCI for a navigation and control task.
Clothing provides a challenging test domain for research in the field of cognitive robotics. On the one hand, robots have to make use of commonsense knowledge to be able to understand the socially constructed meaning and function of garments. On the other hand, the variance resulting from deformations and differences between individual items of clothing calls for implicit representations which have to be learned from experience. Our robot uses topological, geometric, and subsymbolic knowledge representations for the manipulation of clothes with its anthropomorphic hands. read more »
The SARAFun project has been formed to enable a non-expert user to integrate a new bi-manual assembly task on a robot in less than a day. This will be accomplished by augmenting the robot with cutting edge sensory and cognitive abilities as well as reasoning abilities required to plan and execute an assembly task.
Because of the complex anatomy of the human hand, in the absence of external constraints a large number of postures and force combinations can be used to attain a stable grasp.
Which relations exist between properties of animals or people and their kinematic patterns? For example, can we tell, who performed a hand-over of which kind of object under which conditions just by looking at the sequence of joint angles? We try to find answers to these questions by employing a 3D motion tracking system.
“Scaffolding” is a powerful principle taken from psychology for organizing and optimizing the progress of a learner. The goal in this project is to combine this powerful structuring of a learning process with recent advances in deep and hierarchical reinforcement learning in order to make learning of an artificial agent more efficient and feasible when interaction data is much more limited than considered in current studies.
In the project REBA+, funded within DFG priority program "Autonomous Learning", we develop, implement and evaluate rich extensions of a robot's body schema, along with learning algorithms that use these representations as strong priors in order to enable rapid and autonomous usage of tools and a flexible coping with novel mechanical linkages between the body, the grasped tool and target objects.
Learning to play piano is a highly demanding activity, which is characterized by a high mental load. To reduce the mental load and, by doing so, to accelerate the learning process, we pursue to use exoskeleton Dexmo.
With the help of haptic puzzles, we investigate goal-oriented haptic exploration, search, learning and memory in complex 3D environments in order to both; enable multi-fingered robots with a sense of touch, and gain more insights into human meta-learning.
In this project we investigate approaches for unsupervised segmentation of interaction sequences based on multimodal data. The proposed procedure estimates segment borders across all modalities in a single pass.
We develop a 3D VR serious game to integrate valid haptic performance testing in highly adaptable and motivating virtual game scenarios. One goal is to offer stroke patients who need to improve or relearn their proprioceptive and tactile abilities an appealing and motivating tool. It allows them to monitor their progress in a more detailed manner than traditional methods, to select a suitable level of task difficulty, and potentially to improve their self-efficacy. For that purpose we combined multiple devices to a hardware framework creating a touchable virtual world. A set of scenarios is designed where players encounter different haptic discrimination or manipulation challenges in order to evaluate detailed hand and finger movements.
Twister is a novel haptic interface, consisting of a table with an integrated rotational joint on which a test object can be mounted. Twister measures the angular velocity and the orientation of the mounted object, it also detects when the contact with the object has been established or lost. Twister can provide vibro-tactile feedback. The objects can be rotated either by the study participant or controlled by the integrated motor. Our current research targets the influence of different factors on the performance during haptic rotation. We also pursue application of Twister for hand rehabilitation of stroke patients.
This project investigates the relationship between cortical activity (Electroencephalography, EEG), eye movements and mental representation structures (Structural Dimensional Analysis of Mental Representation, SDA-M) as a complex measure of ongoing cognitive processes during human-machine interaction (HMI) - an approach that has so far received little attention.
For a long time Brain-Computer Interfaces (BCI) had been destined to act as pur spelling devices which enabled paralyzed people to communicate by mere thought. Our current projects aim to extend the scope of these devices and develop novel techniques for brain-robot interaction. A successful application of BCIs to robotic devices will have the tremendous advantage that the users will not be limited to pure communication tasks but also be able to manipulate their surrounding directly by only imagining actions.
In the last decades, many researchers have used evolutionary algorithms to adapt the topology and connection weights of recurrent neural networks for various control tasks. This has become a useful machine learning technique. Because handling large genomes is difficult, however, these neural networks typically contain only a few neurons. If the genome contains a recipe for construction of the network instead of the network itself, it can be much smaller. We have developed a method than can exactly do this, and performs very well on a number of different problems.
While language provides us with a concise code capturing much of the movement complexity of our mouth, we still lack a comparable representation for the movement of our hands. This project aims to create a database of human hand interaction patterns from a variety of multimodal data sources. An associated goal is to develop methods for the clustering of captured trajectory data into physics-based models of manual interaction. We hope that the resulting database can make a contribution towards a better grounding of control strategies for anthropomorphic robot hands and develop for robotics a similar utility as the WordNet database has for linguistics.
We gradually increase our manual competence by exploring manual interaction spaces for many different kinds of objects. This is an active process that is very different from passive perception of "samples". The availability of humanoid robot hands offers the opportunity to investigate different strategies for such active exploration in realistic settings. In the present project, the investigation of such strategies shall be pursued from the perspective of „multimodal proprioception:“ correlating joint angles, partial contact information from touch sensors and joint torques as well as visual information about changes in finger and object position in such a way as to make predictions about "useful aspects" for shaping the ongoing interaction.
Unlike most existing approaches to the grasp selection task for anthropomorphic robot hands, this vision-based project aims for a solution, which does not depend on an a-priori known 3D shape of the object. Instead it uses a decomposition of the object view (obtained from mono or stereo cameras) into local, grasping-relevant shape primitives, whose optimal grasp type and approach direction are known or learned beforehand. Based on this decomposition a list of possible grasps can be generated and ordered according to the anticipated overall grasp quality.
Manipulation of paper is a rich domain of manual intelligence that we encounter in many daily tasks. The present project attempts to analyse and implement the "web" of visuo-motor coordination skills to endow an anthropomorphic robot hand with the ability to manipulate paper (and paper-like objects) in a variety of situations of increasing complexity. This will include aspects such as modeling interaction with compliant objects, action based representation as well as bimanual coordination to enable object transformations such as tearing and folding.
The spatio-temporal contact pattern during manipulation is a valuable source of information about object identity and object state, especially in uncertain environments. Using a bimanual robot manipulator setup with two 256 "pixel" touch sensor arrays, the present project is creating a "haptic pattern database" and investigates machine learning techniques to analyse the information contents of different haptic features and to extract identity and state information from haptic patterns. A closely connected goal are dynamic control strategies for contact movements with deformable or plastic objects, such as clay.
The goal of this project is to investigate the principles underlying co-evolution of a body shape and its neural controller. As a specific model system, we consider a robot hand that is controlled by a neural network. In contrast to existing work, we focus on the genetic regulation of neural circuits and morphological development. Our interest is directed at a better understanding of the facilitatory potential of co-evolution for the emergence of complex new functions, the interplay between development and evolution, the response of different genetic architectures to changing environments, as well as the role of important boundary constraints, such as wiring and tissue costs.
What principles enable rapid and adaptive alignment in coordination?
Acquiring a profound knowledge about the cognitive processes underlying human-robot interaction is essential to better exploit the measurable components for brain-robot interfaces. The better the processes are understood, the better the EEG components originating from these processes can be used. A systematic evaluation of these components in connection with human-robot interaction is missing until today. Hence, it appears to be worthwhile to take a closer and impartial look at what is really happening on the cognitive level, as far as determinable by EEG signals.
MONARCA will develop and validate solutions for multi-parametric, long term monitoring of behavioural and physiological information relevant to bipolar disorder.
We explore a new BCI design for the control of robotic devices. Specifically, we show the first use of a code-modulating, Visually-Evoked Potential (cVEP)-based BCI for a navigation and control task.
Clothing provides a challenging test domain for research in the field of cognitive robotics. On the one hand, robots have to make use of commonsense knowledge to be able to understand the socially constructed meaning and function of garments. On the other hand, the variance resulting from deformations and differences between individual items of clothing calls for implicit representations which have to be learned from experience. Our robot uses topological, geometric, and subsymbolic knowledge representations for the manipulation of clothes with its anthropomorphic hands. 
The SARAFun project has been formed to enable a non-expert user to integrate a new bi-manual assembly task on a robot in less than a day. This will be accomplished by augmenting the robot with cutting edge sensory and cognitive abilities as well as reasoning abilities required to plan and execute an assembly task.
Because of the complex anatomy of the human hand, in the absence of external constraints a large number of postures and force combinations can be used to attain a stable grasp.
Which relations exist between properties of animals or people and their kinematic patterns? For example, can we tell, who performed a hand-over of which kind of object under which conditions just by looking at the sequence of joint angles? We try to find answers to these questions by employing a 3D motion tracking system.
In the project REBA+, funded within DFG priority program "Autonomous Learning", we develop, implement and evaluate rich extensions of a robot's body schema, along with learning algorithms that use these representations as strong priors in order to enable rapid and autonomous usage of tools and a flexible coping with novel mechanical linkages between the body, the grasped tool and target objects. 
Learning to play piano is a highly demanding activity, which is characterized by a high mental load. To reduce the mental load and, by doing so, to accelerate the learning process, we pursue to use exoskeleton Dexmo.
There is a growing trend to teach playing an instrument such as
With the help of haptic puzzles, we investigate goal-oriented haptic exploration, search, learning and memory in complex 3D environments in order to both; enable multi-fingered robots with a sense of touch, and gain more insights into human meta-learning.
To download the CAD models of the MHSB please use the following link: 
In this project we investigate approaches for unsupervised segmentation of interaction sequences based on multimodal data. The proposed procedure estimates segment borders across all modalities in a single pass.
We develop a 3D VR serious game to integrate valid haptic performance testing in highly adaptable and motivating virtual game scenarios. One goal is to offer stroke patients who need to improve or relearn their proprioceptive and tactile abilities an appealing and motivating tool. It allows them to monitor their progress in a more detailed manner than traditional methods, to select a suitable level of task difficulty, and potentially to improve their self-efficacy. For that purpose we combined multiple devices to a hardware framework creating a touchable virtual world. A set of scenarios is designed where players encounter different haptic discrimination or manipulation challenges in order to evaluate detailed hand and finger movements.
Twister is a novel haptic interface, consisting of a table with an integrated rotational joint on which a test object can be mounted. Twister measures the angular velocity and the orientation of the mounted object, it also detects when the contact with the object has been established or lost. Twister can provide vibro-tactile feedback. The objects can be rotated either by the study participant or controlled by the integrated motor. Our current research targets the influence of different factors on the performance during haptic rotation. We also pursue application of Twister for hand rehabilitation of stroke patients.
This project investigates the relationship between cortical activity (Electroencephalography, EEG), eye movements and mental representation structures (Structural Dimensional Analysis of Mental Representation, SDA-M) as a complex measure of ongoing cognitive processes during human-machine interaction (HMI) - an approach that has so far received little attention.