Neuroinformatics Group

Universität BielefeldTechnische FakultätNI

Cognitive Robotics

Cognitive Robotics draws from classical robotics, artificial intelligence, cognitive science and neurobiology to elucidate and synthesize aspects of action-oriented intelligence. Using a robot system with a multi-fingered manipulator and an active binocular camera head we investigate strategies how to coordinate the actions of such system with those of a human partner. We focus on dextrous manipulation of objects, combining tactile and visual sensing, the joining of action primitives into action sequences and the development of learning algorithms.

Related Research Projects

Physical Reasoning AI

Understanding and reasoning about physics is an important ability of intelligent agents. We are developing an AI agent capable of solving physical reasoning tasks. If you would like to know more about this project/thesis opportunity, check the websites [1][2] or contact Dr. Andrew Melnik.

read more »


The NeuTouch ITN aims at improving artificial tactile systems, by training a new generation of researchers that study how human and animal’s tactile systems work, develop a new type of technology that is based on the same principles, and use this technology for building robots that can help humans in daily tasks and artific

read more »

Autonomous Haptic Exploration

Optimization for a floating Myrmex

gazebo_sim With the help of a physics-driven simulation environment and reinforcement learning, our goal is to learn autonomous haptic exploration behavior in a three-dimensional environment.

read more »

REBA+: Robots Exploring Tools

In the project REBA+, funded within DFG priority program "Autonomous Learning", we develop, implement and evaluate rich extensions of a robot's body schema, along with learning algorithms that use these representations as strong priors in order to enable rapid and autonomous usage of tools and a flexible coping with novel mechanical linkages between the body, the grasped tool and target objects.

read more »

Human grasping forces and positions synergies

Because of the complex anatomy of the human hand, in the absence of external constraints a large number of postures and force combinations can be used to attain a stable grasp.

read more »

What does a hand-over tell? - Comparative analysis of kinematic data

Which relations exist between properties of animals or people and their kinematic patterns? For example, can we tell, who performed a hand-over of which kind of object under which conditions just by looking at the sequence of joint angles? We try to find answers to these questions by employing a 3D motion tracking system.

read more »

Scaffolding for hierarchical interaction learning of invertible processes

“Scaffolding” is a powerful principle taken from psychology for organizing and optimizing the progress of a learner. The goal in this project is to combine this powerful structuring of a learning process with recent advances in deep and hierarchical reinforcement learning in order to make learning of an artificial agent more efficient and feasible when interaction data is much more limited than considered in current studies.

read more »

SARAFun: Smart Assembly Robots

The SARAFun project has been formed to enable a non-expert user to integrate a new bi-manual assembly task on a robot in less than a day. This will be accomplished by augmenting the robot with cutting edge sensory and cognitive abilities as well as reasoning abilities required to plan and execute an assembly task.

read more »

A Minimal Cognitive Architecture: Planning Ahead through Mental Simulation

Trial and error planning exploiting the internal model.

A minimal cognitive system is realized in a bottom-up approach in which an existing behavior-based robot control system is extended towards a system that is capable of planning ahead. A grounded internal body model is recruited in internal simulation which allows to test different behaviors mentally before actually carrying a suitable and not dangerous one out on the real system. In the project such a system is used for the control of a hexapod walker.

read more »

Incipient Slip Detection

This 16x16 tactile sensor array is our workhorse for tactile-based robotics research. Providing 256 taxels on a grid of size 8cm x 8cm, we have a spatial resolution of 5mm. The modular design allows to stitch individual modules together to yield a larger tactile-sensitive surface.

read more »

Bimanual Interaction with Clothes

Clothing provides a challenging test domain for research in the field of cognitive robotics. On the one hand, robots have to make use of commonsense knowledge to be able to understand the socially constructed meaning and function of garments. On the other hand, the variance resulting from deformations and differences between individual items of clothing calls for implicit representations which have to be learned from experience. Our robot uses topological, geometric, and subsymbolic knowledge representations for the manipulation of clothes with its anthropomorphic hands. read more »


   Roboticists have made huge efforts to mimic the human hand, not only from the form but also from the functionalities. read more »

Gestalt Learning as a Basis for Adaptive Alignment

CLM What principles enable rapid and adaptive alignment in coordination?

This project investigates Gestalt principles and their generalization from the perceptual into the action/cooperation domain for modeling adaptive alignment and its functional replication in human-robot cooperation. Departing from learning algorithms for dynamic Gestalt formation in layered recurrent networks (Competitive Layer Model CLM), we develop a hybrid, hierarchical architecture for adaptive alignment in cooperation that integrates elements from connectionist and symbol-based representations. We evaluate its performance in a human-robot cooperation scenario involving two anthropomorphic hands mounted on a bimanual robot platform.

read more »

Dextrous Manipulation

Together with the advances in building anthropomorphic robot hands, we are facing the question of how to dexterously control such complex robots with up to 20 degrees of freedom in up to five fingers and a wrist. Implementing fixed grasp and manipulation programs does not lead to satisfying results. In our work, we propagate a manifold representation of such movements recorded from human demonstration. The main idea is to construct manifolds embedded in the finger joint angle space which represent the subspace of hand postures associated with a specific manipulation movement.

read more »

Co-evolution of neural and morphological development for grasping

The goal of this project is to investigate the principles underlying co-evolution of a body shape and its neural controller. As a specific model system, we consider a robot hand that is controlled by a neural network. In contrast to existing work, we focus on the genetic regulation of neural circuits and morphological development. Our interest is directed at a better understanding of the facilitatory potential of co-evolution for the emergence of complex new functions, the interplay between development and evolution, the response of different genetic architectures to changing environments, as well as the role of important boundary constraints, such as wiring and tissue costs.

read more »

Tactile Exploration Database

Tactile Sensor Array The spatio-temporal contact pattern during manipulation is a valuable source of information about object identity and object state, especially in uncertain environments. Using a bimanual robot manipulator setup with two 256 "pixel" touch sensor arrays, the present project is creating a "haptic pattern database" and investigates machine learning techniques to analyse the information contents of different haptic features and to extract identity and state information from haptic patterns. A closely connected goal are dynamic control strategies for contact movements with deformable or plastic objects, such as clay.

read more »

Manipulating Paper

Manipulation of paper is a rich domain of manual intelligence that we encounter in many daily tasks. The present project attempts to analyse and implement the "web" of visuo-motor coordination skills to endow an anthropomorphic robot hand with the ability to manipulate paper (and paper-like objects) in a variety of situations of increasing complexity. This will include aspects such as modeling interaction with compliant objects, action based representation as well as bimanual coordination to enable object transformations such as tearing and folding.

read more »

Adaptive alignment in human-robot-cooperation

Alignment is not restricted to support language, dialog functions and the execution of speech production. Similar mechanisms support the cooperative execution of more general actions. Besides the extension of alignment into the action domain, we hypothese that the formation of alignment is adaptive: Repetitions of actions facilitate the alignment and its adaptation is important for acquiring team expertise. In our opinion, adaptation and adaptive alignment paves the way for smooth and effective common acting.

read more »

Autonomous Exploration of Manual Interaction Space

We gradually increase our manual competence by exploring manual interaction spaces for many different kinds of objects. This is an active process that is very different from passive perception of "samples". The availability of humanoid robot hands offers the opportunity to investigate different strategies for such active exploration in realistic settings. In the present project, the investigation of such strategies shall be pursued from the perspective of „multimodal proprioception:“ correlating joint angles, partial contact information from touch sensors and joint torques as well as visual information about changes in finger and object position in such a way as to make predictions about "useful aspects" for shaping the ongoing interaction.

read more »

From action capture to a database of physics-based manual interaction

While language provides us with a concise code capturing much of the movement complexity of our mouth, we still lack a comparable representation for the movement of our hands. This project aims to create a database of human hand interaction patterns from a variety of multimodal data sources. An associated goal is to develop methods for the clustering of captured trajectory data into physics-based models of manual interaction. We hope that the resulting database can make a contribution towards a better grounding of control strategies for anthropomorphic robot hands and develop for robotics a similar utility as the WordNet database has for linguistics.

read more »

Imitation Learning

Human Assistance ScenarioIn the near future, more and more people will need assistance in everyday tasks while they still want to maintain a high degree of self-reliance. Cognitive robot servants will fulfill their individual needs. One promising way develop robots with a sufficiently high adaptability is to equip cognitive robots with task learning abilities, that lets them learn a task from demonstrations of naive (non-expert) users. This paradigm is widely known as Programming by Demonstration (PbD) or Imitation Learning.

read more »

Robots Exploring their Bodies Autonomously (REBA)

Motivated by the need of complex anthropomorphic robots to manage sophisticated spatial relationships between parts of their body and the environment, together with recent findings from the neurosciences about how the brain solves this challenge by means of a highly adaptable “body schema”, the present proposal pursues the goal of
read more »

Vision-based Grasping

Unlike most existing approaches to the grasp selection task for anthropomorphic robot hands, this vision-based project aims for a solution, which does not depend on an a-priori known 3D shape of the object. Instead it uses a decomposition of the object view (obtained from mono or stereo cameras) into local, grasping-relevant shape primitives, whose optimal grasp type and approach direction are known or learned beforehand. Based on this decomposition a list of possible grasps can be generated and ordered according to the anticipated overall grasp quality.

read more »

Representation of manual actions for adaptive alignment in human-robot-cooperation

Priming of relevant motor degrees of freedom to achieve rapid alignment of motor actions can be conceptualised as the rapid selection of low-dimensional action manifolds that capture the essential motor degrees of freedom. The present project investigates the construction of such manifolds from training data and how observed action trajectories can be decomposed into traversals of manifolds from a previously acquired repertoire. To this end we focus on manual actions of an anthropomorphic hand and combine Unsupervised Kernel Regression (UKR, a recent statistical learning method) with Competitive Layer Models (CLM, a recurrent neural network architecture) to solve the tasks of manifold construction and dynamic action segmentation.

read more »

Learning Control Behaviour within the Control Basis Framework

The Control Basis Framework (by Grupen et. al. 1998) is a powerful approach to closed loop control. This project aims at providing a library implementing the Control Basis Framework idea and possibly extending it to concurrent execution. Additional research is planned to investigate how to make machines learn to utilize the control affordances provided by synthesized controllers. read more »

Sensory-motor representations & error learning - experimental analysis of manual intelligence in first order & virtual reality

One central issue for the cognitive control of movement is the compensation of errors and learning processes that enhance error compensation mechanisms. This is especially true for very precise movements such as many manual actions. The present project combines methods and conventional experimental settings (first order reality) with approaches from Virtual Reality and Augmented Reality to embed subjects in interaction loops in which the occurrence and perception of errors can be manipulated and studied in novel ways. In this way we hope to gain new clues about error correction mechanisms, error compensation learning and their replication in technical systems such as robots. read more »

From Cognitive Representation to Technical Synthesis of Manual Action

What insights can we gain from psychological measurements of biomechanical parameters and subjective judgements of manual actions (like object grasping) about the structures of the underlying cognitive representations? In this project, we will bring together statistical methods (like structure dimensional and principal components analysis) with connectionist approaches employing artificial neural networks to test different hypotheses about the cognitive structure of manual actions. A major goal will be to emulate and control grasping behavior for a broad range of objects in kinematic simulations and - as a longer term objective - in real physics on a robot platform. read more »