Cognition-enabled Robotic Agents


Dr. Dorothe Poggel (HWK)


Prof. Dr. Michael Beetz, Institute for Artificial Intelligence, University of Bremen


1.1.2019 - 31.12.2021

Statement of Problem

In the evolution of the human brain the cognitive capabilities have co-evolved with manipulation capabilities. Therefore, a promising approach to getting valuable insights into human cognition is the investigation of generative information processing models that enable autonomous robotic agents to master human-scale manipulation tasks. Despite the importance of the research focus it has received surprisingly little attention so far. This is presumably the case because conducting research in the focus area requires researchers to build complete robotic agents that operate autonomously in the real world. It also requires to bring together approaches and software components from different fields including computer vision, cognitive science, artificial intelligence, machine learning, and robot control. While the importance of the field can be seen in that the leading international conferences in robotics, artificial intelligence, and autonomous agents offer special tracks on this topic a common conference and communication forum for this field is still missing.

The investigation of generative models of robot agency is also the ultimate goal of the collaborative research center Everyday Activity Science and Engineering (EASE). The goal of EASE is the study of the design, realization, and analysis of information processing models that enable robotic agents (and humans) to master complex human-scale manipulation tasks that are mundane and routine. EASE not only investigates action selection and control but also the methods needed to acquire the knowledge, skills, and competence required for flexible, reliable, and efficient mastery of these activities.

As EASE is following an open research, open data, open source, and open teaching approach, founding an international and interdisciplinary study group is an essential complementary component of the EASE research agenda.


The ultimate goal of the study group is to support the establishment of the research area cognition-enabled robot agency and to position the CRC EASE as a key player in this field. In order to accomplish this goal we aim at the following objectives:

  • discuss recent developments in the areas of cognitive architectures in order to set up a agenda for research on architectures for robotic agents;
  • discuss and understand recent developments in cognitive models of humans accomplishing everyday manipulation tasks;
  • explore leading-edge approaches in knowledge representation and reasoning for agents accomplishing everyday manipulation tasks including data intensive methods for understanding intentional actions;
  • study information processing methods that can generate actionable knowledge from observation and language instructions;
  • investigate methods to acquire action knowledge from virtual reality and games with a purpose;
  • and study the injection of knowledge representation and reasoning methods into the perception-action loops of robotic agents.

An important secondary goal is to work towards building an international community of experts in Everyday Activity Science and Engineering, who investigate the problem of intelligent physical agency from a holistic perspective. To this end, we plan to organize workshops on selected topics in the research field; give guest researchers the opportunity to do focused research at HWK and in close cooperation with EASE scientists; and foster existing and initiate new collaborations with top universities and research institutes all over the world.

Work Plan

The focus in the first year will be on cognitive architectures and knowledge representation and reasoning in cognition-enabled robotic agents. The study of these topics set the stage for the planning of the EASE CRC for the second phase, which will focus on conceptual frameworks for Everyday Activity Science and Engineering. The discussion will also provide crossfertilizations with the planning of a proposal for a European flagship on robotics.

Our focus group also plans to have a close cooperation with the ZiF Research Group on Cognitive Behavior of Humans, Animals, and Machines: Situation Model Perspectives which has recently been approved to be funded for the period October 2019 - July 2020. This research group is headed by Werner Schneider and Helge Ritter from University of Bremen. Helge Ritter is also PI of an EASE subproject. Results of the first year could be a short position paper on cognitive architectures and knowledge representations for robotic agents. We plan to cooperate on the research topic “cognitive architectures” with David Vernon and possibly John Laird — two of the internationally leading experts in the field. On the aspects of cognitive models of humans we plan to work together with Thomas Schack and Werner Schneider from the CITEC excellence cluster in Bielefeld. We will also try to involve Bernhard Hommel (University of Leiden) and Anna Schuboe (University of Marburg). The main events that are planned to take place at the Hanse Wissenschaftskolleg are the following ones:

  • Kickoff meeting of the focus group (1 day) in Spring 2019.
  • 2-day symposium: cognitive architectures for robotic agents
  • 1-day symposium: cognitive models of human manipulation
  • 1-day or 2-day symposium: from language to action
  • 1-day or 2-day symposium: from observation, video, and VR to action
  • 1-day or 2-day symposium: from action to control
  • 1-day or 2-day symposium: from perception to action
  • Guest stays: postdoctoral EASE guest researcher

As major outcomes of the focus group we aiming to achieve:

  • A journal special issue on "Cognition-enabled Robots: Mastering Everyday Activities"
  • Significant contributions from HWK guest researchers to a planned combined textbook and Massive Open Online Course (MOOC) on the topic of the focus group
  • The establishment of an international conference on "Cognition-enabled Robots: Mastering Everyday Activities"


  • Collaborative Research Center 1320 EASE, Bremen, Germany
  • David Vernon, Carnegie Mellon University Africa, Rwanda
  • Frank Guerin, University of Aberdeen, Scotland
  • Alessandro Saffiotti, Örebro University, Sweden
  • Byoung-Tak Zhang, Seoul National University, Seoul, Korea
  • Kei Okada, University of Tokyo, Tokyo, Japan
  • Tom Ziemke, Linköping University, Sweden
  • Justus Piater, Universität Innsbruck, Austria
  • Giulio Sandini, Istituto Italiano die Tecnologia, Genova, Italy
  • Rachid Alami, LAAS/CNRS, Toulouse, France
  • Florentin Wörgötter, Georg-August-Universität Göttingen, Germany