Interdisciplinary Exploration of Generalizable Manipulation Policy Learning:
Paradigms and Debates

Introduction

Acquiring generalizable manipulation policies through data-driven, learning-based approaches is a fundamental problem in robotics. Despite advancements in this field, there are many unresolved questions that have resulted in a variety of, sometimes conflicting, paradigms. This workshop is designed to bring together researchers and practitioners from a diverse range of disciplines, robotics, vision, graphics, machine learning to share their insights and perspectives, with the goal of advancing interdisciplinary collaboration and promoting the development of innovative, new paradigms in generalizable manipulation policy learning. More specifically, we aim to discuss questions that are important for the future development of this field, including but not limited to:

• What action space is suitable for generalizable manipulation policy learning?
• What are the criteria for simulation in the context of generalizable manipulation policy learning?
• How realistic should the simulation be for sim-to-real?
• What data collection approaches are suitable for generalizable manipulation policy learning?
• What visual observation representation is suitable for generalizable manipulation policy learning?

Our targeted participants are researchers and engineers who are interested in the field of robotics and embodied AI, and have a background in machine learning, computer vision, computer graphics, natural language processing or related areas. The workshop will also be of interest to practitioners in industry who are working on the development of intelligent robots and other related systems.

In affiliation to this workshop, we are also organizing the ManiSkill2 Challenge, which is a large-scale robotic manipulation challenge that seeks to benchmark generalizable robotic agents that can interact with the 3D world. We will announce winners and host winner presentations in this workshop.

Workshop Schedule

Start Time (KST, UTC+9)	Event	Chair
9:00	Welcome and introduction	Rui Chen
9:15	Invited talks: Andy Zeng: From words to actions Xingyu Lin: Generalizable Manipulation with Large Internet Data and Small Robot Data Mini discussion	Rui Chen
10:15	Coffee break & Poster session
10:50	Invited talks: Shan Luo: Empowering Robotic Manipulation: Bridging Sim-to-Real with Tactile Representations Yuzhe Qin: Learning Generalizable Dexterous Manipulation from Vision, Touch and Human Demonstration Mini discussion	Rui Chen
11:50	Lunch break
13:30	Invited talks: Georgia Chalvatzaki: Real robots learn with structure Joseph J. Lim: Skill-based Robot Learning Mini discussion	Jiayuan Gu
14:30	Coffee break
15:00	Contributed talks: Siddhant Haldar: Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations Moritz Reuss: Goal-Conditioned Imitation Learning using Score-based Diffusion Policies Pietro Mazzaglia: World Models for Robotic Manipulation	Jiayuan Gu
15:30	Award ceremony of the ManiSkill2 challenge	Jiayuan Gu
15:40	Winner presentation: Xuetao Li	Jiayuan Gu
16:00	Panel discussion	Jiayuan Gu
16:50	Closing remark

Papers

Not Only Domain Randomization: Universal Policy with Embedding System Identification [PDF][Poster]

Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations [PDF][Poster]

MoVie: Visual Model-Based Policy Adaptation for View Generalization[PDF][Poster]

Learning Autonomous Ultrasound via Latent Task Representation and Robotic Skills Adaptation[PDF][Poster]

H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation[PDF][Poster]

SE(3)-Diffusion Policy: Online target adaptation for dynamic robotic manipulation[PDF][Poster]

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System[PDF][Poster]

Uncertainty-driven Affordance Discovery for Efficient Robotics Manipulation[PDF][Poster]

Towards General Food Acquisition with Human-Informed Actions[PDF][Poster]

FOCUS: Object-Centric World Models for Robotics Manipulation[PDF][Poster]

SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks[PDF][Poster]

A Two-stage Fine-tuning Strategy for Generalizable Manipulation Skill of Embodied AI[PDF][Poster]

The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning[PDF][Poster]

A Curriculum-Based Approach for Imitating Versatile Skills[PDF][Poster]

Goal-Conditioned Imitation Learning using Score-based Diffusion Policies[PDF][Poster]

Plug-And-Play Object-Centric Representations From "What" and "Where" Foundation Models[PDF][Poster]