Skills Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpoint

News: Our STEADY paper has been accepted to IROS 2024. Thanks to all coauthors, see you in Abu Dhabi!

We study skill transfer and discovery in the context of robotics control using representation learning.

We draw inspiration from the analysis of spectral decomposition of Markov Decision Processes, where representations coming from the spectral decomposition of transition dynamics can be regarded as skill sets. The representations can linearly represent the state-action value function induced by any policies, thus enabling the transferability of the representation-based skill sets across arbitrary tasks with the same transition dynamics.

Moreover, to handle the sim-to-real gap in the dynamics, we propose a skill discovery algorithm that learns new skills induced by the sim-to-real gap from real-world data. We encourage discovery of new skills by enforcing orthogonal constraints between the skills to learn and the skills from simulators, and then synthesize the policy using the combined skill sets. We demonstrate our methodologies by transferring quadrotor controllers from simulators to Crazyflie 2.1 quadrotors. We show that we can learn the representations from a single simulator task and transfer these to multiple different real-world tasks including hovering, taking off, landing and trajectory tracking. Our skill discovery approach helps narrow the sim-to-real gap and improve the real-world controller performance.