CoRL 2024 Papers — Page 2
Conference on Robot Learning · 264 papers
InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation
Andrew Choong-Won Lee (University of California Davis), Iman Soltani (University of California Davis)
Robotic IntelligenceConvolutional Neural NetworkTransformerMultimodality
🎯 What it does: Proposed a simulation learning framework called InterACT for dual-arm manipulation, which uses a hierarchical attention transformer to capture mutual dependencies between dual-arm joint states and visual inputs, thereby generating synchronized action sequences.
JA-TN: Pick-and-Place Towel Shaping from Crumpled States based on TransporterNet with Joint-Probability Action Inference
Halid Abdulrahim Kadi (University of St Andrews), Kasim Terzić (University of St Andrews)
Domain AdaptationRobotic IntelligenceTransformerImagePoint CloudBenchmark
🎯 What it does: This paper proposes a joint probability action inference model, JA-TN, based on TransporterNet, specifically designed for single-handed pick-and-place towel folding tasks, covering the complete process from wrinkled states to flatness and then folding, and achieving a vision-based deep learning controller from simulation to real-world environments;
Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation
Jan Bruedigam, Simon Le Cleac'h (Boston Dynamics AI Institute)
OptimizationRobotic IntelligenceReinforcement Learning
🎯 What it does: Proposed a general dynamic feasibility planner that generates whole-body and dexterous manipulation state-action trajectories, and directly injects demonstrations into reinforcement learning to achieve sim-to-real transfer.
JointMotion: Joint Self-Supervision for Joint Motion Prediction
Royden Wagner (Karlsruhe Institute of Technology), Carlos Fernandez
Autonomous DrivingTransformerSupervised Fine-TuningAuto EncoderMultimodalityTime Series
🎯 What it does: In the joint motion prediction task for autonomous vehicles, the self-supervised pre-training framework JointMotion is proposed, which pre-trains the model using scene-level and instance-level objectives.
KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidance
Jingxian Lu (Renmin University of China), Xuelong Li (Shanghai Artificial Intelligence Laboratory)
OptimizationComputational EfficiencyRobotic IntelligenceTransformerReinforcement LearningVision Language ModelOptical FlowVideoMultimodalitySequential
🎯 What it does: Propose a hybrid key state-guided online imitation learning method called KOI, which improves trajectory matching rewards by extracting semantic and motion key states, thereby achieving efficient online imitation learning.
KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation
Hongyi Chen (Carnegie Mellon University), Jeffrey Ichnowski (Georgia Institute of Technology)
Explainability and InterpretabilityRobotic IntelligenceConvolutional Neural NetworkWorld ModelImage
🎯 What it does: Designed an imitation learning method called KOROL that leverages the Koopman operator for visual interpretable object feature learning, enabling robot manipulation without requiring real object state labels.
Language-guided Manipulator Motion Planning with Bounded Task Space
Thies Oelerich (TU Wien), Andreas Kugi (TU Wien)
OptimizationRobotic IntelligenceTransformerLarge Language ModelPrompt EngineeringVision Language ModelVision-Language-Action ModelImageText
🎯 What it does: Developed a zero-shot, no-training robotic arm motion planning framework based on a language model, converting natural language instructions into safe and smooth trajectories;
Large Scale Mapping of Indoor Magnetic Field by Local and Sparse Gaussian Processes
Iad ABDUL-RAOUF (Universite Paris-Saclay), Alexis Paljic (Centre De Robotique)
Mixture of ExpertsPhysics Related
🎯 What it does: Propose a magnetic field mapping framework based on local sparse Gaussian process (Gradient-DTC) and local Bayesian committee machine (LBCM), which can efficiently and smoothly construct magnetic field maps and estimate uncertainty in large-scale indoor environments.
Learning a Distributed Hierarchical Locomotion Controller for Embodied Cooperation
Chuye Hong (Tsinghua University), Huaping Liu (Tsinghua University)
Robotic IntelligenceRecurrent Neural NetworkReinforcement LearningBenchmark
🎯 What it does: Proposed a distributed hierarchical reinforcement learning framework enabling robots to achieve multi-body collaboration through gait control under fully decentralized conditions.
Learning Compositional Behaviors from Demonstration and Language
Weiyu Liu (Stanford University), Jiajun Wu (Stanford University)
Robotic IntelligenceTransformerLarge Language ModelDiffusion modelText
🎯 What it does: Proposed the BLADE framework, integrating large language models, language-annotated demonstrations, and high-level abstract behaviors to achieve long-term robotic manipulation tasks;
Learning Decentralized Multi-Biped Control for Payload Transport
Bikram Pandit (Oregon State University), Alan Fern (Oregon State University)
Robotic IntelligenceRecurrent Neural NetworkReinforcement Learning
🎯 What it does: This paper proposes a decentralized multi-bipedal robot carrier controller named decMBC, aiming to enable any number and configuration of bipedal robots to cooperatively transport a load on rough terrain, and verifies its scalability and robustness through simulation-to-real hardware transfer.
Learning Differentiable Tensegrity Dynamics using Graph Neural Networks
Nelson Chen (Rutgers University), Mridul Aanjaneya (Rutgers University)
Explainability and InterpretabilityComputational EfficiencyRobotic IntelligenceGraph Neural NetworkImageGraphTime SeriesOrdinary Differential Equation
🎯 What it does: Use graph neural networks to learn the dynamic behavior of tension structure robots, and achieve simulation through a hybrid analytical and learning model.
Learning Granular Media Avalanche Behavior for Indirectly Manipulating Obstacles on a Granular Slope
Haodi Hu (University of Southern California), Daniel Seita (University of Southern California)
OptimizationRobotic IntelligenceTransformerImage
🎯 What it does: Propose the GRAIN method, which uses the leg digging of a walking robot to trigger a sand pile avalanche, enabling indirect movement of obstacles;
Learning H-Infinity Locomotion Control
Junfeng Long (Shanghai AI Laboratory), Jiangmiao Pang (Shanghai AI Laboratory)
Robotic IntelligenceReinforcement Learning
🎯 What it does: Studied enhancing the robustness of quadruped robots in different terrains and strongly perturbed environments by utilizing a learnable adversarial perturber within the H∞ learning framework.
Learning Long-Horizon Action Dependencies in Sampling-Based Bilevel Planning
Bartłomiej Cieślar (MIT), Jorge Mendez-Mendez (Stony Brook University)
OptimizationData-Centric LearningRobotic IntelligenceTransformerDiffusion model
🎯 What it does: Studied how to learn to predict long-term action dependencies and utilized this prediction for heuristic backtracking search in sample-based hierarchical task-motion planning.
Learning Performance-oriented Control Barrier Functions Under Complex Safety Constraints and Limited Actuation
Lakshmideepakreddy Manda (Johns Hopkins University), Mahyar Fazlyab (Johns Hopkins University)
OptimizationBenchmarkPhysics Related
🎯 What it does: Proposed a new self-supervised learning framework for learning control barrier functions (CBF) under complex safety constraints and limited driving conditions to ensure the safety and performance of nonlinear control systems.
Learning Quadruped Locomotion Using Differentiable Simulation
Yunlong Song (University of Zurich), Davide Scaramuzza (University of Zurich)
Robotic IntelligenceReinforcement LearningWorld ModelVideo
🎯 What it does: By constructing a differentiable simulation framework that combines high-fidelity non-differentiable simulation with a simplified single rigid-body dynamics proxy model, the gait control strategy for quadruped robots (Mini Cheetah) is trained to achieve rapid learning of walking under single-machine non-parallelized conditions with minimal samples, and directly transfer to real robots.
Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning
Dhruva Tirumala (Google DeepMind), Nicolas Heess (Google DeepMind)
Knowledge DistillationRobotic IntelligenceRecurrent Neural NetworkReinforcement LearningNeural Radiance FieldImage
🎯 What it does: This paper utilizes multi-agent deep reinforcement learning to end-to-end train robot soccer strategies based solely on self-perception from a helmet-mounted RGB camera, achieving zero-shot transfer from simulation to real robots.
Learning Robotic Locomotion Affordances and Photorealistic Simulators from Human-Captured Data
Alejandro Escontrela (University of California, Berkeley), Pieter Abbeel (University of California, Berkeley)
Data SynthesisRobotic IntelligenceTransformerSupervised Fine-TuningGaussian SplattingSimultaneous Localization and MappingImageVideoPoint CloudMesh
🎯 What it does: Propose the PAWS system, which uses a portable three-camera setup combined with foot markers to capture human walking trajectories, extracting camera poses, foot positions, scene meshes, and 3D Gaussian radiance fields; construct the 10-hour indoor and outdoor PAWS-Data dataset; train the Locomotion Affordance Model (PAWS-LAM) based on this dataset, demonstrating safe navigation through feasibility planning on quadruped robots; simultaneously develop PAWS-Sim, a high-speed photorealistic rendering simulator integrated with IsaacSim.
Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching
Eugenio Chisari (University of Freiburg), Abhinav Valada (University of Freiburg)
Robotic IntelligenceConvolutional Neural NetworkDiffusion modelFlow-based ModelPoint CloudBenchmark
🎯 What it does: Designed and implemented a robot imitation learning algorithm called PointFlowMatch based on point cloud observations and Conditional Flow Matching (CFM), for learning strategies in complex manipulation tasks.
Learning to Look: Seeking Information for Decision Making via Policy Factorization
Shivin Dass (University of Texas at Austin), Roberto Martín-Martín (University of Texas at Austin)
Robotic IntelligenceReinforcement Learning
🎯 What it does: This paper proposes a dual-strategy framework called DISaM for robotic manipulation tasks requiring active information seeking. First, the Information Reception (IR) strategy completes the given context operation, then the Information Seeking (IS) strategy actively collects context through perception, and both strategies collaborate to complete the task.
Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning
Zhecheng Yuan (Tsinghua University), Huazhe Xu (Tsinghua University)
Robotic IntelligenceConvolutional Neural NetworkReinforcement LearningContrastive LearningImage
🎯 What it does: Propose the Maniwhere framework in visual reinforcement learning, training robots to achieve general control under various visual perturbations (perspective, appearance, lighting, objects).
Learning to Open and Traverse Doors with a Legged Manipulator
Mike Zhang (ETH Zurich), Marco Hutter (ETH Zurich)
Knowledge DistillationRobotic IntelligenceRecurrent Neural NetworkSupervised Fine-TuningReinforcement LearningTime Series
🎯 What it does: Designed and deployed a learning-based control strategy enabling the limb-equipped quadruped robot ANYmal to open and pass through doors without prior knowledge of the door's opening direction or swinging direction.
Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models
Jacob Levy (University of Texas at Austin), David Fridovich-Keil (University of Texas at Austin)
Robotic IntelligenceReinforcement LearningWorld ModelTime SeriesSequential
🎯 What it does: Developed and implemented a semi-structured dynamics model (SSRL), combining Lagrangian physical constraints with an external force estimator based on historical information, successfully enabling a real quadruped robot to learn locomotion using only 30 minutes of real-world data.
Learning Transparent Reward Models via Unsupervised Feature Selection
Daulet Baimukashev (Aalto University), Ville Kyrki (Aalto University)
Explainability and InterpretabilityRepresentation LearningData-Centric LearningReinforcement LearningBenchmark
🎯 What it does: Learn a transparent reward model by reconstructing the reward function from expert demonstrations using unsupervised feature selection and maximum entropy inverse reinforcement learning, and train policies with this reward.
Learning Visual Parkour from Generated Images
Alan Yu (MIT CSAIL, Institute for AI and Fundamental Interactions), Phillip Isola (MIT CSAIL, Institute for AI and Fundamental Interactions)
GenerationData SynthesisDepth EstimationDomain AdaptationRobotic IntelligenceTransformerLarge Language ModelReinforcement LearningPrompt EngineeringDiffusion modelGaussian SplattingOptical FlowImageVideo
🎯 What it does: Train a robot dog to complete visual parkour tasks in a simulated environment using generative models and physics-guided image synthesis, achieving zero-shot direct transfer to the real world.
Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions
Juan Del Aguila Ferrandis (University of Edinburgh), Sethu Vijayakumar (Alan Turing Institute)
Robotic IntelligenceRecurrent Neural NetworkReinforcement LearningImageMultimodalityTime Series
🎯 What it does: Achieved non-grasping manipulation tasks under occlusion conditions by training a visual tactile state estimator and embedding it into a reinforcement learning control loop.
Legolas: Deep Leg-Inertial Odometry
Justin Wasserman (University of Illinois at Urbana-Champaign), Abhinav Gupta (Carnegie Mellon University)
Data SynthesisRobotic IntelligenceConvolutional Neural NetworkSimultaneous Localization and MappingTime SeriesSequential
🎯 What it does: Proposes a purely data-driven leg inertial odometry method called Legolas, which can predict small-step motions using quadruped robot leg joint states and IMU information without relying on analytical modeling or foot contact sensors, achieving zero-tuning parameter transfer from simulation data to real robots.
LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Video
Noriaki Hirose (University of California Berkeley), Sergey Levine (University of California Berkeley)
Depth EstimationRobotic IntelligenceLarge Language ModelVision Language ModelVision-Language-Action ModelVideoTextMultimodality
🎯 What it does: Leverage first-person videos without labels or actions, combined with large models to automatically generate language instructions and corresponding action labels, to train a language-conditioned navigation policy.
Lessons from Learning to Spin “Pens”
Jun Wang (University Of California San Diego), Xiaolong Wang (University Of California San Diego)
Domain AdaptationRobotic IntelligenceTransformerSupervised Fine-TuningReinforcement LearningImageMultimodalityPoint Cloud
🎯 What it does: Studied a learning-based robotic grasping system capable of continuously rotating pen-shaped objects and achieving multiple complete rotations on a real robot.
Let Occ Flow: Self-Supervised 3D Occupancy Flow Prediction
Yili Liu (Zhejiang University), Yue Wang (Zhejiang University)
Autonomous DrivingOptimizationRepresentation LearningTransformerFlow-based ModelNeural Radiance FieldOptical FlowVideoPoint CloudSequential
🎯 What it does: Designed a self-supervised 3D occupancy and occupancy flow prediction framework named Let Occ Flow, trained using multi-camera sequences without requiring 3D annotations;
Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation
Tong Zhang (Shanghai Artificial Intelligence Laboratory), Yang Gao (Shanghai Artificial Intelligence Laboratory)
Robotic IntelligenceConvolutional Neural NetworkVision-Language-Action ModelTextPoint Cloud
🎯 What it does: Proposes the SGRv2 framework, enhancing sample efficiency in vision-motion learning by introducing action locality.
Leveraging Mutual Information for Asymmetric Learning under Partial Observability
Hai Huu Nguyen (Northeastern University), Robert Platt (Northeastern University)
Domain AdaptationRepresentation LearningRobotic IntelligenceTransformerReinforcement LearningSequential
🎯 What it does: Propose an asymmetric reinforcement learning framework based on mutual information, which encourages information gathering and memory using the mutual information between the state and history, and trains a Transformer-based memory network to perform tasks in partially observable environments.
LiDARGrid: Self-supervised 3D Opacity Grid from LiDAR for Scene Forecasting
Chuanyu Pan (Honda Research Institute), Aolin Xu (Honda Research Institute)
Autonomous DrivingRepresentation LearningConvolutional Neural NetworkNeural Radiance FieldAuto EncoderPoint Cloud
🎯 What it does: Propose a self-supervised 3D opacity grid (LiDARGrid) representation based on LiDAR point clouds, and use this representation to achieve scene temporal prediction, motion area detection, and depth completion.
Lifelong Autonomous Improvement of Navigation Foundation Models in the Wild
Kyle Stachowicz (University of California Berkeley), Sergey Levine (University of California Berkeley)
Autonomous DrivingRobotic IntelligenceReinforcement LearningImage
🎯 What it does: Developed and deployed a visual navigation foundation model called LiReN, which, after offline reinforcement learning pre-training, continuously performs autonomous online fine-tuning in real environments, ultimately achieving a success rate improvement from 40% to 75%.
LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning
Dantong Niu (University Of California Berkeley), Roei Herzig (University Of California Berkeley)
Robotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelVision-Language-Action ModelImageMultimodality
🎯 What it does: This paper proposes LLARVA, a multimodal model for robot learning through structured instruction tuning, which jointly predicts 2-D visual trajectories to achieve unified learning and generalization across multiple robots, tasks, and scenarios.
MaIL: Improving Imitation Learning with Selective State Space Models
Xiaogang Jia (Karlsruhe Institute of Technology), Gerhard Neumann (Karlsruhe Institute of Technology)
Robotic IntelligenceReinforcement LearningVision Language ModelDiffusion modelImageBenchmark
🎯 What it does: This study proposes the MaIL architecture, which achieves visual imitation learning using the Mamba state space model, replacing Transformer.
Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
Jiafei Duan (University of Washington), Ranjay Krishna (University of Washington)
Robotic IntelligenceTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: Propose a framework named MANIPULATE-ANYTHING based on vision-language models, capable of generating automated robot demonstrations in real environments and achieving zero-shot task execution while generating training data
ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
Zeyi Liu (Stanford University), Shuran Song (Stanford University)
Robotic IntelligenceTransformerVision-Language-Action ModelDiffusion modelVideoMultimodalityAudio
🎯 What it does: This paper proposes an 'ear-in-hand' gripper that simultaneously collects visual and acoustic data during human demonstrations, and trains robots to learn perception and control strategies for manipulation tasks with rich contact interactions.
MBC: Multi-Brain Collaborative Control for Quadruped Robots
Hang Liu (University of Michigan), Houde Liu (Tsinghua University)
Robotic IntelligenceReinforcement LearningAuto EncoderPoint Cloud
🎯 What it does: By designing a multi-brain collaborative control (MBC) system that integrates blind navigation strategies with perception strategies, quadruped robots can maintain stable locomotion even when perception fails, and support hot-swapping of perception modules.
Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skills
Tianhao Wei (Carnegie Mellon University), Changliu Liu (Carnegie Mellon University)
Robotic IntelligenceTransformerLarge Language ModelPrompt Engineering
🎯 What it does: Propose the Meta-Control framework, which leverages large language models (LLMs) to automatically synthesize model-based control systems tailored for different tasks, enabling adaptive robot skill generation.
MILES: Making Imitation Learning Easy with Self-Supervision
Georgios Papagiannis (Imperial College London), Edward Johns (Imperial College London)
Robotic IntelligenceRecurrent Neural NetworkContrastive LearningMultimodalityTime Series
🎯 What it does: Using self-supervised data collection with a single demonstration and one environment reset, supplementary trajectories are generated and behavior cloning is performed to enable the robot to learn to execute multiple complex tasks.
MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation
Kelin Yu (Georgia Institute of Technology), Ye Zhao (Georgia Institute of Technology)
Representation LearningRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningContrastive LearningImageMultimodalityAudio
🎯 What it does: MimicTouch collects multimodal tactile (GelSight visual sensors) and audio data by allowing human hands to directly demonstrate tactile interactions, then learns low-dimensional representations, generates offline policies using non-parametric imitation learning, and fine-tunes them through online residual reinforcement learning to address the embodiment gap between human hands and robot grippers.
Mobile ALOHA: Learning Bimanual Mobile Manipulation using Low-Cost Whole-Body Teleoperation
Zipeng Fu (Stanford University), Chelsea Finn (Stanford University)
Robotic IntelligenceDiffusion modelImage
🎯 What it does: Designed and implemented a low-cost, portable dual-arm mobile robot called Mobile ALOHA, collecting demonstration data through whole-body teleoperation; utilized this data along with static ALOHA data for co-training, achieving successful execution of seven complex mobile dual-arm tasks.
Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs
Zhuo Xu (Google DeepMind), Jie Tan (Google DeepMind)
Robotic IntelligenceVision Language ModelVision-Language-Action ModelSimultaneous Localization and MappingVideoTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: Based on recorded demonstration videos and multimodal instructions (text + image), a hierarchical robot navigation strategy called Mobility VLA was designed, achieving understanding and execution of multimodal navigation instructions in real environments.
Modeling Drivers’ Situational Awareness from Eye Gaze for Driving Assistance
Abhijat Biswas (Carnegie Mellon University), Henny Admoni (Carnegie Mellon University)
SegmentationAutonomous DrivingConvolutional Neural NetworkImageVideoTime Series
🎯 What it does: Proposed an interactive annotation protocol to collect continuous, dense object-level situational awareness (SA) labels in a VR driving simulator, and trained a semantic segmentation model that integrates eye movement history with scene information to predict drivers' awareness states toward traffic objects.
Modeling the Real World with High-Density Visual Particle Dynamics
William F Whitney, Vikas Sindhwani (Google DeepMind)
Robotic IntelligenceTransformerNeural Radiance FieldMultimodalityPoint Cloud
🎯 What it does: Propose a high-density visual particle dynamics (HD-VPD) model that can learn and predict physical dynamics involving robot actions in real robot scenarios, supporting three-dimensional point cloud representations with over 100,000 particles.
Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor
Anish Bhattacharya (University of Pennsylvania), Davide Scaramuzza (University of Zurich)
Depth EstimationDomain AdaptationAutonomous DrivingOptimizationKnowledge DistillationRepresentation LearningConvolutional Neural NetworkRecurrent Neural NetworkSupervised Fine-TuningImage
🎯 What it does: This paper proposes a method for autonomous obstacle avoidance for quadrotors in unknown static obstacle environments using only monocular event cameras, achieving simulation-to-real world transfer.
MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking
Huaxiaoyue Wang (Cornell University), Sanjiban Choudhury (Cornell University)
Robotic IntelligenceTransformerLarge Language ModelReinforcement LearningVision Language ModelImageTextMultimodality
🎯 What it does: This paper proposes MOSAIC, a modular cooking collaboration architecture based on a multi-modal large pre-trained model, enabling two robots to interact with humans through natural language, predict human actions, and perform diverse kitchen tasks.
Multi-agent Reinforcement Learning with Hybrid Action Space for Free Gait Motion Planning of Hexapod Robots
Huiqiao Fu (Nanjing University), Chunlin Chen (Nanjing University)
Robotic IntelligenceReinforcement Learning
🎯 What it does: In an irregular plum blossom cluster environment, a free gait motion planning method is proposed, treating each leg of the hexapod robot as an independent agent. The method jointly plans gait, center of mass (COM), and foothold sequence to achieve the shortest path from a random starting point to the target area.
Multi-Strategy Deployment-Time Learning and Adaptation for Navigation under Uncertainty
Abhishek Paudel (George Mason University), Gregory J. Stein (George Mason University)
Domain AdaptationReinforcement LearningGenerative Adversarial NetworkImage
🎯 What it does: In unknown partial map environments, multi-strategy deployment is used with learning and visual domain adaptation, and offline alternative strategy replay is employed to real-time update the performance lower bounds of each strategy, enabling rapid policy selection within non-stationary policy sets.
Multi-Task Interactive Robot Fleet Learning with Visual World Models
Huihan Liu (University of Texas at Austin), Yuke Zhu (University of Texas at Austin)
Anomaly DetectionRobotic IntelligenceReinforcement Learning from Human FeedbackConvolutional Neural NetworkTransformerWorld ModelImageBenchmark
🎯 What it does: Proposed the SIRIUS-FLEET framework to achieve multi-task interactive robot fleet learning, enabling runtime monitoring and continuous improvement through visual world models and adaptive anomaly predictors.
Multi-Transmotion: Pre-trained Model for Human Motion Prediction
Yang Gao (EPFL), Alexandre Alahi (EPFL)
Pose EstimationRepresentation LearningTransformerSupervised Fine-TuningAuto EncoderMultimodalitySequential
🎯 What it does: This study integrates seven multimodal human motion datasets to construct a unified data framework and proposes the Multi-Transmotion pre-trained Transformer model, which utilizes multiple masking strategies for cross-modal prediction and is fine-tuned under multi-task scenarios.
Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping
Qianxu Wang, Leonidas Guibas
Robotic IntelligenceTransformerContrastive LearningImageMultimodalityPoint Cloud
🎯 What it does: Propose Neural Attention Field (NAF) to achieve one-shot dexterous grasping, enabling semantic-aware transfer of hand pose perception across different scenarios.
Neural Inverse Source Problem
Youngsun Wi (University of Michigan), Nima Fazeli (University of Michigan)
Robotic IntelligencePoint CloudPhysics RelatedOrdinary Differential Equation
🎯 What it does: Propose a framework for solving inverse source problems based on Physics-Informed Neural Networks (PINN), which can simultaneously infer unknown external source functions and complete states in robotic systems using partial noisy observational data;
NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors
Shuo Cheng (Georgia Institute of Technology), Danfei Xu (Georgia Institute of Technology)
OptimizationRobotic IntelligenceBenchmark
🎯 What it does: NOD-TAMP combines neural object descriptors (NOD) with task and motion planning (TAMP) frameworks to extract transferable short-term operations from a few human demonstrations and solve long-horizon, fine-grained manipulation tasks by composing these operations.
Non-rigid Relative Placement through 3D Dense Diffusion
Eric Cai (Carnegie Mellon University), David Held (Carnegie Mellon University)
Robotic IntelligenceTransformerDiffusion modelMultimodalityPoint Cloud
🎯 What it does: Proposed a non-rigid relative placement framework called TAX3D based on point cloud diffusion, which can learn to predict cross-displacement of deformable objects (e.g., fabrics), achieve multi-modal target prediction, and demonstrate excellent generalization capabilities in both simulated and real environments.
Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures
Kensuke Nakamura (Carnegie Mellon University), Andrea Bajcsy (Carnegie Mellon University)
Autonomous DrivingOptimizationTransformerAuto EncoderTime SeriesSequential
🎯 What it does: Designed a system-level metric based on regret to detect trajectory prediction failures in human-robot interaction.
Object-Centric Dexterous Manipulation from Human Motion Data
Yuanpei Chen (Stanford University), Karen Liu
Pose EstimationRobotic IntelligenceTransformerReinforcement LearningTime SeriesSequential
🎯 What it does: Train a hierarchical strategy using human hand motion capture data, enabling dual-armed robotic hands to perform operations such as grasping, moving, and rotating objects along predetermined trajectories.
OCCAM: Online Continuous Controller Adaptation with Meta-Learned Models
Hersh Sanghvi (University of Pennsylvania), Camillo Jose Taylor (University of Pennsylvania)
Robotic IntelligenceMeta LearningTabular
🎯 What it does: Proposed the OCCAM framework, which utilizes meta-learning and Bayesian recursive estimation to online adapt controller parameters, enabling rapid adaptation for different robots (racing car, quadruped robot, quadrotor) in new environments.
OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
Jinhan Li (UT Austin), Yuke Zhu (UT Austin)
Robotic IntelligenceTransformerVision Language ModelVision-Language-Action ModelVideo
🎯 What it does: Propose OKAMI, a method for humanoid robots to learn manipulation skills through single-shot RGB-D video.
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
Tairan He (Carnegie Mellon University), Guanya Shi (Shanghai Jiao Tong University)
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningVision-Language-Action ModelDiffusion modelTextMultimodalitySequential
🎯 What it does: Propose the OmniH2O system, achieving full-scale humanoid robot whole-body control and autonomous learning, supporting VR, RGB, language, and other interfaces.
One Model to Drift Them All: Physics-Informed Conditional Diffusion Model for Driving at the Limits
Franck Djeumou (Toyota Research Institute), John Subosits (Toyota Research Institute)
Autonomous DrivingDiffusion modelTime SeriesSequentialStochastic Differential Equation
🎯 What it does: Designed and verified a physics-informed conditional diffusion model to learn vehicle dynamics parameters from unlabeled datasets, enabling real-time control of autonomous vehicles under grip-limit (drift) conditions.
One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion
Nico Bohlinger, Davide Tateo (Technical University Of Darmstadt)
Robotic IntelligenceTransformerReinforcement Learning
🎯 What it does: Designed and implemented a unified robotic morphology architecture (URMA) that can simultaneously train a policy to control multiple legged robot morphologies in simulation and achieve zero-shot transfer to real robots.
Online Transfer and Adaptation of Tactile Skill: A Teleoperation Framework
Xiao Chen (Technical University of Munich), Sami Haddadin (Technical University of Munich)
Robotic Intelligence
🎯 What it does: This paper proposes a teleoperation framework that enables real-time learning and adaptation of tactile skills through a remote teacher robot.
OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation
Aadhithya Iyer (New York University), Lerrel Pinto (New York University)
Robotic IntelligenceVideo
🎯 What it does: Proposes OPEN TEACH, a multifunctional robotic teleoperation system that supports various robot arms and hands, allows mobile operations without calibration, and functions in both simulated and real environments.
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
Xuxin Cheng (UC San Diego), Xiaolong Wang (UC San Diego)
Representation LearningRobotic IntelligenceTransformerVideo
🎯 What it does: This study proposes and implements OpenTeleVision, an immersive teleoperation system based on VR devices, active head cameras, and stereo vision. It uses human demonstration data collected through this system to train imitation learning strategies, successfully achieving fine-grained long-term tasks (such as can classification, insertion, folding, and unloading) on two humanoid robots.
OpenVLA: An Open-Source Vision-Language-Action Model
Moo Jin Kim (Stanford University), Chelsea Finn (Stanford University)
Robotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningVision-Language-Action ModelMultimodality
🎯 What it does: This paper proposes OpenVLA, a 7B-parameter open-source Vision-Language-Action model that can directly achieve general manipulation on multi-robot platforms.
OrbitGrasp: SE(3)-Equivariant Grasp Learning
Boce Hu (Northeastern University), Robert Platt (Northeastern University)
Pose EstimationRobotic IntelligenceGraph Neural NetworkPoint Cloud
🎯 What it does: Developed OrbitGrasp, a grasp pose detection method based on SE(3) equivariant point cloud networks.
Perceive With Confidence: Statistical Safety Assurances for Navigation with Learning-Based Perception
Anushri Dixit (Princeton University), Anirudha Majumdar (Princeton University)
Autonomous DrivingRobotic IntelligenceTransformerPoint Cloud
🎯 What it does: Propose a framework named PWC, which combines pre-trained perception models with a safety planner through conformal prediction calibration and non-deterministic filters, providing statistical safety guarantees for closed-loop distribution shifts and false detection rates in new environments.
Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics
Jad Abou-Chakra (Queensland University of Technology), Niko Suenderhauf (Queensland University of Technology)
Robotic IntelligenceGaussian SplattingWorld ModelImagePhysics Related
🎯 What it does: Propose a Gaussian-Particle world model that simultaneously incorporates physical prediction and visual correction for robot real-time tracking and 3D reconstruction.
PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations
Cheng Qian, Jan Peters
Knowledge DistillationRobotic IntelligenceTransformerReinforcement LearningDiffusion modelAuto EncoderVideoSequential
🎯 What it does: Extract finger trajectories and keyboard states from YouTube videos and accompanying MIDI files, train RL expert strategies for each piece, and distill all expert strategies into a single universal policy capable of playing piano for any given song.
Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation
Ruoxuan Feng, Xuelong Li (Shenzhen Taobotics Co Ltd)
Explainability and InterpretabilityRobotic IntelligenceRecurrent Neural NetworkTransformerVision-Language-Action ModelImageMultimodalityAudio
🎯 What it does: Proposed a phase-guided dynamic multi-modal fusion method called MS-Bot to address multi-sensory robotic manipulation tasks such as pouring and keyhole peg insertion.
PointPatchRL - Masked Reconstruction Improves Reinforcement Learning on Point Clouds
Balazs Gyenes (Karlsruhe Institute of Technology), Gerhard Neumann (Karlsruhe Institute of Technology)
Robotic IntelligenceTransformerReinforcement LearningAuto EncoderPoint Cloud
🎯 What it does: Propose PointPatchRL, which discretizes point clouds into patches, applies Transformer encoding, and combines masked reconstruction self-supervised loss to enhance RL performance.
Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation
Vivek Myers (University of California, Berkeley), Sergey Levine (University of California, Berkeley)
Robotic IntelligenceMeta LearningConvolutional Neural NetworkLarge Language ModelMultimodality
🎯 What it does: Propose a method called PALO that utilizes a vision-language model (VLM) for language decomposition, combined with a pre-trained language-driven robot policy, achieving fast parameter-free adaptation to new tasks with only a small amount of demonstration data.
PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators
Kuo-Hao Zeng (Allen Institute for AI), Luca Weihs (Allen Institute for AI)
Robotic IntelligenceTransformerReinforcement LearningImageBenchmark
🎯 What it does: Trained a POLIFORMER agent for indoor navigation using only RGB input and a full Transformer architecture, achieving direct real-world deployment without fine-tuning after large-scale simulation training.
Pre-emptive Action Revision by Environmental Feedback for Embodied Instruction Following Agents
Jinyeon Kim (Seoul National University), Jonghyun Choi (Seoul National University)
Robotic IntelligenceReinforcement Learning from Human FeedbackLarge Language ModelVision-Language-Action ModelText
🎯 What it does: Propose a Pre-Action Revision Framework (PRED) based on environmental feedback, helping embedded task executors dynamically adjust action plans before executing commands based on perceived environmental differences.
Progressive Multi-Modal Fusion for Robust 3D Object Detection
Rohit Mohan (University of Freiburg), Abhinav Valada (University of Freiburg)
Object DetectionAutonomous DrivingConvolutional Neural NetworkTransformerImageMultimodalityPoint Cloud
🎯 What it does: Proposes a multi-modal 3D object detection framework named ProFusion3D, which progressively fuses radar and camera features in view spaces (BEV and PV) and at the query level, and introduces a self-supervised multi-modal mask modeling pre-training scheme.
Promptable Closed-loop Traffic Simulation
Shuhan Tan (University Of Texas Austin), Marco Pavone (Nvidia)
GenerationData SynthesisAutonomous DrivingTransformerLarge Language ModelPrompt EngineeringTextMultimodality
🎯 What it does: Proposed ProSim, a closed-loop traffic simulation framework controllable via multimodal prompts, enabling users to specify the behavior and intent of each traffic participant using numerical, categorical, or text prompts, and generating realistic traffic scenarios through closed-loop iterative processes;
Provably Safe Online Multi-Agent Navigation in Unknown Environments
Zhan Gao (University of Cambridge), Amanda Prorok (University of Cambridge)
Autonomous DrivingOptimizationSafty and PrivacyGraph Neural NetworkPoint Cloud
🎯 What it does: This paper proposes an online exploration-based Control Lyapunov Barrier Function (OE-CLBF) controller, which utilizes LiDAR sensors to real-time learn safety constraints in unknown environments and generates safe navigation commands through CBF-CLF-QP;
Q-SLAM: Quadric Representations for Monocular SLAM
Chensheng Peng (University Of California Berkeley), Wei Zhan (Nvidia)
Pose EstimationDepth EstimationTransformerNeural Radiance FieldSimultaneous Localization and MappingImage
🎯 What it does: Propose a dense map construction method based on quadric representation, improving depth estimation, 3D reconstruction, and joint optimization of camera pose in monocular SLAM.
RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation
Yuxuan Kuang (University of Southern California), Yue Wang (Stanford University)
Domain AdaptationRobotic IntelligenceDiffusion modelImageTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: Proposes RAM, a retrieval-based affine transfer framework, achieving generalization for zero-shot robot manipulation.
Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling
Aman Sinha (Waymo), Shimon Whiteson (Waymo)
Autonomous DrivingTabular
🎯 What it does: Proposed and implemented a framework based on Bayesian Adaptive Multi-Fidelity Sampling (BAMS) for safety rate estimation in autonomous vehicles (AVs) and active discovery of high-impact failure cases.
Real-to-Sim Grasp: Rethinking the Gap between Simulation and Real World in Grasp Detection
Jia-Feng Cai (School of Computer Science and Engineering, Sun Yat-sen University), Wei-Shi Zheng (School of Computer Science and Engineering, Sun Yat-sen University)
Object DetectionData SynthesisPose EstimationDepth EstimationDomain AdaptationData-Centric LearningRobotic IntelligenceConvolutional Neural NetworkVision-Language-Action ModelImageBenchmark
🎯 What it does: Proposes the R2SGrasp framework, achieving high-precision 6-DoF object grasping by repairing or enhancing real-world data and features to align with simulated data during the inference phase.
Reasoning Grasping via Multimodal Large Language Model
Shiyu Jin (Baidu Research), Liangjun Zhang (Baidu Research)
Pose EstimationRobotic IntelligenceConvolutional Neural NetworkTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringVision Language ModelImageTextMultimodality
🎯 What it does: Propose the 'reasoning grasping' task, combining multi-modal large language models with visual grasping frameworks to enable robots to generate grasping poses under implicit instructions.
Region-aware Grasp Framework with Normalized Grasp Space for Efficient 6-DoF Grasping
Siang Chen (Tsinghua University), Guijin Wang (Tsinghua University)
Pose EstimationComputational EfficiencyRobotic IntelligenceConvolutional Neural NetworkImage
🎯 What it does: Proposed a region-based grasping framework and a normalized grasping space (NGS), and designed an efficient region-normalized grasping network (RNGNet) to achieve real-time 6-DoF grasping detection.
Reinforcement Learning with Foundation Priors: Let Embodied Agent Efficiently Learn on Its Own
Weirui Ye (Tsinghua University), Yang Gao (Tsinghua University)
Robotic IntelligenceTransformerReinforcement LearningPrompt EngineeringVision Language ModelDiffusion modelImage
🎯 What it does: Propose a reinforcement learning framework based on foundation model priors, RLFP, and implement the Foundation-guided Actor-Critic (FAC) algorithm, enabling robots to complete multiple manipulation tasks within one hour of training.
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
Wenlong Huang, Li Fei-Fei (Stanford University)
OptimizationRobotic IntelligenceTransformerLarge Language ModelVision Language ModelImagePoint Cloud
🎯 What it does: Propose a framework that describes robotic grasping and manipulation tasks through visualization constraint functions (ReKep), automatically generating constraints using large visual models and vision-language models, and achieving real-time closed-loop control via hierarchical optimization.
ReMix: Optimizing Data Mixtures for Large Scale Imitation Learning
Joey Hejna (Stanford University), Dorsa Sadigh
OptimizationData-Centric LearningRobotic IntelligenceReinforcement LearningSequential
🎯 What it does: This work proposes the Re-Mix method, which automatically determines the mixing weights of robot datasets using distributed robust optimization, thereby enhancing model generalization in large-scale imitation learning pre-training.
ResPilot: Teleoperated Finger Gaiting via Gaussian Process Residual Learning
Patrick Naughton (University of Illinois Urbana-Champaign), Soshi Iba (Honda Research Institute USA)
OptimizationRobotic IntelligenceTabular
🎯 What it does: Studied a finger gait teleoperation method based on residual Gaussian process learning, which can map human hand postures to multi-finger robotic hands, achieving a larger reachable workspace.
RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation
Chongkai Gao (National University Of Singapore), Huazhe Xu (Tsinghua University)
Robotic IntelligenceTransformerVision-Language-Action ModelPoint Cloud
🎯 What it does: Proposed the RiEMann framework to achieve end-to-end, near-real-time SE(3) equivariant robotic manipulation learning without requiring point cloud segmentation.
RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Hanxiao Jiang (Columbia University), Yunzhu Li (Columbia University)
Robotic IntelligenceGraph Neural NetworkTransformerLarge Language ModelVision-Language-Action ModelImageMultimodality
🎯 What it does: Propose the RoboEXP framework, enabling robots to build action-conditioned 3D scene graphs (ACSG) through active interaction exploration, and utilize this graph to complete complex object recognition and manipulation tasks.
RoboKoop: Efficient Control Conditioned Representations from Visual Input in Robotics using Koopman Operator
Hemant Kumawat (Georgia Institute of Technology), Saibal Mukhopadhyay (Georgia Institute of Technology)
Robotic IntelligenceReinforcement LearningContrastive LearningImage
🎯 What it does: A visual representation learning framework named RoboKoop, combining contrastive learning with Koopman operations, is studied for learning linearized control representations from pixel inputs and optimizing control strategies using SAC.
RoboPoint: A Vision-Language Model for Spatial Affordance Prediction in Robotics
Wentao Yuan (University of Washington Allen Institute for Artifical Intelligence), Dieter Fox (University of Washington Allen Institute for Artifical Intelligence)
Robotic IntelligenceTransformerSupervised Fine-TuningVision Language ModelImageText
🎯 What it does: Proposed and implemented ROBOPOINT, a vision-language model that predicts spatial affordance points in images based on language instructions, and applied it to tasks such as robotic manipulation, navigation, and augmented reality.
Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction
Justin Kerr (University Of California Berkeley), Angjoo Kanazawa (University Of California Berkeley)
Pose EstimationDepth EstimationRobotic IntelligenceGaussian SplattingImageVideoPoint Cloud
🎯 What it does: This paper proposes a system named Robot See Robot Do (RSRD), which enables motion learning and robot simulation for articulated objects through a single multi-view static object scan and a monocular RGB human demonstration video;
Robotic Control via Embodied Chain-of-Thought Reasoning
Michał Zawalski (University Of Warsaw), Sergey Levine (Stanford University)
Robotic IntelligenceLarge Language ModelVision Language ModelVision-Language-Action ModelMultimodalityChain-of-Thought
🎯 What it does: Propose Embodied Chain-of-Thought (ECoT), training vision-language-action (VLA) models to perform multi-step text reasoning before executing actions, encompassing task rephrasing, planning, subtask judgment, low-level motion commands, and spatial descriptions of visual perception and robotic arm states.
RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards
Fatemeh Zargarbashi (ETH Zürich), Stelian Coros (ETH Zürich)
Robotic IntelligenceTransformerReinforcement LearningVideoSequential
🎯 What it does: This work proposes a reinforcement learning-based keyframe control framework that enables quadruped robots to precisely reach specified partial or full pose targets within a predetermined time while maintaining natural motion styles;
Robust Manipulation Primitive Learning via Domain Contraction
Teng Xue (Idiap Research Institute), Sylvain Calinon (Idiap Research Institute)
Domain AdaptationOptimizationRobotic IntelligenceReinforcement LearningImageTabular
🎯 What it does: Propose a dual-layer approach that combines parameter-enhanced multi-model reinforcement learning with domain contraction techniques to learn robust operational primitives against uncertain physical parameters.
RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learning
Lawrence Yunliang Chen (University Of California Berkeley), Ken Goldberg (University Of California Berkeley)
Domain AdaptationRobotic IntelligenceTransformerDiffusion modelImageSequential
🎯 What it does: Propose RoVi-Aug, which automatically enhances robot appearance and perspective through diffusion models to improve the generalizability of cross-robot learning and zero-shot deployment capabilities.
RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands
Yi Zhao (Aalto University), Dieter Büchler
Data SynthesisRobotic IntelligenceConvolutional Neural NetworkTransformerReinforcement LearningDiffusion modelTextTime SeriesBenchmark
🎯 What it does: Constructed a large-scale robot piano motion dataset RP1M, containing approximately 2k songs and 1M expert trajectories, and proposed an unlabeled training method using optimal transport to automatically generate fingerings.