CoRL 2024 Papers with AI Summaries
Conference on Robot Learning · 264 papers
→ CoRL 2024 papers with code (32)
Each paper below shows an AI-generated one-line summary. Get the full 6-part summary (innovation, method, data, results, limitations) and search all 264 CoRL 2024 papers by keyword, author or institution —
free trial on arXivSub.
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
Tsung-Wei Ke (Carnegie Mellon University), Katerina Fragkiadaki (Carnegie Mellon University)
Robotic IntelligenceTransformerVision-Language-Action ModelDiffusion modelImageMultimodality
🎯 What it does: Proposes the 3D Diffuser Actor, a robotic control policy that integrates 3D scene representation with diffusion models, enabling the learning of multimodal action distributions from demonstrations.
3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing
Binghao Huang (Columbia University), Yunzhu Li (Columbia University)
Robotic IntelligenceDiffusion modelMultimodalityPoint Cloud
🎯 What it does: A 3D-ViTac system integrating a dense flexible tactile sensor array with visual perception was constructed, and a unified three-dimensional visuo-tactile synthetic representation was developed based on this system for discretized learning to accomplish multi-arm fine-grained manipulation.
A Dual Approach to Imitation Learning from Observations with Offline Datasets
Harshit Sikchi (University of Texas at Austin), Scott Niekum (University of Massachusetts Amherst)
Reinforcement LearningImageMultimodalityTabular
🎯 What it does: Propose a imitation learning method DILO that does not require expert actions and only utilizes observed trajectories and offline data, directly learning a multi-step utility function to achieve access distribution matching.
A Planar-Symmetric SO(3) Representation for Learning Grasp Detection
Tianyi Ko (Woven by Toyota), Koichi Nishiwaki (Woven by Toyota)
Pose EstimationRobotic IntelligenceConvolutional Neural NetworkMesh
🎯 What it does: Designed and implemented a planar symmetric parallel gripper SO(3) rotation representation using 2D Bingham distribution, integrated into a 3D convolutional grasping detection network, addressing issues of rotational discontinuity and multimodal inconsistency in traditional regression.
A3VLM: Actionable Articulation-Aware Vision Language Model
Siyuan Huang (Shanghai Jiao Tong University), Hongsheng Li (Chinese University of Hong Kong)
Object DetectionGenerationPose EstimationRobotic IntelligenceTransformerVision Language ModelVision-Language-Action ModelDiffusion modelContrastive LearningImageText
🎯 What it does: Designed and trained an object-centric, operable, and joint-aware model called A3VLM based on a vision-language model, which identifies movable parts and their joint structures using a single RGB image, and outputs operable representations directly mappable to robotic execution;
Accelerating Visual Sparse-Reward Learning with Latent Nearest-Demonstration-Guided Explorations
Ruihan Zhao (University of Texas at Austin), Mariano Phielipp (Intel AI Lab)
Robotic IntelligenceTransformerReinforcement LearningAuto EncoderContrastive LearningWorld ModelImage
🎯 What it does: The study trains robotic manipulation tasks using a small number of demonstrations and visual sparse rewards, proposing the LaNE method.
ACE: A Cross-platform and visual-Exoskeletons System for Low-Cost Dexterous Teleoperation
Shiqi Yang (UC San Diego), Xiaolong Wang (UC San Diego)
Pose EstimationRobotic IntelligenceImage
🎯 What it does: Developed a cross-platform, low-cost visual exoskeleton system named ACE, enabling remote control for fine manipulation across multiple robot platforms.
Action Space Design in Reinforcement Learning for Robot Motor Skills
Julian Eßer (Fraunhofer Institute for Material Flow and Logistics), Pulkit Agrawal (Fraunhofer Institute for Material Flow and Logistics)
Robotic IntelligenceReinforcement Learning
🎯 What it does: This study systematically evaluates and compares the impact of different configuration spaces and action spaces (such as joint position, velocity, torque, etc.) on the performance of robot reinforcement learning, covering bipedal and quadrupedal robots as well as a set of simulation tasks, and transfers the learned policies to real robots.
Adapting Humanoid Locomotion over Challenging Terrain via Two-Phase Training
Wenhao Cui (Noetix Robotics), Zheyuan Jiang (Noetix Robotics)
Robotic IntelligenceReinforcement LearningAuto Encoder
🎯 What it does: This paper proposes a two-stage simulation-to-real transfer training framework, enabling humanoid robots to achieve robust walking on various challenging terrains and successfully deploying the learned policy on real robots.
Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigation
Youwei Yu (Indiana University), Lantao Liu (Indiana University)
GenerationData SynthesisAutonomous DrivingReinforcement LearningDiffusion modelImage
🎯 What it does: This paper proposes an Adaptive Terrain Generator (ADTG) based on a denoising diffusion model, which generates challenging irregular terrains that match the current performance of RL policies by dynamically optimizing the initial noise.
Adaptive Language-Guided Abstraction from Contrastive Explanations
Andi Peng, Andreea Bobu
Explainability and InterpretabilityReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningContrastive LearningText
🎯 What it does: Proposed an iterative reward learning framework named ALGAE, which combines language models to identify missing semantic features in demonstrations and automatically generates corresponding code, then verifies and updates the reward using inverse reinforcement learning.
ALOHA Unleashed: A Simple Recipe for Robot Dexterity
Tony Z. Zhao (Google DeepMind), Ayzaan Wahid (Google DeepMind)
Robotic IntelligenceReinforcement Learning from Human FeedbackConvolutional Neural NetworkTransformerDiffusion modelVideoSequential
🎯 What it does: Collected over 26,000 human teleoperation demonstrations on the ALOHA 2 platform, and trained an end-to-end robot policy using a Transformer-Encoder-Decoder architecture combined with diffusion loss; achieved significant success in 5 real-world dual-arm flexible manipulation tasks (e.g., tying shoelaces, hanging a shirt) and 3 simulation tasks, outperforming existing baselines;
An Open-Source Soft Robotic Platform for Autonomous Aerial Manipulation in the Wild
Erik Bauer (ETH Zurich), Robert K. Katzschmann (ETH Zurich)
Robotic IntelligenceSimultaneous Localization and MappingImageVideoPoint Cloud
🎯 What it does: Built and validated a fully autonomous soft aerial manipulator platform based on onboard perception, capable of autonomously locating, searching, and grasping various unknown objects in indoor and outdoor environments.
ANAVI: Audio Noise Awareness using Visual of Indoor environments for NAVIgation
Vidhi Jain, Yonatan Bisk
Data SynthesisRobotic IntelligenceConvolutional Neural NetworkImagePoint CloudAudio
🎯 What it does: This study proposes a vision-based robot noise perception framework called ANAVI, which can predict the maximum sound pressure level that a listener would experience at different indoor locations and plan quieter paths accordingly.
AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch
Max Yang (University of Bristol), Nathan F. Lepora (University of Bristol)
Domain AdaptationKnowledge DistillationRobotic IntelligenceConvolutional Neural NetworkReinforcement LearningMultimodality
🎯 What it does: Designed and implemented a system called AnyRotate, capable of achieving multi-axis, gravity-invariant hand-internal object rotation through dense tactile perception under any hand orientation.
APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMs
Huaxiaoyue Wang (Cornell University), Sanjiban Choudhury (Stanford University)
Robotic IntelligenceReinforcement Learning from Human FeedbackLarge Language ModelReinforcement LearningVision Language ModelWorld ModelTextMultimodality
🎯 What it does: Propose the APRICOT method, which utilizes visual demonstrations and minimal user interaction to learn users' natural language preferences for household organization tasks (e.g., fridge arrangement) and generates robot execution plans that satisfy both preferences and physical constraints.
Automated Creation of Digital Cousins for Robust Policy Learning
Tianyuan Dai (Stanford University), Li Fei-Fei (Stanford University)
SegmentationGenerationData SynthesisDepth EstimationDomain AdaptationRobotic IntelligenceTransformerLarge Language ModelVision Language ModelImageMultimodality
🎯 What it does: Proposed the concept of 'Digital Cousin' and designed the ACDC fully automated pipeline, which can generate interactive simulation scenes from a single RGB image and train robot strategies in these scenes to achieve zero-shot sim-to-real transfer.
Autonomous Improvement of Instruction Following Skills via Foundation Models
Zhiyuan Zhou (University Of California Berkeley), Sergey Levine (University Of California Berkeley)
Robotic IntelligenceVision Language ModelDiffusion modelImageTextMultimodality
🎯 What it does: Propose the SOAR system, enabling robots to autonomously collect data and self-improve their instruction-following strategies without human intervention through internet-based pre-trained models.
Autonomous Interactive Correction MLLM for Robust Robotic Manipulation
Chuyan Xiong (Peking University), Hao Dong (Peking University)
Robotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodalityChain-of-Thought
🎯 What it does: Constructed the AIC MLLM framework, leveraging multimodal large language models to autonomously correct low-level SE(3) pose prediction in articulated object manipulation based on failure interaction experiences.
Avoid Everything: Model-Free Collision Avoidance with Expert-Guided Fine-Tuning
Adam Fishman (University of Washington), Dieter Fox (NVIDIA Inc)
OptimizationRobotic IntelligenceTransformerSupervised Fine-TuningPoint Cloud
🎯 What it does: Proposes an end-to-end collision avoidance system called 'Avoid Everything,' which generates safe robotic arm motions using point cloud inputs.
Bi-Level Motion Imitation for Humanoid Robots
Wenshuai Zhao (Aalto University), Michael Muehlebach (Max Planck Institute For Intelligent Systems)
Robotic IntelligenceReinforcement LearningAuto EncoderTime SeriesSequential
🎯 What it does: This paper proposes a bi-level optimization framework (Bi-Level Motion Imitation, BMI), which alternates optimization between the robot's policy and the reference motion generator, enabling the robot to learn human MoCap data with physically feasible reference trajectories that preserve the original motion patterns.
BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark
Nikita Chernyadev (Dyson Robot Learning Lab), Stephen James (Dyson Robot Learning Lab)
Robotic IntelligenceReinforcement LearningDiffusion modelImageMultimodalityBenchmark
🎯 What it does: Proposed the BiGym benchmark, which includes 40 mobile bimanual manipulation tasks and provides real human demonstrations for each task.
Bimanual Dexterity for Complex Tasks
Kenneth Shaw (Carnegie Mellon University), Deepak Pathak (Carnegie Mellon University)
Robotic IntelligenceTransformerVideoSequential
🎯 What it does: This paper proposes and implements BiDex, a low-cost, low-latency, portable dual-arm bimanual teleoperation system for performing high-degree-of-freedom bimanual grasping and manipulation in desktop and mobile environments, and collects expert demonstration data through the system to train a behavior cloning policy.
Body Transformer: Leveraging Robot Embodiment for Policy Learning
Carmelo Sferrazza (University of California Berkeley), Pieter Abbeel (University of California Berkeley)
Computational EfficiencyRobotic IntelligenceTransformerReinforcement LearningGraph
🎯 What it does: Propose a Transformer architecture based on the robot's body structure (Body Transformer), which restricts information flow through masking in the attention layer, allowing each sensor/actuator node to aggregate only its own and direct neighbors' features, thereby enhancing the performance of robot policy learning (imitation learning and reinforcement learning).
Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight
Jiaxu Xing (University of Zurich), Davide Scaramuzza (University of Zurich)
Robotic IntelligenceConvolutional Neural NetworkReinforcement LearningImage
🎯 What it does: A three-stage learning framework is used to train quadrotor drone racing control strategies using only monocular RGB or door corner visual inputs, ultimately achieving high-speed and precise track navigation.
Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learning
Piotr Kicki (IDEAS NCBR), Krzysztof Walas (IDEAS NCBR)
Robotic IntelligenceReinforcement Learning
🎯 What it does: This paper proposes a framework called CNP3O, which integrates learning to plan, motion primitives (MP), and safe reinforcement learning (SafeRL). It uses B-spline MP to generate trajectories that satisfy safety constraints under known constraints, and verifies them in two tasks: robot heavy object transportation and air hockey striking.
Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
Haoran He (Shanghai Jiao Tong University), Weinan Zhang (Shanghai Jiao Tong University)
Domain AdaptationKnowledge DistillationConvolutional Neural NetworkReinforcement LearningContrastive LearningSequential
🎯 What it does: Propose a Historical Information Bottleneck (HIB) method based on the Information Bottleneck principle, which compresses historical trajectories to extract hidden knowledge, achieving single-stage knowledge distillation during simulation-to-real conversion and enhancing the generalization performance of local policies.
Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision
Alberta Longhini (Kth Royal Institute Of Technology), Danica Kragic (Kth Royal Institute Of Technology)
Pose EstimationGraph Neural NetworkGaussian SplattingImage
🎯 What it does: Propose Cloth-Splatting, a prediction-update framework for 3D fabric state estimation using RGB images;
ClutterGen: A Cluttered Scene Generator for Robot Learning
Yinsen Jia (Duke University), Boyuan Chen (Duke University)
GenerationRobotic IntelligenceConvolutional Neural NetworkReinforcement LearningPoint Cloud
🎯 What it does: Developed an autoregressive simulation scene generator called ClutterGen, capable of generating physically feasible, cluttered, and diverse object layouts in robot learning, supporting stable placement of up to ten objects on restricted tables.
Conformal Prediction for Semantically-Aware Autonomous Perception in Urban Environments
Achref Doula (Technical University of Darmstadt), Alejandro Sanchez Guinea
Autonomous Driving
🎯 What it does: Introduce Knowledge-Refined Prediction Sets (KRPS) to achieve multi-task semantic consistent confidence set construction.
Context-Aware Replanning with Pre-Explored Semantic Map for Object Navigation
Po-Chen Ko (National Taiwan University), Winston H. Hsu (National Taiwan University)
Autonomous DrivingVision Language ModelContrastive LearningImagePoint Cloud
🎯 What it does: This paper proposes the Context-Aware Replanning (CARe) method, which utilizes uncertainty and multi-view consistency metrics in a pre-explored semantic map to replan navigation without requiring additional labels.
Continuous Control with Coarse-to-fine Reinforcement Learning
Younggyo Seo (Dyson Robot Learning Lab), Stephen James (Dyson Robot Learning Lab)
Reinforcement Learning
🎯 What it does: Proposes the Coarse-to-fine Reinforcement Learning (CRL) framework and its implementation, the Coarse-to-fine Q-Network (CQN) algorithm, achieving high-precision control and sample efficiency in continuous control tasks through multi-level coarse-to-fine action discretization.
Continuously Improving Mobile Manipulation with Autonomous Real-World RL
Russell Mendonca (Carnegie Mellon University), Deepak Pathak (Carnegie Mellon University)
Robotic IntelligenceReinforcement LearningImagePoint Cloud
🎯 What it does: By enabling the Spot legged mobile manipulator to continuously learn and improve four mobility and manipulation tasks in the real world using autonomous reinforcement learning (RL), achieving an average success rate of about 80%.
Contrast Sets for Evaluating Language-Guided Robot Policies
Abrar Anwar (University of Southern California), Jesse Thomason (University of Southern California)
Robotic IntelligenceImageTextMultimodality
🎯 What it does: Proposed and verified a method for efficiently evaluating language-driven robot strategies using contrast sets, achieving low-cost, diverse testing in simulation manipulation and physical navigation tasks;
Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation
Teli Ma (Hong Kong University of Science and Technology), Junwei Liang (Hong Kong University of Science and Technology)
Robotic IntelligenceTransformerReinforcement LearningVision-Language-Action ModelContrastive LearningImageTextMultimodality
🎯 What it does: Propose an end-to-end Contrastive Imitation Learning (CL) framework applied to language-guided multi-task 3D robotic manipulation, and implement Sigma-agent;
Control with Patterns: A D-learning Method
Quan Quan (Beihang University), Chenyu Wang (Beihang University)
Autonomous DrivingReinforcement LearningImage
🎯 What it does: In the image visual servo control of multi-rotor drones, a dataset-based control mode (CWP) and D-learning method are proposed, which can achieve closed-loop stable control by learning the Lyapunov function and its derivative without requiring system dynamic knowledge.
CoViS-Net: A Cooperative Visual Spatial Foundation Model for Multi-Robot Applications
Jan Blumenkamp (University of Cambridge), Amanda Prorok (University of Cambridge)
Pose EstimationRobotic IntelligenceGraph Neural NetworkTransformerContrastive LearningSimultaneous Localization and MappingImage
🎯 What it does: Developed CoViS-Net, a decentralized, real-time, monocular vision-based multi-robot visual spatial foundation model that can estimate relative pose and predict bird's-eye-view (BEV) representations even in the absence of line-of-sight intersections;
CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning
Luke Rowe (Mila), Christopher Pal (Mila)
Autonomous DrivingTransformerReinforcement LearningSequential
🎯 What it does: Proposed the CtRL-Sim framework, utilizing return-conditioned offline reinforcement learning in a physics-enhanced Nocturne environment to generate reactive, controllable multi-agent driving behaviors;
D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement
Yixuan Wang (Columbia University), Yunzhu Li (University of Illinois, Urbana-Champaign)
Robotic IntelligenceGraph Neural NetworkVision Language ModelNeural Radiance FieldImagePoint Cloud
🎯 What it does: Propose D3 Fields, a training-free, zero-shot implicit 3D descriptor field for accomplishing various robotic rearrangement tasks.
D$^3$RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation
Songlin Wei (Peking University), He Wang (Peking University)
Depth EstimationRobotic IntelligenceDiffusion modelImagePoint Cloud
🎯 What it does: Propose a stereo depth estimation framework D3RoMa based on diffusion models, which can accurately recover depth on transparent or mirror surfaces.
DeliGrasp: Inferring Object Properties with LLMs for Adaptive Grasp Policies
William Xie (University of Colorado at Boulder), Nikolaus Correll (University of Colorado at Boulder)
Robotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringImageTextBenchmarkChain-of-Thought
🎯 What it does: Leverages large language models to infer mass, friction coefficients, and spring constants from object semantic descriptions, and generates executable adaptive grasping control strategies based on these physical properties.
Detect Everything with Few Examples
Xinyu Zhang (Rutgers University), Abdeslam Boularias (Rutgers University)
Object DetectionTransformerContrastive LearningImage
🎯 What it does: Proposed a zero-shot few-shot object detection framework DE-ViT without fine-tuning, achieving precise localization through region propagation and a learnable spatial integration layer, using pre-trained ViT features for subspace projection to reduce the performance gap between base classes and new classes.
DexCatch: Learning to Catch Arbitrary Objects with Dexterous Hands
Fengbo Lan (Tsinghua University), Tao Zhang (Tsinghua University)
Robotic IntelligenceReinforcement LearningPoint Cloud
🎯 What it does: Propose an LTC framework based on model-agnostic reinforcement learning, training simulations to handle throwing and catching objects of arbitrary shapes and mass distributions, achieving robust lateral palm grasping performance.
DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes
Jialiang Zhang (Peking University), He Wang (Peking University)
GenerationData SynthesisPose EstimationDomain AdaptationConvolutional Neural NetworkDiffusion modelPoint CloudBenchmark
🎯 What it does: This paper proposes DexGraspNet 2.0, a large-scale synthetic dense scene grasping benchmark, and designs a two-stage generative grasping method that utilizes a local feature conditional diffusion model to achieve grasping pose prediction.
DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics
Tyler Ga Wei Lum (Stanford University), Karl Van Wyk (NVIDIA)
Domain AdaptationKnowledge DistillationRepresentation LearningRobotic IntelligenceReinforcement LearningImageMultimodality
🎯 What it does: Trained and deployed a depth-image-based pixel-to-action geometric fabric guidance policy, DextrAH-G, which can continuously, safely, and rapidly grasp and transport various objects on real robots.
Differentiable Discrete Elastic Rods for Real-Time Modeling of Deformable Linear Objects
Yizhou Chen (University of Michigan), Ram Vasudevan (University of Michigan)
OptimizationComputational EfficiencyVideoTime SeriesPhysics Related
🎯 What it does: Propose the DEFORM framework, combining differentiable discrete elastic rod (DDER) with residual learning to achieve real-time, long-term prediction for deformable linear objects (DLOs);
Differentiable Robot Rendering
Ruoshi Liu (Columbia University), Carl Vondrick (Columbia University)
OptimizationRobotic IntelligenceGaussian SplattingImageTextPoint Cloud
🎯 What it does: Propose a differentiable robot rendering framework called Dr. Robot, which directly maps a robot's visual appearance to control parameters, enabling control and planning of robots through gradients from a visual model.
DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasets
Xiaoyu Huang (University Of California Berkeley), Koushil Sreenath (University Of California Berkeley)
Robotic IntelligenceTransformerDiffusion modelMultimodality
🎯 What it does: The paper proposes a framework named DiffuseLoco, which uses diffusion models to learn multi-skill leg control policies from offline multi-modal datasets and performs real-time inference on real robots.
DiffusionSeeder: Seeding Motion Optimization with Diffusion for Rapid Motion Planning
Huang Huang (UC Berkeley), Dieter Fox (Nvidia)
OptimizationRobotic IntelligenceTransformerDiffusion modelImage
🎯 What it does: Designed and implemented DiffusionSeeder, a diffusion model-based initial trajectory generator for rapidly generating multi-modal trajectories to initiate motion optimization.
Discovering Robotic Interaction Modes with Discrete Representation Learning
Liquan Wang (Georgia Institute of Technology), Animesh Garg (Georgia Institute of Technology)
Representation LearningRobotic IntelligenceTransformerAuto EncoderMultimodality
🎯 What it does: This paper proposes a self-supervised learning framework called ActAIM2, which discovers and discretizes interaction patterns in robotic manipulation, and predicts low-level actions through a multi-view Transformer.
Distribution Discrepancy and Feature Heterogeneity for Active 3D Object Detection
Huang-Yu Chen (National Taiwan University), Winston H. Hsu (National Taiwan University)
Object DetectionAutonomous DrivingPoint Cloud
🎯 What it does: Proposed a DDFH active learning framework based on distribution differences and feature heterogeneity for efficient annotation in LiDAR 3D object detection.
Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing
Jonathan DeCastro (Toyota Research Institute), Guy Rosman (Toyota Research Institute)
Autonomous DrivingReinforcement LearningWorld ModelTime Series
🎯 What it does: In high-speed racing, the DREAM2ASSIST framework is proposed, which uses a recursive state space model to infer human drivers' intentions and value functions, and provides assistance to humans in shared control based on the inference results.
Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
Junbang Liang (Columbia University), Carl Vondrick (Columbia University)
Object TrackingRobotic IntelligenceSupervised Fine-TuningDiffusion modelVideo
🎯 What it does: Fine-tune a pre-trained video diffusion model using human demonstration videos to generate stereo videos of humans using trackable tools to complete tasks; subsequently perform 3D tracking of tools in the video, directly converting the obtained trajectory into robot actions in a real environment, achieving visual motion policy learning.
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
Xiaoyu Tian (Tsinghua University), Hang Zhao (Tsinghua University)
Autonomous DrivingComputational EfficiencyTransformerVision Language ModelImagePoint CloudChain-of-Thought
🎯 What it does: Propose the DriveVLM system, which leverages Vision-Language Models (VLMs) through Chain-of-Thought (CoT) reasoning to accomplish scene description, scene analysis, and hierarchical planning. Furthermore, the DriveVLM-Dual dual-stream architecture is designed to integrate VLM with traditional 3D perception and trajectory planning, achieving a real-time, deployable autonomous driving solution.
Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling
Mingtong Zhang (University of Illinois Urbana-Champaign), Yunzhu Li (Columbia University)
Robotic IntelligenceGraph Neural NetworkGaussian SplattingVideo
🎯 What it does: Dense 3D tracking using dynamic 3D Gaussian splats from multi-view RGB-D videos, and constructing a graph neural network (GNN) to learn object dynamics models, enabling 3D video prediction and model-driven planning based on robotic actions.
Dynamics-Guided Diffusion Model for Sensor-less Robot Manipulator Design
Xiaomeng Xu (Stanford University), Shuran Song (Stanford University)
Robotic IntelligenceDiffusion model
🎯 What it does: Propose a sensorless robotic finger design framework based on dynamics-guided diffusion models, capable of rapidly generating task-compliant finger geometries without requiring task-specific training.
Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation
Zheyu Zhuang (Kth Royal Institute Of Technology), Danica Kragic (Kth Royal Institute Of Technology)
Domain AdaptationRobotic IntelligenceRecurrent Neural NetworkDiffusion modelImage
🎯 What it does: Developed a pixel-level dynamic hybrid visual augmentation method called RoboSaGA based on task-related saliency, enhancing the robustness of visual behavior cloning under visual domain shift scenarios (lighting, shadows, occlusions, background).
Environment Curriculum Generation via Large Language Models
William Liang (University of Pennsylvania), Yecheng Jason Ma (University of Pennsylvania)
Robotic IntelligenceTransformerLarge Language ModelReinforcement LearningAgentic AIPrompt EngineeringText
🎯 What it does: Using large language models (LLM) to automatically generate and evolve procedural environment curricula, training quadruped robots to complete diverse obstacle parkour tasks.
EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning
Jingyun Yang (Stanford University), Jeannette Bohg (Stanford University)
Robotic IntelligenceDiffusion modelPoint Cloud
🎯 What it does: Proposes EquiBot, a diffusion strategy based on SIM(3) equivariance, for learning robot manipulation policies that generalize across different scales, poses, and positions from minimal human demonstrations.
EquiGraspFlow: SE(3)-Equivariant 6-DoF Grasp Pose Generative Flows
Byeongdo Lim (Seoul National University), Frank C. Park (Seoul National University)
GenerationPose EstimationRobotic IntelligenceFlow-based ModelPoint Cloud
🎯 What it does: Propose a SE(3)-equivariant 6-DoF grasping pose generation model called EquiGraspFlow based on Continuous Normalizing Flows (CNF), which can directly generate diverse grasping poses that conform to object rotations and translations from point clouds.
Equivariant Diffusion Policy
Dian Wang (Northeastern University), Robert Platt (Northeastern University)
Robotic IntelligenceConvolutional Neural NetworkDiffusion modelImageSequential
🎯 What it does: Propose Equivariant Diffusion Policy, achieving visuomotor control behavior cloning by introducing SO(2) equivariant symmetry in the diffusion process.
EscIRL: Evolving Self-Contrastive IRL for Trajectory Prediction in Autonomous Driving
Siyue Wang (CIDI Lab), Albert Sibo Hu (CIDI Lab)
Autonomous DrivingReinforcement LearningContrastive LearningTime SeriesSequential
🎯 What it does: Propose a trajectory prediction framework based on evolutionary self-contrastive inverse reinforcement learning (ESCIRL), which can learn a general and robust reward function under the condition of only a few mixed scenario examples;
Evaluating Real-World Robot Manipulation Policies in Simulation
Xuanlin Li (UC San Diego), Ted Xiao (Google Deepmind)
Robotic IntelligenceImageBenchmark
🎯 What it does: Built a simulation evaluation environment named SIMPLER to assess general robot grasping and manipulation strategies trained in the real world within simulations, and compare the results with real-world evaluations.
Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion
Tianyi Xiong (University of Maryland), Christopher Metzler (University of Maryland)
Robotic IntelligenceGaussian Splatting
🎯 What it does: Proposes Event3DGS, an efficient 3D reconstruction framework based on event cameras;
Exploring Under Constraints with Model-Based Actor-Critic and Safety Filters
Ahmed Agha (Volkswagen Group Germany), Justin Bayer (Volkswagen Group Germany)
Robotic IntelligenceReinforcement LearningBenchmark
🎯 What it does: Integrate model-driven constrained reinforcement learning with planning-based safety filters to enhance robot safety during exploration phases
EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
Jesse Zhang (University of Southern California), Rasool Fakoor (Amazon Web Services)
Robotic IntelligenceTransformerReinforcement LearningVision Language ModelAuto EncoderMultimodalitySequential
🎯 What it does: This paper proposes the EXTRACT framework, which leverages pre-trained vision-language models to unsupervisedly extract discrete and parameterizable skills from offline trajectories, and trains a decoder to map skill IDs and continuous parameters to variable-length action sequences; subsequently, fast transfer is achieved on target tasks using reinforcement learning based on these skills;
FetchBench: A Simulation Benchmark for Robot Fetching
Beining Han (Princeton University), Jia Deng (Princeton University)
Robotic IntelligenceTransformerPoint CloudBenchmark
🎯 What it does: Propose the FetchBench simulation benchmark, construct diverse procedural grasping scenarios, and evaluate perception-planning-execution pipelines and end-to-end imitation learning methods on this benchmark.
Fleet Supervisor Allocation: A Submodular Maximization Approach
Oguzhan Akcin (University of Texas at Austin), Sandeep P. Chinchali (University of Texas at Austin)
OptimizationRobotic IntelligenceReinforcement LearningTabular
🎯 What it does: Propose two multi-robot supervised allocation strategies, ASA and n-ASA, based on stochastic submodular optimization, addressing the issues of limited human supervision resources and network uncertainty.
Flow as the Cross-domain Manipulation Interface
Mengda Xu (Stanford University), Shuran Song (Stanford University)
Domain AdaptationRobotic IntelligenceTransformerReinforcement LearningDiffusion modelOptical FlowVideo
🎯 What it does: Proposed the Im2Flow2Act framework, which leverages object flow as a cross-domain interface, enabling robots to learn real-world manipulation skills from human videos and simulated data without requiring real-world training data.
FlowBotHD: History-Aware Diffuser Handling Ambiguities in Articulated Objects Manipulation
Yishu Li (Carnegie Mellon University), David Held (Carnegie Mellon University)
Robotic IntelligenceTransformerDiffusion modelMultimodalityPoint Cloud
🎯 What it does: Propose FlowBotHD, a history-aware diffusion network, to address multi-modal ambiguity caused by visual blur and occlusion in art object manipulation.
FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learning
Li-Heng Lin (Stanford University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceMeta LearningAuto EncoderOptical FlowVideo
🎯 What it does: This paper proposes a motion similarity retrieval method called FLOWRETRIEVAL based on optical flow, for few-shot imitation learning.
FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversariality
Keyu Chen (Tsinghua University), Sifa Zheng (University of Hong Kong)
GenerationAutonomous DrivingReinforcement LearningSequentialBenchmark
🎯 What it does: Propose a safety-critical scenario generation method FREA based on the Largest Feasible Region (LFR), using the feasibility of CBV to guide adversarial strategies, generating scenarios that are both adversarial and safely traversable by AVs.
Gaitor: Learning a Unified Representation Across Gaits for Real-World Quadruped Locomotion
Alexander Luis Mitchell, Ingmar Posner (University of Oxford)
Representation LearningRobotic IntelligenceAuto EncoderTabularTime Series
🎯 What it does: Learn and leverage a unified, interpretable two-dimensional latent space to represent various gaits, enabling continuous gait transitions and terrain-adaptive locomotion.
Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination
Duy Phuong Nguyen (Princeton University), Jaime Fernández Fisac (Princeton University)
Safty and PrivacyAdversarial AttackRobotic IntelligenceReinforcement Learning
🎯 What it does: Proposed a prediction safety filter based on game theory called Gameplay Filter, which utilizes self-adversarial reinforcement learning to train safety policies and worst-case perturbations in simulation, and during runtime predicts and intercepts potential unsafe actions through a complete game replay;
Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks
Alex Quach (MIT), Daniela Rus (MIT)
Convolutional Neural NetworkRecurrent Neural NetworkGaussian SplattingImageTime Series
🎯 What it does: Designed and implemented a complete pipeline for drone visual navigation from simulation to the real world, generating high-fidelity visual + dynamic data by integrating 3D Gaussian Splatting with flight dynamics, and achieving zero-shot transfer and multi-step navigation in real environments using liquid neural networks (LTC/CfC) for behavioral cloning.
GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy
Yixuan Wang (Columbia University), Yunzhu Li (Columbia University)
Robotic IntelligenceTransformerVision Language ModelDiffusion modelMultimodalityPoint Cloud
🎯 What it does: Proposed a framework that utilizes 3D Semantic Fields to achieve category-level generalizable diffusion policies, combining visual foundation models to generate point clouds and semantic information for more robust robotic manipulation;
General Flow as Foundation Affordance for Scalable Robot Learning
Chengbo Yuan (Tsinghua University), Yang Gao (Tsinghua University)
Robotic IntelligenceFlow-based ModelAuto EncoderVideoPoint Cloud
🎯 What it does: This paper proposes using three-dimensional flow (General Flow) as a general affordance to achieve zero-shot transfer from human videos to robotic grasping actions.
Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior
Ruihan Yang (University of California San Diego), Xiaolong Wang (University of California San Diego)
Robotic IntelligenceReinforcement LearningGenerative Adversarial NetworkSequential
🎯 What it does: This paper proposes a reinforcement learning framework named Versatile Instructable Motion Prior (VIM), which enables quadruped robots to learn a unified low-level motion prior through diverse reference motions generated by animal capture data, motion generation models, and trajectory optimization. This allows the robot to continuously perform various agile actions (e.g., running, jumping, backflipping) and train high-level policies to accomplish complex tasks based on this prior.
Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
Zeyu Dong (Stony Brook University), Yu Sun (Sunrise Technology Inc)
Autonomous DrivingTransformerLarge Language ModelPrompt EngineeringImageVideoMultimodalityChain-of-Thought
🎯 What it does: Proposed a closed-loop architecture that integrates a large multimodal language model (LLM) with an end-to-end driving model, enabling rapid generalization from a small amount of training data in real environments. The LLM periodically generates high-level driving instructions, which guide the end-to-end model to output low-level control actions.
Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph
Utkarsh Aashu Mishra, Danfei Xu (Georgia Institute of Technology)
Robotic IntelligenceGraph Neural NetworkDiffusion model
🎯 What it does: Proposes Generative Factor Chaining (GFC), a factor graph learning planning framework based on diffusion models, for generating long-range geometric feasible plans for multi-arm collaborative operations.
Generative Image as Action Models
Mohit Shridhar (Dyson Robot Learning Lab), Stephen James (Dyson Robot Learning Lab)
Robotic IntelligenceTransformerSupervised Fine-TuningDiffusion modelImage
🎯 What it does: This paper proposes a novel robotic behavior cloning framework called GENIMA, which maps robot joint actions to the image space, allowing a stable diffusion model to directly generate target joint positions;
Genetic Algorithm for Curriculum Design in Multi-Agent Reinforcement Learning
Yeeho Song (Carnegie Mellon University), Jeff Schneider (Carnegie Mellon University)
OptimizationReinforcement Learning
🎯 What it does: Propose a multi-agent reinforcement learning self-play training method called GEMS based on genetic algorithms, and introduce an unsupervised open-source GenOpt opponent to form an adaptive training curriculum.
GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
Pu Hua (Tsinghua University), Lirui Wang (MIT CSAIL)
Data SynthesisRobotic IntelligenceTransformerLarge Language ModelReinforcement LearningPrompt EngineeringImageMultimodalityPoint CloudChain-of-Thought
🎯 What it does: Propose GenSim2, an extensible robot simulation task and data generation framework that leverages multimodal and reasoning large language models to automatically generate long-duration simulation tasks involving up to 100 articulated objects, and generates training data through a demonstration generation pipeline.
Gentle Manipulation of Tree Branches: A Contact-Aware Policy Learning Approach
Jay Jacob (University of Sydney), Fabio Ramos (University of Sydney)
Data SynthesisRobotic IntelligenceReinforcement Learning
🎯 What it does: Achieve flexible, low-damage manipulation of branches using proprioceptive contact detectors and reinforcement learning (PCAP) without visual or external force/torque sensors; enable zero-shot migration to real branch scenarios through programmatic L-system forests and domain randomization training.
Get a Grip: Multi-Finger Grasp Evaluation at Scale Enables Robust Sim-to-Real Transfer
Tyler Ga Wei Lum (Stanford University), Jeannette Bohg (Stanford University)
Domain AdaptationRobotic IntelligenceDiffusion modelNeural Radiance FieldImageMultimodalityPoint CloudBenchmark
🎯 What it does: Propose a large-scale multi-finger grasp evaluation dataset containing 3.5M grasp samples, 4.3K objects, RGB images, point clouds, NeRF representations, and soft labels, and train a discriminative grasp evaluator using this dataset to achieve strong sim-to-real transfer.
Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance
RenMing Huang, Heng Tao Shen
Robotic IntelligenceReinforcement LearningDiffusion modelSequential
🎯 What it does: This paper proposes a method to learn long-term goal-reaching strategies using non-expert, no-action observation data. It first generates reasonable subgoals with a diffusion model and produces exploration rewards through a state-goal value function. Both are then embedded into an offline actor-critic framework to achieve efficient online learning.
GraspSplats: Efficient Manipulation with 3D Feature Splatting
Mazeyu Ji (University Of California San Diego), Xiaolong Wang (University Of California San Diego)
Object TrackingComputational EfficiencyRobotic IntelligenceGaussian SplattingImageTextPoint Cloud
🎯 What it does: Propose GraspSplats, a method that utilizes 3D Gaussian Splatting to build an editable, high-quality scene representation, enabling zero-shot, zero-shot component-level grasping and dynamic manipulation.
Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation
Jean Pierre Sleiman, Marco Hutter (ETH Zurich)
Robotic IntelligenceReinforcement Learning
🎯 What it does: This study proposes a control strategy that utilizes a single trajectory planning demonstration combined with reinforcement learning (RL) to train agents capable of executing complex manipulation tasks in multi-contact environments (e.g., pushing/pulling spring doors, opening/closing heavy dishwashers).
Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning
Jonas Günster (TU Darmstadt), Davide Tateo (TU Darmstadt)
Safty and PrivacyReinforcement Learning
🎯 What it does: By integrating ATACOM with distributed reinforcement learning, long-term safety constraints are learned, enabling risk-aware safe reinforcement learning.
Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions
Zhenyu Jiang (University of Texas at Austin), Yuke Zhu (University of Texas at Austin)
GenerationRobotic IntelligenceTransformerVision Language ModelDiffusion modelTextMultimodality
🎯 What it does: Generate full-body humanoid robot actions from free-text descriptions, enhancing action naturalness and executability through VLM editing and human motion priors generated by physics-guided diffusion models.
Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving
Kairui Ding (Institute for AI Industry Research), Hao Zhao (Mercedes-Benz Group China Ltd.)
Autonomous DrivingExplainability and InterpretabilityTransformerLarge Language ModelVision Language ModelVision-Language-Action ModelImageTextPoint Cloud
🎯 What it does: Developed an integrated framework Hint-AD that combines end-to-end autonomous driving with natural language explanations, achieving aligned explanations across the entire perception-prediction-planning process.
HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
Jianke Zhang (Tsinghua University), Jianyu Chen (Tsinghua University)
Computational EfficiencyRobotic IntelligenceTransformerSupervised Fine-TuningVision Language ModelVision-Language-Action ModelContrastive LearningMultimodality
🎯 What it does: Propose the HiRT framework, which achieves high-frequency and high-performance robot control by using a large VLM as a low-frequency slow system to extract semantic features, complemented by a lightweight high-frequency system for rapid execution.
Humanoid Parkour Learning
Ziwen Zhuang (Shanghai Qi Zhi Institute), Hang Zhao (Shanghai Qi Zhi Institute)
Depth EstimationKnowledge DistillationRobotic IntelligenceRecurrent Neural NetworkReinforcement LearningImage
🎯 What it does: This paper designs and trains an end-to-end visual-driven whole-body control strategy for parkour, enabling the humanoid robot Unitree H1 to autonomously perform various parkour actions such as vaulting, hurdle jumping, and gap crossing in indoor and outdoor environments, and automatically select appropriate skills during motion based on rotation commands.
HumanPlus: Humanoid Shadowing and Imitation from Humans
Zipeng Fu (Stanford University), Chelsea Finn (Stanford University)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerReinforcement LearningImageVideo
🎯 What it does: Designed the HumanPlus full-stack system, enabling humanoid robots to real-time follow full-body human movements using a single RGB camera and learn autonomous skills from collected visual data.
HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation
Jin Wang (Istituto Italiano di Tecnologia), Nikos Tsagarakis (Istituto Italiano di Tecnologia)
OptimizationRobotic IntelligenceLarge Language ModelReinforcement LearningVision Language ModelVision-Language-Action ModelImageTextMultimodalityPoint Cloud
🎯 What it does: Propose the HYPERmotion framework to achieve multi-task hybrid motion learning for high-degree-of-freedom robots using RL and whole-body optimization, and enable autonomous hybrid motion and manipulation via LLM and VLM for language-driven behavior planning and morphology selection under unsupervised conditions.
I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences
Zihan Wang (University of Washington), Maya Cakmak (University of Washington)
Explainability and InterpretabilityRobotic IntelligenceLarge Language ModelVision Language ModelMultimodality
🎯 What it does: The RONAR system achieves natural language narratives of a robot's real-time experiences through multimodal key event selection, experience summarization, and LLM-based narrative generation;
IMAGINATION POLICY: Using Generative Point Cloud Models for Learning Manipulation Policies
Haojie Huang (Northeastern University), Robin Walters (Northeastern University)
Robotic IntelligenceConvolutional Neural NetworkVision-Language-Action ModelFlow-based ModelPoint Cloud
🎯 What it does: Propose IMAGINATION POLICY, a multi-task keyframe control strategy that estimates target pose by generating point clouds.
Implicit Grasp Diffusion: Bridging the Gap between Dense Prediction and Sampling-based Grasping
Pinhao Song (KU Leuven), Renaud Detry (KU Leuven)
Pose EstimationRobotic IntelligenceConvolutional Neural NetworkDiffusion modelPoint Cloud
🎯 What it does: Propose the Implicit Grasp Diffusion (IGD) framework, which combines dense prediction with sampling methods. It extracts local features through implicit neural representations and samples grasp poses based on diffusion models. Finally, a two-stage probabilistic grasp evaluator filters high-quality grasp poses.
In-Flight Attitude Control of a Quadruped using Deep Reinforcement Learning
Tarek El-Agroudi (Norwegian University of Science and Technology), Kostas Alexis (Norwegian University of Science and Technology)
Robotic IntelligenceReinforcement LearningTime SeriesSequential
🎯 What it does: This paper designs and implements a flight attitude control strategy based on deep reinforcement learning, utilizing the reaction mass of the quadruped robot's legs to achieve three-dimensional attitude adjustment in free fall and rotating rod experiments.
InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment
Yuxing Long (Peking University), Hao Dong (Peking University)
Robotic IntelligenceTransformerLarge Language ModelVision-Language-Action ModelImageTextPoint CloudChain-of-Thought
🎯 What it does: This paper proposes InstructNav, a zero-shot general-purpose instruction navigation system capable of executing various types of language instructions in unmapped, untrained continuous environments.