CoRL 2023 Papers with AI Summaries
Conference on Robot Learning · 199 papers
→ CoRL 2023 papers with code (34)
Each paper below shows an AI-generated one-line summary. Get the full 6-part summary (innovation, method, data, results, limitations) and search all 199 CoRL 2023 papers by keyword, author or institution —
free trial on arXivSub.
$\alpha$-MDF: An Attention-based Multimodal Differentiable Filter for Robot State Estimation
Xiao Liu (Arizona State University), Heni Ben Amor (Arizona State University)
OptimizationRepresentation LearningRobotic IntelligenceTransformerMultimodality
🎯 What it does: Studied an attention-based multimodal differentiable filter called α-MDF for robot state estimation.
4D-Former: Multimodal 4D Panoptic Segmentation
Ali Athar (Waabi), Raquel Urtasun (Waabi)
Object TrackingSegmentationAutonomous DrivingTransformerImageMultimodalityPoint Cloud
🎯 What it does: Designed and implemented the 4D-Former model, which integrates LiDAR point clouds and RGB images to achieve 4D panoptic segmentation and tracking, outputting semantic labels for each point and time-consistent instance IDs.
A Bayesian approach to breaking things: efficiently predicting and repairing failure modes via sampling
Charles Dawson (MIT), Chuchu Fan (MIT)
Anomaly DetectionOptimizationRobotic IntelligenceStochastic Differential Equation
🎯 What it does: Proposed a Bayesian inference-based simulation framework for efficiently predicting and repairing failure modes of autonomous systems.
A Bayesian Approach to Robust Inverse Reinforcement Learning
Ran Wei (Texas Aandm University), Mingyi Hong (University Of Minnesota)
Reinforcement LearningBenchmark
🎯 What it does: Propose a Bayesian framework for offline model-based inverse reinforcement learning, jointly estimating the expert's reward function and its internal environment dynamics model
A Data-efficient Neural ODE Framework for Optimal Control of Soft Manipulators
Mohammadreza Kasaei (University of Edinburgh), Mohsen Khadem (University of Edinburgh)
OptimizationRobotic IntelligenceSequentialOrdinary Differential Equation
🎯 What it does: Established a continuous forward kinematics model for soft continuum robots and implemented a GPU parallel controller based on MPPI, achieving high-precision trajectory tracking and obstacle avoidance with only 25 scattered data points.
A Data-Efficient Visual-Audio Representation with Intuitive Fine-tuning for Voice-Controlled Robots
Peixin Chang (University of Illinois at Urbana-Champaign), Katherine Rose Driggs-Campbell (University of Illinois at Urbana-Champaign)
Representation LearningRobotic IntelligenceReinforcement LearningVision-Language-Action ModelContrastive LearningMultimodality
🎯 What it does: This paper proposes a data-efficient visual-audio representation named Dif-VAR for speech-controlled robots, enabling non-experts to intuitively fine-tune the model with a small number of visual-audio pairs after deployment, and utilizing this representation to generate intrinsic rewards for navigation and manipulation tasks.
A Policy Optimization Method Towards Optimal-time Stability
Shengjie Wang (Tsinghua University), Yang Gao (Tsinghua University)
OptimizationRobotic IntelligenceReinforcement Learning
🎯 What it does: Proposed a sampling-based Lyapunov stability condition, combined with the Actor-Critic framework to design the Adaptive Lyapunov-based Actor-Critic (ALAC) algorithm, achieving optimal-time stability in robot control.
A Universal Semantic-Geometric Representation for Robotic Manipulation
Tong Zhang (Tsinghua University), Yang Gao (Tsinghua University)
Robotic IntelligenceVision Language ModelVision-Language-Action ModelImagePoint Cloud
🎯 What it does: This paper proposes a unified perception module called Semantic-Geometric Representation (SGR), which integrates semantic and geometric information for robot vision-motor control tasks.
Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
Theophile Gervet (Carnegie Mellon University), Katerina Fragkiadaki (Carnegie Mellon University)
Robotic IntelligenceTransformerVision-Language-Action ModelImageTextMultimodalityPoint Cloud
🎯 What it does: Proposed Act3D, a language-conditioned 3D feature field Transformer for multi-task 6-DoF robot manipulation, capable of adaptively performing coarse-to-fine 3D point sampling and featureization within the workspace, directly generating high-resolution action maps in 3D space;
Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
Jianlan Luo (University of California, Berkeley), Sergey Levine (University of California, Berkeley)
Robotic IntelligenceReinforcement LearningAuto EncoderBenchmark
🎯 What it does: Propose a state-conditioned action quantization (SAQ) method based on VQ-VAE, which discretizes the continuous action space for offline reinforcement learning, enabling precise computation of constraints or conservative terms in offline RL and improving performance.
AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer
Allen Z. Ren (Princeton University), Anirudha Majumdar (Princeton University)
Domain AdaptationRobotic IntelligenceMeta LearningReinforcement Learning
🎯 What it does: Proposes the AdaptSim framework, using task-driven simulation parameter adaptation to enhance sim-to-real transfer effectiveness;
ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation
ZiZhang Wu, Jian Pu (Fudan University)
Depth EstimationKnowledge DistillationTransformerImageBenchmark
🎯 What it does: This paper proposes a monocular depth estimation framework ADU-Depth based on knowledge distillation, where the teacher network learns 3D geometric information from left and right views, guiding the student network to generate more accurate depth maps under monocular input.
Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation
Jay Sarva (Waabi), Raquel Urtasun (Waabi)
Autonomous DrivingOptimizationAdversarial AttackPoint Cloud
🎯 What it does: Proposed the ADV3D framework, which uses closed-loop LiDAR simulation to attack a complete autonomous driving system (perception, prediction, and planning) in real traffic scenarios, generating 3D shapes critical to system safety.
Affordance-Driven Next-Best-View Planning for Robotic Grasping
Xuechao Zhang (Shanghai Jiao Tong University), Jianping He (Shanghai Jiao Tong University)
Robotic IntelligenceConvolutional Neural NetworkNeural Radiance FieldSimultaneous Localization and MappingPoint Cloud
🎯 What it does: This paper proposes a Next-Best-View (NBV) planning framework called ACE-NBV, based on grasping affordance, for grasping occluded target objects in stacked environments. The framework combines perspective perception and perspective generation to predict grasping quality in unobserved views, and uses the predicted maximum grasping quality to determine the next observation view, forming a closed-loop active perception and grasping process.
An Unbiased Look at Datasets for Visuo-Motor Pre-Training
Sudeep Dasari (Carnegie Mellon University), Abhinav Gupta (Carnegie Mellon University)
Representation LearningData-Centric LearningRobotic IntelligenceSupervised Fine-TuningAuto EncoderImageBenchmark
🎯 What it does: This study systematically evaluates the impact of different visual pre-training data on robotic vision-motion tasks by conducting unified Masked Auto-Encoder pre-training on multiple datasets and fine-tuning with behavioral cloning in both simulated and real environments.
AR2-D2: Training a Robot Without a Robot
Jiafei Duan (University of Washington), Ranjay Krishna (University of Washington)
Data SynthesisPose EstimationDepth EstimationRobotic IntelligenceTransformerSupervised Fine-TuningVideoPoint Cloud
🎯 What it does: Propose AR2-D2, an AR-based demonstration collection system that records videos of humans manipulating objects using the iPhone camera and synchronizes the AR robotic arm with hand gestures through hand pose tracking, generating demonstration data directly usable for training real robot behavior cloning; subsequently validated its effectiveness on a real Franka Panda robot.
Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback
Max Balsells I Pamies, Abhishek Gupta (University of Washington)
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningText
🎯 What it does: This paper studies methods for achieving robot autonomous reinforcement learning in real environments, using occasional binary feedback from non-expert humans to guide exploration without requiring manual rewards or resets.
Batch Differentiable Pose Refinement for In-The-Wild Camera/LiDAR Extrinsic Calibration
Lanke Frank Tarimo Fu (University of Oxford), Maurice Fallon (University of Oxford)
Autonomous DrivingOptimizationConvolutional Neural NetworkImagePoint Cloud
🎯 What it does: Proposed a targetless, batch differentiable pose refinement method to achieve extrinsic calibration between cameras and LiDAR in outdoor environments.
BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities
Binyu Zhao (Harbin Institute of Technology), Zhaonian Zou (Harbin Institute of Technology)
Object DetectionDepth EstimationAutonomous DrivingComputational EfficiencyTransformerImageMultimodalityPoint Cloud
🎯 What it does: Proposes BM2CP, a multimodal collaborative perception framework that integrates LiDAR and camera data to achieve efficient collaborative perception.
Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance
Jesse Zhang (University Of Southern California), Joseph J Lim
Robotic IntelligenceLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: Propose the BOSS method, which utilizes large language model (LLM)-guided skill bootstrapping to automatically expand from a few initial skills to the execution of various long-horizon tasks.
BridgeData V2: A Dataset for Robot Learning at Scale
Homer Rich Walke, Sergey Levine (University Of California Berkeley)
Robotic IntelligenceTransformerReinforcement LearningVision-Language-Action ModelDiffusion modelContrastive LearningImageTextMultimodalityBenchmark
🎯 What it does: Propose the BridgeData V2 dataset, containing 60,096 trajectories spanning 24 environments and 13 skills, supporting multi-task robot learning with target images and natural language conditions; evaluate multiple offline learning methods on this dataset.
CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controller
Yuxiang Yang (University of Washington), Byron Boots (University of Washington)
OptimizationRobotic IntelligenceReinforcement Learning
🎯 What it does: Achieve continuous, adaptive jumping on quadruped robots and successfully cross gaps up to 70cm on real robots.
CALAMARI: Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation
Youngsun Wi (University of Michigan), Nima Fazeli (University of Michigan)
Robotic IntelligenceConvolutional Neural NetworkTransformerReinforcement LearningVision-Language-Action ModelContrastive LearningImageTextMultimodality
🎯 What it does: This paper proposes a language-based, contact-aware spatial action mapping method (CALAMARI), which realizes multi-step contact-rich manipulation by predicting binary plane maps indicating tool-surface contact.
CAT: Closed-loop Adversarial Training for Safe End-to-End Driving
Linrui Zhang (Tsinghua University), Bolei Zhou (Ucla)
Autonomous DrivingAdversarial AttackReinforcement LearningVideoSequential
🎯 What it does: This study proposes a closed-loop adversarial training (CAT) framework to enhance the safety of end-to-end driving models by generating safety-critical adversarial scenarios in real-time;
ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation
Zhou Xian (Carnegie Mellon University), Katerina Fragkiadaki (Carnegie Mellon University)
Robotic IntelligenceTransformerVision-Language-Action ModelDiffusion modelTextMultimodalityPoint Cloud
🎯 What it does: Proposed a unified strategy named ChainedDiffuser, combining macro action prediction with trajectory diffusion to learn 6-DoF robot manipulation from demonstrations.
CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning
Jinxin Liu (Zhejiang University), Donglin Wang (Westlake University)
Reinforcement LearningAuto EncoderBenchmark
🎯 What it does: Proposes a framework called CLUE for offline reinforcement learning that leverages limited expert data to relabel intrinsic rewards for unlabeled transfer.
Cold Diffusion on the Replay Buffer: Learning to Plan from Known Good States
Zidan Wang (Northwestern University), Bradly C. Stadie
Robotic IntelligenceDiffusion modelSequential
🎯 What it does: Proposes a Cold Diffusion on Replay Buffer (CDRB) method to generate feasible trajectories from known good states.
Composable Part-Based Manipulation
Weiyu Liu, Jiajun Wu (Stanford)
Robotic IntelligenceTransformerDiffusion modelPoint Cloud
🎯 What it does: Proposed a composable part-based manipulation framework called CPM, which predicts robot manipulation trajectories by leveraging object part decomposition and part-to-part correspondence, supporting zero-shot reasoning.
Compositional Diffusion-Based Continuous Constraint Solvers
Zhutian Yang (Massachusetts Institute of Technology), Leslie Pack Kaelbling (Massachusetts Institute of Technology)
OptimizationDiffusion modelGraphBenchmark
🎯 What it does: Propose a diffusion-based continuous constraint satisfaction problem solver (Diffusion-CCSP), which converts problems into factor graphs and combines energy models of different constraint types to directly generate continuous decision variables that satisfy all constraints.
Context-Aware Deep Reinforcement Learning for Autonomous Robotic Navigation in Unknown Area
Jingsong Liang (National University of Singapore), Guillaume Adrien Sartoretti (National University of Singapore)
Robotic IntelligenceGraph Neural NetworkReinforcement LearningImage
🎯 What it does: Proposed a context-aware deep reinforcement learning-based mapless mobile robot navigation framework that achieves rapid decision-making through constructing an environmental belief map.
Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs
Haonan Chang (Rutgers University), Abdeslam Boularias (Rutgers University)
Object DetectionRetrievalGraph Neural NetworkLarge Language ModelVision Language ModelContrastive LearningPoint CloudGraph
🎯 What it does: Studied context-aware entity alignment based on open-vocabulary 3D scene graphs.
Continual Vision-based Reinforcement Learning with Group Symmetries
Shiqi Liu (Carnegie Mellon University), Ding Zhao (Carnegie Mellon University)
Robotic IntelligenceConvolutional Neural NetworkReinforcement LearningImageBenchmark
🎯 What it does: Proposes COVERS, a vision-based reinforcement learning framework that leverages group symmetry for task grouping and learns separate strategies for each task group in continual learning.
Contrastive Value Learning: Implicit Models for Simple Offline RL
Bogdan Mazoure (Apple), Jonathan Tompson (Google DeepMind)
Reinforcement LearningContrastive LearningImageTabular
🎯 What it does: Propose the Contrastive Value Learning (CVL) algorithm, which uses contrastive learning to learn an implicit multi-step occupancy distribution model, thereby directly estimating Q-values and performing policy optimization without requiring explicit model prediction or TD updates.
Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable
Liangtao Zheng (Shanghai Qi Zhi Institute), Hang Zhao (Tsinghua University)
Autonomous DrivingTransformerImagePoint Cloud
🎯 What it does: This paper investigates the generalization bottleneck of camera-driven 3D object detection across different datasets. First, it evaluates the effects of single-dataset training across datasets and multi-dataset joint training. Subsequently, it proposes a sensor alignment scheme—synchronizing camera intrinsic parameters, extrinsic parameters, and ego coordinate systems—to achieve unified 3D-2D mapping across datasets, significantly improving model generalization performance.
Curiosity-Driven Learning of Joint Locomotion and Manipulation Tasks
Clemens Schwarke (ETH Zurich), Marco Hutter (ETH Zurich)
Robotic IntelligenceReinforcement Learning
🎯 What it does: In simulation and real environments, a curiosity-driven reinforcement learning method (RND + curiosity state) enabled a wheeled-legged robot to simultaneously perform walking and manipulation tasks, such as pushing doors and carrying packages, without requiring extensive task-specific reward designs.
DATT: Deep Adaptive Trajectory Tracking for Quadrotor Control
Kevin Huang (University of Washington), Byron Boots (University of Washington)
Robotic IntelligenceReinforcement Learning
🎯 What it does: Propose DATT, which combines reinforcement learning (RL) with L1 adaptive control to achieve trackable arbitrary trajectories under unknown disturbances.
Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomy
Haimin Hu (Princeton University), Jaime Fernández Fisac (Princeton University)
Autonomous DrivingRobotic IntelligenceTransformerReinforcement LearningVideoSequential
🎯 What it does: Propose the Deception Game framework, integrating a robot's online learning (inference of human behavior) with a safety decision-making closed loop to address uncertainties in human intent, semantic categories, etc.
DEFT: Dexterous Fine-Tuning for Hand Policies
Aditya Kannan (Carnegie Mellon University), Deepak Pathak (Carnegie Mellon University)
Data-Centric LearningRobotic IntelligenceTransformerReinforcement LearningVision-Language-Action ModelAuto EncoderVideo
🎯 What it does: This paper proposes the DEFT method, which utilizes human gesture priors learned from internet videos on a soft robotic hand, enabling the completion of various complex hand tasks with minimal real-world interaction.
Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play
Irmak Guzey (New York University), Lerrel Pinto (New York University)
Representation LearningRobotic IntelligenceConvolutional Neural NetworkContrastive LearningImage
🎯 What it does: Propose the T-DEX framework, which achieves multi-finger robot grasping and manipulation through self-supervised pre-training of a tactile encoder combined with a small number of demonstrations;
Dexterous Functional Grasping
Ananye Agarwal (Carnegie Mellon University), Deepak Pathak (Carnegie Mellon University)
OptimizationRepresentation LearningRobotic IntelligenceTransformerReinforcement LearningVision-Language-Action ModelContrastive LearningImageMesh
🎯 What it does: Leverage pre-trained visual models for one-shot affordance matching, followed by training a low-frequency closed-loop RL strategy to enable multi-fingered hand grasping of tools and subsequent functional actions.
Diff-LfD: Contact-aware Model-based Learning from Visual Demonstration for Robotic Manipulation via Differentiable Physics-based Simulation and Rendering
Xinghao Zhu (University of California Berkeley), Lin Shao (National University of Singapore)
Robotic IntelligenceReinforcement LearningDiffusion modelWorld ModelVideo
🎯 What it does: Proposes a visual demonstration learning framework called Diff-LfD based on differentiable physics simulation and rendering, which can self-supervisedly estimate object shape and pose from raw RGB videos, generate long-term contact-aware operation sequences, and ultimately train neural policies to achieve real-world robot manipulation.
Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
William Shen (MIT CSAIL), Phillip Isola (MIT CSAIL)
OptimizationKnowledge DistillationRobotic IntelligenceMeta LearningVision Language ModelVision-Language-Action ModelNeural Radiance FieldContrastive LearningImageText
🎯 What it does: This study proposes a framework that distills dense features from 2D vision and vision-language models (e.g., DINO, CLIP) into a 3D feature field (Distilled Feature Field, DFF), enabling robots to perform few-shot learning and natural language-guided 6-DOF grasping and placing tasks in unknown scenarios.
DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking
Qing LIAN, Jiangmiao Pang (Shanghai AI Laboratory)
Object DetectionObject TrackingAutonomous DrivingRecurrent Neural NetworkVideo
🎯 What it does: This paper proposes the DORT framework, which leverages temporal information from multi-camera videos to jointly estimate the motion and position of dynamic objects, achieving 3D detection and tracking.
DROID: Learning from Offline Heterogeneous Demonstrations via Reward-Policy Distillation
Sravan Jayanthi (Georgia Institute of Technology), Matthew Gombolay (Georgia Institute of Technology)
Knowledge DistillationRobotic IntelligenceReinforcement LearningImage
🎯 What it does: Propose a method called DROID for offline learning from heterogeneous demonstrations, which can simultaneously distill rewards and policies to address issues of demonstration diversity and limited data;
Dynamic Handover: Throw and Catch with Bimanual Hands
Binghao Huang (University Of California San Diego), Xiaolong Wang (University Of California San Diego)
Robotic IntelligenceReinforcement Learning
🎯 What it does: Construct a dual-arm grasping and throwing system, train throwing and grasping strategies using multi-agent reinforcement learning in simulation, and then transfer these strategies to real robots to perform throwing and grasping tasks.
Dynamic Multi-Team Racing: Competitive Driving on 1/10-th Scale Vehicles via Learning in Simulation
Peter Werner (MIT), Daniela Rus (MIT)
Autonomous DrivingReinforcement LearningWorld ModelTime Series
🎯 What it does: Train hierarchical reinforcement learning strategies for multi-vehicle racetrack teams in simulation and directly transfer them to real hardware without additional training (zero-shot transfer).
DYNAMO-GRASP: DYNAMics-aware Optimization for GRASP Point Detection in Suction Grippers
Boling Yang (University of Washington), Joshua Smith (University of Washington)
OptimizationRobotic IntelligenceTransformer
🎯 What it does: Proposed a dynamic perception-based suction cup grasping point detection method called DYNAMO-GRASP
Efficient Sim-to-real Transfer of Contact-Rich Manipulation Skills with Online Admittance Residual Learning
Xiang Zhang (University of California at Berkeley), Masayoshi Tomizuka (University of California at Berkeley)
Domain AdaptationOptimizationRobotic IntelligenceReinforcement Learning
🎯 What it does: In simulation, model-free reinforcement learning (with domain randomization) is used to offline learn robot motion trajectories and adapt initial damping parameters; subsequently, real-time force/torque sensor data is collected on the physical robot, and online adaptation is achieved by optimizing residual damping parameters, enabling contact-rich operations such as insertion, pivoting, and screw tightening.
Embodied Lifelong Learning for Task and Motion Planning
Jorge Mendez-Mendez (MIT), Tomás Lozano-Pérez (MIT)
Robotic IntelligenceMeta LearningMixture of ExpertsDiffusion modelBenchmark
🎯 What it does: In the long-term deployment of home robots, a lifelong learning framework based on Task and Motion Planning (TAMP) is proposed, aiming to improve planning success rates by learning continuous parameter samplers.
Enabling Efficient, Reliable Real-World Reinforcement Learning with Approximate Physics-Based Models
Tyler Westenbroek (University of Texas at Austin), David Fridovich-Keil (University of Texas at Austin)
Robotic IntelligenceReinforcement LearningWorld Model
🎯 What it does: A policy gradient-based reinforcement learning framework is proposed by combining approximate physical models with low-level feedback control, enabling efficient and reliable learning of control policies with limited real-world data.
Energy-based Potential Games for Joint Motion Forecasting and Control
Christopher Diehl (TU Dortmund University), Torsten Bertram (TU Dortmund University)
Autonomous DrivingOptimizationGraph Neural NetworkVideoGraph
🎯 What it does: Proposes a multi-agent motion prediction and control framework based on Energy-based Potential Game (EPO), integrating neural networks for inferring game parameters with a differentiable game optimization layer;
Equivariant Motion Manifold Primitives
Byeongho Lee (Seoul National University), Frank C. Park (Seoul National University)
GenerationRepresentation LearningRobotic IntelligenceAuto EncoderSequential
🎯 What it does: Proposes a motion primitive model (MMP) capable of learning continuous motion trajectory manifolds, and further constructs equivariant motion primitives (EMMP) by leveraging task symmetry
Equivariant Reinforcement Learning under Partial Observability
Hai Huu Nguyen (Northeastern University), Christopher Amato (Northeastern University)
Robotic IntelligenceConvolutional Neural NetworkRecurrent Neural NetworkReinforcement LearningSequential
🎯 What it does: Proposes a method to improve the sample efficiency of reinforcement learning in partially observable environments by leveraging group equivariance;
Expansive Latent Planning for Sparse Reward Offline Reinforcement Learning
Robert Gieselmann (KTH Royal Institute of Technology), Florian T. Pokorny (KTH Royal Institute of Technology)
Robotic IntelligenceReinforcement LearningAuto EncoderGenerative Adversarial NetworkWorld ModelImageBenchmark
🎯 What it does: Propose a sampling-based expansion tree planning algorithm called VELAP, which integrates a discrete action generation model, dynamic model, and value function to perform global search in latent space for solving offline reinforcement learning tasks with sparse rewards.
FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing
Kyle Stachowicz (University Of California Berkeley), Sergey Levine (University Of California Berkeley)
Autonomous DrivingConvolutional Neural NetworkReinforcement LearningImageTabularTime Series
🎯 What it does: A FastRLAP system was built and validated on a 1/10 scale autonomous vehicle, which learns high-speed driving from visual observations through deep reinforcement learning (RL) and achieves autonomous execution without human intervention in real-world environments.
Few-Shot In-Context Imitation Learning via Implicit Graph Alignment
Vitalis Vosylius (Imperial College London), Edward Johns (Imperial College London)
Pose EstimationOptimizationRobotic IntelligenceGraph Neural NetworkTransformerContrastive LearningPoint Cloud
🎯 What it does: Proposed a few-shot demonstration context imitation learning framework that enables robots to complete tasks on new objects using only 3-4 demonstrations with a graph energy model.
Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching
H.J. Terry Suh (MIT), Russ Tedrake (MIT)
Reinforcement LearningDiffusion modelScore-based ModelImageTabular
🎯 What it does: Proposed a gradient-based offline model-based reinforcement learning (MBRL) planning method called Score-Guided Planning (SGP), which guides the planning process by learning the gradient of the data distribution (score function), ensuring planned trajectories remain within the training data distribution to reduce model bias.
FindThis: Language-Driven Object Disambiguation in Indoor Environments
Arjun Majumdar (Georgia Institute of Technology), Leonidas Guibas (Google DeepMind)
Object DetectionTransformerLarge Language ModelVision-Language-Action ModelImageTextPoint Cloud
🎯 What it does: Propose the multi-round interactive language-driven indoor object localization task FindThis, and develop the GoFind algorithm to achieve fine-grained object localization through natural language and visual attributes.
Fine-Tuning Generative Models as an Inference Method for Robotic Tasks
Orr Krupnik (Technion Israel Institute of Technology), Aviv Tamar (Technion Israel Institute of Technology)
Robotic IntelligenceSupervised Fine-TuningScore-based ModelAuto EncoderPoint CloudTabular
🎯 What it does: Propose a framework (MACE) that rapidly fine-tunes deep generative models for robotic task inference using the cross-entropy method
Finetuning Offline World Models in the Real World
Yunhai Feng (University of California San Diego), Xiaolong Wang
Robotic IntelligenceReinforcement LearningWorld ModelImageSequential
🎯 What it does: This paper studies how to use offline data for pre-training world models on real robots, and then perform fine-tuning through a small amount of online interaction to achieve data-efficient reinforcement learning.
Fleet Active Learning: A Submodular Maximization Approach
Oguzhan Akcin (University of Texas at Austin), Sandeep P. Chinchali (University of Texas at Austin)
Autonomous DrivingOptimizationFederated LearningConvolutional Neural NetworkTransformerImage
🎯 What it does: Propose a multi-robot active learning framework (Fleet Active Learning, FAL) based on submodular function maximization, enabling robots to collaboratively select the most informative samples to enhance the performance of cloud-based deep learning models without sharing all data.
FlowBot++: Learning Generalized Articulated Objects Manipulation via Articulation Projection
Harry Zhang (Carnegie Mellon University), David Held (Carnegie Mellon University)
Robotic IntelligenceConvolutional Neural NetworkOptical FlowPoint Cloud
🎯 What it does: Developed the FlowBot++ system, which can learn and manipulate new movable objects (e.g., doors, drawers) without prior interaction experience
General In-hand Object Rotation with Vision and Touch
Haozhi Qi (UC Berkeley), Jitendra Malik (UC Berkeley)
Robotic IntelligenceTransformerReinforcement LearningImageMultimodalityPoint Cloud
🎯 What it does: Train a strategy that learns multi-axis rotation in a simulated environment, and achieve no-finetuning rotation of fingers on objects of different shapes in the real world by fusing visual, tactile, and proprioceptive modalities through a Transformer.
Generalization of Heterogeneous Multi-Robot Policies via Awareness and Communication of Capabilities
Pierce Howell (Georgia Institute of Technology), Harish Ravichandar (Georgia Institute of Technology)
Robotic IntelligenceGraph Neural NetworkReinforcement LearningTabular
🎯 What it does: In multi-robot collaboration, this study investigates how robots can adaptively collaborate through perceiving and communicating their capabilities, achieving training-free transfer to new team compositions, scales, and entirely new robots.
Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Rendering
Yasasa Abeysirigoonawardena (University of Toronto), Florian Shkurti (University of Toronto)
Autonomous DrivingOptimizationAdversarial AttackNeural Radiance FieldImage
🎯 What it does: Construct a differentiable simulator based on NeRF, insert editable objects, and generate transferable autonomous driving attack scenarios through gradient optimization.
Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models
Utkarsh Aashu Mishra (Georgia Institute of Technology), Danfei Xu (Georgia Institute of Technology)
Robotic IntelligenceTransformerDiffusion model
🎯 What it does: Studied a generative skill chain (GSC) framework based on diffusion models for long-horizon robot manipulation task planning under a given skill skeleton.
Geometry Matching for Multi-Embodiment Grasping
Maria Attarian (Google DeepMind), Jonathan Tompson (Google DeepMind)
Robotic IntelligenceGraph Neural NetworkPoint Cloud
🎯 What it does: Propose a multi-configuration grasping method called GeoMatch based on graph neural networks, which can generate stable and diverse grasping postures for various end-effectors (from 2-finger to 5-finger) within a single model.
Gesture-Informed Robot Assistance via Foundation Models
Li-Heng Lin (Stanford University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceTransformerLarge Language ModelVision-Language-Action ModelImageTextMultimodality
🎯 What it does: This paper proposes the GIRAF framework, which utilizes large language models combined with gestures and language to enable robot collaboration under multimodal instructions.
GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields
Yanjie Ze (AWS AI, Amazon), Xiaolong Wang (Shanghai Jiao Tong University)
Robotic IntelligenceTransformerDiffusion modelNeural Radiance Field
🎯 What it does: Propose GNFactor, integrating multi-task robot learning with generalizable neural feature fields;
Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control
Vivek Myers (University of California Berkeley), Sergey Levine (University of California Berkeley)
Representation LearningRobotic IntelligenceTransformerVision Language ModelContrastive LearningImageTextSequential
🎯 What it does: This paper proposes the Goal Representations for Instruction Following (GRIF) method, which enables robots to perform operations according to natural language instructions by aligning task representations using a small amount of language-labeled data and a large amount of unlabeled trajectories.
Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environments
Jason Xinyu Liu (Brown University), Ankit Shah (Brown University)
Robotic IntelligenceTransformerLarge Language ModelReinforcement LearningPrompt EngineeringTextGraph
🎯 What it does: A system is proposed to convert natural language commands into linear temporal logic (LTL) specifications for enabling robots to perform navigation tasks in unseen environments.
HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation
Wenxuan Zhou (Carnegie Mellon University), David Held (Carnegie Mellon University)
Pose EstimationRobotic IntelligenceReinforcement LearningPoint Cloud
🎯 What it does: Train reinforcement learning strategies to accomplish non-grasping 6D pose alignment tasks.
HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation
Vainavi Viswanath (University of California Berkeley), Ken Goldberg
Object TrackingData SynthesisDomain AdaptationConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: Designed and implemented a learning-based tracking and cross-identification framework called HANDLOOM for spatial state estimation of long flexible linear objects (e.g., cables, ropes) in semi-planar configurations, applied to multi-cable inspection, instructional knot-tying, and robotic untangling tasks.
Heteroscedastic Gaussian Processes and Random Features: Scalable Motion Primitives with Guarantees
Edoardo Caldarelli (Institut de Rob'tica i Inform' atica Industrial, CSIC - UPC), Carme Torras (Institut de Rob'tica i Inform' atica Industrial, CSIC - UPC)
Computational EfficiencyRobotic IntelligenceSequential
🎯 What it does: Investigated the combination of heteroscedastic Gaussian processes (HGP) with random features to achieve scalable motion primitives and provide theoretical guarantees.
Hierarchical Planning for Rope Manipulation using Knot Theory and a Learned Inverse Model
Matan Sudry (Technion - Israel Institute of Technology), Erez Karpas (Technion - Israel Institute of Technology)
Robotic IntelligenceSequential
🎯 What it does: Proposed the TWISTED system, which employs knot theory for high-level topological planning and uses a self-supervised inverse model at the low level to achieve rope knotting actions;
Hijacking Robot Teams Through Adversarial Communication
Zixuan Wu (Georgia Institute of Technology), Matthew Gombolay (Georgia Institute of Technology)
Adversarial AttackRobotic IntelligenceReinforcement LearningTabularSequential
🎯 What it does: Design a black-box adversarial communication attack method that constructs a proxy strategy using known observations, messages, and action data, and learns the attack strategy in an offline environment. This method successfully reduces the reward and collision rate of multi-agent collaborative systems without interacting with the environment or requiring reward information.
HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTs
Esteve Valls Mascaro (Technische Universitat Wien), Dongheui Lee (Technische Universitat Wien)
Object DetectionComputational EfficiencyRobotic IntelligenceTransformerVision-Language-Action ModelContrastive LearningVideo
🎯 What it does: Propose a real-time Transformer framework called HOI4ABOT for detecting and predicting human-object interactions from videos, enabling collaborative robots to anticipate human intentions and provide proactive assistance.
HomeRobot: Open-Vocabulary Mobile Manipulation
Sriram Yenamandra (Georgia Tech), Chris Paxton (FAIR Meta AI)
Robotic IntelligenceReinforcement LearningVision-Language-Action ModelImagePoint CloudMeshBenchmark
🎯 What it does: Proposes an open-vocabulary mobile manipulation (OVMM) benchmark tailored for home scenarios, achieving reproducible HomeRobot library and baseline algorithms in both simulated and real environments.
How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equations
Franck Djeumou (University of Texas at Austin), ufuk topcu
Robotic IntelligenceDiffusion modelTime SeriesSequentialPhysics RelatedStochastic Differential Equation
🎯 What it does: Construct physics-constrained dynamical models with uncertainty assessment using Neural SDEs, and validate their prediction and control performance on multiple robotic systems (spring-mass-damper, inverted pendulum, hexarotor).
Human-in-the-Loop Task and Motion Planning for Imitation Learning
Ajay Mandlekar (NVIDIA), Dieter Fox (NVIDIA)
Robotic IntelligenceReinforcement Learning from Human FeedbackSequential
🎯 What it does: Propose a human-robot interaction task and motion planning framework called HITL-TAMP, which uses a TAMP-gating mechanism to collect human demonstrations and train policies in robots' long-term, contact-rich tasks.
HYDRA: Hybrid Robot Actions for Imitation Learning
Suneel Belkhale (Stanford University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceSequential
🎯 What it does: Propose the HYDRA method, which combines a hybrid action space of sparse waypoints and dense low-level actions, dynamically switching control modes.
IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors
Gaurav Datta (UC Berkeley), Ken Goldberg (UC Berkeley)
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningBenchmark
🎯 What it does: Propose an Implicit Interactive Fleet Learning (IIFL) framework leveraging energy-based models, capable of learning from diverse, heterogeneous human supervisors and addressing distribution shift and multimodal problems in multi-robot environments.
Im2Contact: Vision-Based Contact Localization Without Touch or Force Sensing
Leon Kim (University of Pennsylvania), Dinesh Jayaraman (University of Pennsylvania)
Robotic IntelligenceConvolutional Neural NetworkOptical FlowImageVideo
🎯 What it does: Under a single RGB-D camera viewpoint, a neural network is trained to locate external contact points between a robot's gripper and the environment using depth maps, reference object maps, and optical flow information, completely without relying on tactile or force tensor sensors.
Imitating Task and Motion Planning with Visuomotor Transformers
Murtaza Dalal (Carnegie Mellon University), Dieter Fox (Carnegie Mellon University)
Robotic IntelligenceTransformerImage
🎯 What it does: Train a vision-motion Transformer to imitate data generated by TAMP, enabling real-time robotic manipulation from image inputs to low-level task space control, addressing long-horizon, multi-object manipulation tasks.
Improving Behavioural Cloning with Positive Unlabeled Learning
Qiang Wang (University College Dublin), Stephen J. Redmond (MPI for Intelligent Systems)
Data-Centric LearningRobotic IntelligenceReinforcement LearningSequential
🎯 What it does: Proposes an offline policy learning method based on positive and unlabeled sample learning—Positive Unlabeled Behavioural Cloning (PUBC), which iteratively filters expert trajectories from a mixed-quality dataset and then learns a control policy using behavioral cloning.
Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning
Xiyang Wu (University of Maryland), Dinesh Manocha (University of Maryland)
Autonomous DrivingRecurrent Neural NetworkGraph Neural NetworkReinforcement Learning
🎯 What it does: Proposed a distributed multi-agent reinforcement learning algorithm called iPLAN for joint trajectory and intent prediction and planning in dense heterogeneous traffic.
KITE: Keypoint-Conditioned Policies for Semantic Manipulation
Priya Sundaresan (Stanford University), Jeannette Bohg (Stanford University)
Robotic IntelligenceConvolutional Neural NetworkLarge Language ModelVision Language ModelVision-Language-Action ModelImageTextPoint Cloud
🎯 What it does: Propose the KITE framework to achieve high-precision multi-step semantic control based on keypoint localization and language instructions;
LabelFormer: Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds
Anqi Joyce Yang (Waabi), Raquel Urtasun (Waabi)
Object TrackingAutonomous DrivingTransformerPoint Cloud
🎯 What it does: Proposed LabelFormer, a trajectory-level LiDAR trajectory refinement model based on Transformer, for automatic annotation.
Language Conditioned Traffic Generation
Shuhan Tan (University Of Texas At Austin), Philipp Kraehenbuehl (University Of Texas At Austin)
GenerationAutonomous DrivingTransformerLarge Language ModelTextChain-of-Thought
🎯 What it does: Proposed the LCTGen model to automatically generate high-fidelity traffic scenes from natural language descriptions;
Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
Adam Rashid (UC Berkeley), Ken Goldberg (UC Berkeley)
Robotic IntelligenceLarge Language ModelVision-Language-Action ModelNeural Radiance FieldImageText
🎯 What it does: Achieving task-oriented grasping based on natural language through zero-shot language embedding radiance fields (LERF).
Language to Rewards for Robotic Skill Synthesis
Wenhao Yu (Google DeepMind), Fei Xia (Google DeepMind)
OptimizationRobotic IntelligenceTransformerLarge Language ModelReinforcement LearningPrompt EngineeringVision Language ModelImageText
🎯 What it does: Proposes a system that leverages large language models (LLMs) to automatically generate robot reward functions, and achieves real-time optimization through MuJoCo MPC, enabling the direct synthesis of complex robot motions from natural language instructions.
Language-Conditioned Path Planning
Amber Xie (University of California, Berkeley), Stephen James
OptimizationRobotic IntelligenceTransformerSupervised Fine-TuningImageTextPoint CloudMesh
🎯 What it does: Propose a language-conditioned path planning (LAPP) framework that learns an acceptable collision function (LACO) using single-view RGB images, language instructions, and robot joint states, enabling intelligent decision-making for acceptable contact in collision planning and supporting complex manipulation tasks requiring contact in real-world environments.
Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter
Georgios Tziafas (University of Groningen), Hamidreza Kasaei (University of Groningen)
Robotic IntelligenceConvolutional Neural NetworkTransformerVision Language ModelVision-Language-Action ModelContrastive LearningImageTextMultimodality
🎯 What it does: This paper proposes an end-to-end language-guided robotic grasping model named CROG, which can predict grasp poses based on natural language instructions in cluttered indoor scenes.
Language-Guided Traffic Simulation via Scene-Level Diffusion
Ziyuan Zhong (Columbia University), Baishakhi Ray (Columbia University)
GenerationAutonomous DrivingTransformerLarge Language ModelDiffusion model
🎯 What it does: Propose a language-guided scene-level conditional diffusion model for generating realistic and controllable traffic simulation trajectories.
Large Language Models as General Pattern Machines
Suvir Mirchandani (Stanford University), Andy Zeng (Google DeepMind)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerLarge Language ModelPrompt EngineeringTextSequential
🎯 What it does: Evaluate the pattern reasoning ability of large language models under zero-shot conditional learning, and apply them as a general-purpose pattern machine to robot control tasks, including sequence transformation, sequence completion, and sequence improvement.
Learning Efficient Abstract Planning Models that Choose What to Predict
Nishanth Kumar (MIT CSAIL), Leslie Pack Kaelbling (MIT CSAIL)
OptimizationComputational EfficiencyRobotic IntelligenceWorld ModelBenchmark
🎯 What it does: Propose a symbolic operator learning method based on planning performance rather than prediction error, learning an abstract model suitable for multi-layer planning through hill-climbing search.
Learning Generalizable Manipulation Policies with Object-Centric 3D Representations
Yifeng Zhu (University of Texas at Austin), Yuke Zhu (University of Texas at Austin)
Robotic IntelligenceTransformerImagePoint Cloud
🎯 What it does: Proposed a novel imitation learning framework called GROOT, which learns closed-loop visual-motor control policies using object-centric 3D point cloud representations, achieving generalization to background variations, camera viewpoints, and new object instances within a single training environment.
Learning Human Contribution Preferences in Collaborative Human-Robot Tasks
Michelle D Zhao (Carnegie Mellon University), Henny Admoni (Carnegie Mellon University)
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement Learning
🎯 What it does: Propose the ICPL framework, which investigates how to learn human preferences regarding their contributions in collaborative tasks, and realizes online learning through Bayesian learning algorithms.
Learning Lyapunov-Stable Polynomial Dynamical Systems Through Imitation
Amin Abyaneh (McGill University), Hsiu-Chin Lin (McGill University)
OptimizationRobotic IntelligenceTime SeriesSequentialOrdinary Differential Equation
🎯 What it does: Design and learn a globally stable polynomial dynamical system as a motion planning strategy for imitation learning, jointly optimizing polynomial dynamics and polynomial Lyapunov candidate functions to ensure global asymptotic stability.
Learning Realistic Traffic Agents in Closed-loop
Chris Zhang (University of Toronto), Raquel Urtasun (University of Toronto)
Autonomous DrivingConvolutional Neural NetworkRecurrent Neural NetworkGraph Neural NetworkSupervised Fine-TuningReinforcement LearningGraphSequential
🎯 What it does: Propose a closed-loop imitation learning (IL) and reinforcement learning (RL) integration method called Reinforcing Traffic Rules (RTR), which learns more realistic traffic agents by matching expert demonstrations under traffic compliance constraints;