CoRL 2023 Papers — Page 2
Conference on Robot Learning · 199 papers
Learning Reusable Manipulation Strategies
Jiayuan Mao (Massachusetts Institute of Technology), Leslie Pack Kaelbling (Massachusetts Institute of Technology)
Robotic IntelligenceConvolutional Neural NetworkPoint CloudPhysics Related
🎯 What it does: This paper proposes a framework that learns reusable operational strategies (mechanisms) from a single demonstration and self-play, integrating the learned mechanisms and specialized samplers into task and motion planning. It demonstrates six mechanisms that can be combined for more complex tasks.
Learning Robot Manipulation from Cross-Morphology Demonstration
Gautam Salhotra (University of Southern California), Gaurav S. Sukhatme (University of Southern California)
OptimizationRobotic IntelligenceReinforcement Learning from Human FeedbackConvolutional Neural NetworkRecurrent Neural NetworkReinforcement LearningImageSequential
🎯 What it does: Proposed the MAIL framework, enabling robots to learn from demonstrators with significantly different numbers of end-effectors, particularly for high-dimensional, contact-rich tasks such as fabric manipulation, ultimately allowing a single-arm robot to complete fabric hanging and folding tasks through dual-arm demonstrations.
Learning Sequential Acquisition Policies for Robot-Assisted Feeding
Priya Sundaresan (Stanford University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceReinforcement LearningWorld ModelImageTextSequential
🎯 What it does: Propose the VAPORS framework, enabling robots to clean food from plates over extended periods using multiple dedicated primitives through hierarchical planning.
Learning to Design and Use Tools for Robotic Manipulation
Ziang Liu (Stanford University), Jiajun Wu (Stanford University)
OptimizationRobotic IntelligenceGraph Neural NetworkReinforcement Learning
🎯 What it does: This paper proposes learning the tool designer's strategy and controller's strategy jointly through reinforcement learning, enabling robots to quickly generate and use appropriate tools to complete manipulation tasks based on task requirements.
Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning
Sachit Kuhar (Georgia Institute of Technology), Danfei Xu (Georgia Institute of Technology)
Representation LearningRobotic IntelligenceReinforcement LearningContrastive LearningSequential
🎯 What it does: This paper proposes the L2D method, which learns from human demonstration data with diverse quality and styles, automatically identifying and filtering high-quality demonstrations.
Learning to Drive Anywhere
Ruizhao Zhu (Boston University), Venkatesh Saligrama (Boston University)
Autonomous DrivingFederated LearningTransformerContrastive Learning
🎯 What it does: Proposed AnyD, a geographically conditioned imitation learning model capable of adaptive driving decision-making across cities with diverse traffic rules, and validated its superior performance on multiple datasets.
Learning to See Physical Properties with Active Sensing Motor Policies
Gabriel B. Margolis (Massachusetts Institute of Technology), Pulkit Agrawal (Massachusetts Institute of Technology)
SegmentationData SynthesisOptimizationRobotic IntelligenceTransformerReinforcement LearningImageVideo
🎯 What it does: Designed and trained active sensing motor policies capable of proactively collecting sufficient information to accurately estimate physical parameters such as ground friction and roughness. Subsequently, these self-supervised labels were used to train a visual module that predicts per-pixel ground physical properties from color images, combined with a cost function learned in simulation for vision-based path planning, achieving adaptive navigation on real quadruped robots for different tasks (free walking and dragging loads).
Leveraging 3D Reconstruction for Mechanical Search on Cluttered Shelves
Seungyeon Kim (Seoul National University), Frank C. Park (Seoul National University)
RecognitionObject DetectionRobotic IntelligencePoint Cloud
🎯 What it does: Studied a framework that uses a standard two-finger gripper to perform mechanical search and eventually grasp completely occluded target objects on a cluttered shelf through pushing and placing actions.
M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place
Wentao Yuan (University of Washington), Dieter Fox (University of Washington)
Pose EstimationRobotic IntelligenceTransformerVision-Language-Action ModelPoint Cloud
🎯 What it does: This study proposes M2T2, a multi-task masked Transformer model that can generate 6-DoF grasp and placement poses for objects in unseen cluttered scenes in a single pass;
ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting
Kushal Kedia (Cornell University), Sanjiban Choudhury (Cornell University)
OptimizationRobotic IntelligenceGraph Neural NetworkSupervised Fine-TuningTime Series
🎯 What it does: Developed the MANICAST framework, which learns a cost-aware human motion prediction model and applies it to real-time MPC planning to achieve smooth human-robot collaborative operations.
Marginalized Importance Sampling for Off-Environment Policy Evaluation
Pulkit Katdare (University of Illinois Urbana-Champaign), Katherine Rose Driggs-Campbell
Robotic IntelligenceReinforcement Learning
🎯 What it does: This paper proposes an edge importance sampling (MIS) method that combines a simulator with offline real-world data, enabling the evaluation of policy performance in the real world without directly deploying robots into real environments.
Measuring Interpretability of Neural Policies of Robots with Disentangled Representation
Tsun-Hsuan Wang (Massachusetts Institute of Technology), Daniela Rus (Massachusetts Institute of Technology)
Explainability and InterpretabilityRepresentation LearningRobotic IntelligenceRecurrent Neural NetworkReinforcement LearningImageTime SeriesOrdinary Differential Equation
🎯 What it does: Proposes a method that uses decision trees to extract logical programs from neural policies, measuring the relationship between the explainability of robot neural networks and separable representations; defines a 'change factor' and constructs quantifiable explainability metrics; evaluates and compares the explainability of different compact network architectures across various robot tasks; and demonstrates the extraction of interpretable behavioral patterns from the activation of individual neurons.
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Ajay Mandlekar, Dieter Fox (NVIDIA)
Robotic IntelligenceRecurrent Neural NetworkSequential
🎯 What it does: This paper proposes the MimicGen system, which automatically generates large-scale, scenario-diverse datasets using a small number of human demonstrations for training robot imitation learning.
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
Chen Wang (Stanford), Anima Anandkumar (NVIDIA)
Robotic IntelligenceTransformerPrompt EngineeringVision-Language-Action ModelVideo
🎯 What it does: This work proposes a hierarchical imitation learning framework called MIMICPLAY, which first learns 3D perceptual latent plans from human free-play videos (hand trajectories), and then combines them with a small number of robot teleoperation demonstrations to train a low-level controller, achieving multi-task long-horizon robot manipulation.
MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
Rafael Rafailov (Stanford University), Chelsea Finn (Stanford University)
Robotic IntelligenceReinforcement LearningAuto EncoderWorld ModelImage
🎯 What it does: Propose a model-based reinforcement learning framework called MOTO for robot learning, combining offline pre-training with online fine-tuning.
Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictors
Sushant Veer (NVIDIA Research), Marco Pavone (NVIDIA Research)
Autonomous DrivingExplainability and InterpretabilityTime SeriesSequential
🎯 What it does: Proposes a multi-predictor fusion (MPF) method that combines a learning-based trajectory predictor with a rule-based hierarchical planner, dynamically balancing their predictions through Bayesian belief;
Multi-Resolution Sensing for Real-Time Control with Vision-Language Models
Saumya Saxena (Carnegie Mellon University), Oliver Kroemer (Carnegie Mellon University)
Robotic IntelligenceConvolutional Neural NetworkTransformerReinforcement LearningVision-Language-Action ModelMultimodality
🎯 What it does: Propose a multi-resolution (spatial and temporal) perception framework that uses a pre-trained vision-language model to extract low-frequency global information, while employing a small network to process high-frequency local visual and force-sensing data, learning multi-task language-conditioned robotic manipulation policies that enable real-time control.
MUTEX: Learning Unified Policies from Multimodal Task Specifications
Rutav Shah (University of Texas at Austin), Yuke Zhu (University of Texas at Austin)
Representation LearningRobotic IntelligenceTransformerReinforcement LearningVision Language ModelVision-Language-Action ModelContrastive LearningImageVideoTextMultimodalityAudio
🎯 What it does: Proposed and implemented a unified multimodal task specification strategy called MUTEX, which can guide robots to complete diverse manipulation tasks based on six modalities (or any combination), such as video, image, text, and speech.
Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning
Dhruv Shah (University Of California Berkeley), Sergey Levine (University Of California Berkeley)
Robotic IntelligenceLarge Language ModelPrompt EngineeringVision Language ModelImageTextMultimodalityChain-of-Thought
🎯 What it does: Propose the Language Frontier Guide (LFG) method, which uses large language models (LLM) to score frontier points as a search heuristic to guide robot navigation in unknown environments;
Neural Field Dynamics Model for Granular Object Piles Manipulation
Shangjie Xue (Georgia Institute of Technology), Danfei Xu (Georgia Institute of Technology)
OptimizationRobotic IntelligenceConvolutional Neural NetworkImage
🎯 What it does: Propose a density-field-based fully convolutional neural network dynamics model (Neural Field Dynamics, NFD) for learning and planning push-pull operations on granular piles, supporting curved trajectories and obstacle avoidance;
Neural Graph Control Barrier Functions Guided Distributed Collision-avoidance Multi-agent Control
Songyuan Zhang (Massachusetts Institute of Technology), Chuchu Fan (Massachusetts Institute of Technology)
Autonomous DrivingGraph Neural NetworkReinforcement LearningPoint Cloud
🎯 What it does: Propose a graph-based control barrier function (GCBF) that learns to achieve collision and obstacle avoidance for large-scale multi-agent systems;
NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities
Ruohan Zhang (Stanford University), Jiajun Wu (Stanford University)
Robotic IntelligenceVision Language ModelVision-Language-Action ModelContrastive LearningImageBiomedical DataRetrieval-Augmented Generation
🎯 What it does: Developed the NOIR system, utilizing non-intrusive EEG brain signals (SSVEP and MI) to enable humans to control robots through mental commands to complete 20 daily household tasks (such as cooking, cleaning, personal care, and entertainment).
On the Utility of Koopman Operator Theory in Learning Dexterous Manipulation Skills
Yunhai Han (Georgia Institute of Technology), Harish Ravichandar (Georgia Institute of Technology)
Robotic IntelligenceSequential
🎯 What it does: This study proposes the KODex framework, which utilizes the Koopman operator to map the high-dimensional nonlinear dynamics of a robot and an object into a linear system, subsequently learning the target trajectory through analytical solving and achieving dexterous manipulation with an inverse dynamics controller;
One-shot Imitation Learning via Interaction Warping
Ondrej Biza (Northeastern University), Robert Platt (Northeastern University)
Robotic IntelligenceMeta LearningPoint CloudMesh
🎯 What it does: Achieve one-shot SE(3) robot manipulation policy learning through the 'Interaction Warping' method, automatically generating grasp and placement actions via point cloud shape deformation and interaction point migration;
One-Shot Imitation Learning: A Pose Estimation Perspective
Pietro Vitiello (Imperial College London), Edward Johns (Imperial College London)
SegmentationPose EstimationRobotic IntelligenceConvolutional Neural NetworkTransformerVision-Language-Action ModelImagePoint Cloud
🎯 What it does: This paper proposes a one-shot imitation learning method that achieves trajectory transfer using unseen object pose estimation under conditions of a single demonstration, no additional data collection, and no prior task or object knowledge.
Online Learning for Obstacle Avoidance
David Snyder (Princeton University), Anirudha Majumdar (Princeton University)
OptimizationRobotic Intelligence
🎯 What it does: A safety control framework based on online learning is proposed to address the robot obstacle avoidance problem, enabling adaptive correction of planned trajectories and resistance to disturbances in real-time environments.
Online Model Adaptation with Feedforward Compensation
ABULIKEMU ABUDUWEILI (Carnegie Mellon University), Changliu Liu (Carnegie Mellon University)
Domain AdaptationRobotic IntelligenceTime Series
🎯 What it does: Propose an online adaptation method that updates the prediction model by leveraging feedforward compensation and similar historical samples from a memory buffer to reduce the error bound;
Open-World Object Manipulation using Pre-Trained Vision-Language Models
Austin Stone (Robotics at Google), Karol Hausman (Robotics at Google)
Robotic IntelligenceTransformerReinforcement LearningVision Language ModelImageTextMultimodality
🎯 What it does: This paper proposes the MOO (Manipulation of Open-World Objects) method, which utilizes a frozen pre-trained vision-language model to locate target objects in instructions, and inputs single-pixel masks along with image and language information into an end-to-end trained robot control policy, achieving zero-shot manipulation of objects outside the training set.
OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data
Shiyang Lu, Kostas Bekris (Rutgers University)
RetrievalConvolutional Neural NetworkTransformerVision Language ModelContrastive LearningTextPoint Cloud
🎯 What it does: Proposed an open-vocabulary 3D instance retrieval method called OVIR-3D that does not require 3D training. It uses a 2D open-vocabulary detector to generate text-aligned region candidates, projects and aggregates them into 3D point clouds, achieving instance-level retrieval for text queries;
PairwiseNet: Pairwise Collision Distance Learning for High-dof Robot Systems
Jihwan Kim (Seoul National University), Frank C. Park (Seoul National University)
Robotic IntelligenceGraph Neural NetworkPoint Cloud
🎯 What it does: Propose PairwiseNet, which estimates the global collision distance of a robotic system by learning the minimal collision distance between two geometric shapes, and then obtains the global distance by taking the minimum of all pairwise distances.
Parting with Misconceptions about Learning-based Vehicle Motion Planning
Daniel Dauner (University of Tübingen), Kashyap Chitta (University of Tübingen)
Autonomous DrivingOptimization
🎯 What it does: Investigated misconceptions in learning-based vehicle motion planning and proposed a hybrid model, PDM-Hybrid, combining rule-based planning with learning-based prediction.
PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play
Lili Chen (Carnegie Mellon University), Deepak Pathak (Carnegie Mellon University)
Robotic IntelligenceConvolutional Neural NetworkVision-Language-Action ModelDiffusion modelAuto EncoderTextMultimodality
🎯 What it does: Propose PlayFusion, a multi-task robot control framework based on diffusion models, capable of learning goal-oriented skills from unstructured 'play' data annotated with post-hoc language instructions;
PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining
Garrett Thomas (Stanford University), Andrey Kolobov (Microsoft Research)
Robotic IntelligenceTransformerSupervised Fine-TuningVideoSequential
🎯 What it does: Propose a Transformer architecture named PLEX, which combines a planner and an executor, pre-trained using task-agnostic visual action trajectories and a large amount of task-annotated videos, and can be fine-tuned with a small number of task-specific demonstration videos;
PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
Shizhe Chen (Inria), Ivan Laptev (Inria)
Robotic IntelligenceGraph Neural NetworkTransformerVision-Language-Action ModelMultimodalityPoint Cloud
🎯 What it does: Propose a language-guided robot manipulation network called PolarNet based on 3D point clouds, which can predict 7-DOF robotic arm actions according to natural language instructions and achieve multi-task manipulation in simulation and real robots.
Policy Stitching: Learning Transferable Robot Policies
Pingcheng Jian (Duke University), Boyuan Chen (Duke University)
Robotic IntelligenceReinforcement LearningImagePoint Cloud
🎯 What it does: Proposes the Policy Stitching framework, which achieves unsupervised transfer between robots and task combinations through modular policy design and latent space alignment.
Polybot: Training One Policy Across Robots While Embracing Variability
Jonathan Heewon Yang (Stanford University), Chelsea Finn (Stanford University)
Domain AdaptationRobotic IntelligenceContrastive LearningImageTabular
🎯 What it does: Train a single policy deployable across multiple robots, achieving cross-robot transfer through unified wrist-mounted camera observations, shared inverse kinematics controllers, and contrastive learning to align internal representations.
Precise Robotic Needle-Threading with Tactile Perception and Reinforcement Learning
Zhenjun Yu (Shanghai Jiao Tong University), Cewu Lu (Shanghai Jiao Tong University)
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement Learning
🎯 What it does: Proposed a tactile perception and reinforcement learning-based needle-threading method T-NT, achieving full robotic operation in two stages: tail end localization and insertion.
PreCo: Enhancing Generalization in Co-Design of Modular Soft Robots via Brain-Body Pre-Training
Yuxing Wang, Xueqian Wang (Tencent AI Lab)
Robotic IntelligenceMeta LearningTransformerReinforcement Learning
🎯 What it does: Propose a brain-body pre-training based co-design method called PreCo, which achieves multi-task zero-shot generalization and few-shot fine-tuning on modular soft robots through a generic co-design strategy.
Predicting Object Interactions with Behavior Primitives: An Application in Stowing Tasks
Haonan Chen (University of Illinois Urbana Champaign), Katherine Rose Driggs-Campbell (University of Illinois Urbana Champaign)
Robotic IntelligenceGraph Neural NetworkReinforcement LearningPoint Cloud
🎯 What it does: A graph neural network (GNN) model learns from single demonstrations and behavioral primitives to perform multi-object stacking stowing tasks in warehouse environments.
Predicting Routine Object Usage for Proactive Robot Assistance
Maithili Patel (Georgia Institute of Technology), Sonia Chernova (Georgia Institute of Technology)
Robotic IntelligenceGraph Neural NetworkTransformerAuto EncoderContrastive LearningTime SeriesSequential
🎯 What it does: Developed a sequential latent temporal model called SLaTe-PRO for predicting daily object usage in robot-assisted environments, supporting proactive assistance and interactive queries.
Preference learning for guiding the tree search in continuous POMDPs
Jiyong Ahn (KAIST), Beomjoon Kim (KAIST)
OptimizationRobotic IntelligenceReinforcement Learning from Human FeedbackTransformerReinforcement LearningAuto Encoder
🎯 What it does: Propose a POMCPOW-guided search framework based on preference learning, which leverages past tree search experience to train a value function and multimodal policy, thereby improving planning efficiency and success rate in continuous POMDPs.
Push Past Green: Learning to Look Behind Plant Foliage by Moving It
Xiaoyu Zhang (University of Illinois at Urbana-Champaign), Saurabh Gupta (University of Illinois at Urbana-Champaign)
Robotic IntelligenceConvolutional Neural NetworkImageAgriculture Related
🎯 What it does: By learning to push plant leaves apart to reveal the hidden space behind them, addressing the problem of plant self-occlusion.
Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
Yevgen Chebotar (Google DeepMind), Sergey Levine (Google DeepMind)
Robotic IntelligenceTransformerReinforcement LearningImageSequential
🎯 What it does: Propose Q-Transformer, an offline reinforcement learning framework based on Transformer, for large-scale multi-task robot control, which can simultaneously leverage demonstration data and self-collected failure samples.
Quantifying Assistive Robustness Via the Natural-Adversarial Frontier
Jerry Zhi-Yang He (University of California Berkeley), Anca Dragan
Adversarial AttackRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningGenerative Adversarial Network
🎯 What it does: Proposed and implemented the RIGID method, which utilizes GAN/discriminator to generate natural-adversarial frontiers in simulating human motions for evaluating the robustness of collaborative robots when facing natural human motions, and validated the effectiveness of this method through VR user experiments and adversarial expert testing.
Ready, Set, Plan! Planning to Goal Sets Using Generalized Bayesian Inference
Jana Pavlasek (University of Michigan), Tucker Hermans (NVIDIA)
OptimizationRobotic IntelligencePoint Cloud
🎯 What it does: Proposes a method for directly planning trajectories on an uncertain target set (composed solely of sample points), utilizing generalized Bayesian inference and Stein variational gradient descent to achieve differentiable trajectory optimization.
Rearrangement Planning for General Part Assembly
Yulong Li (Columbia University), Shuran Song (Columbia University)
Pose EstimationOptimizationRobotic IntelligenceTransformerPoint Cloud
🎯 What it does: Proposed a general-purpose part assembly framework that predicts the precise pose of parts based on the target shape and available part set, enabling assembly of unknown target shapes.
REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation
Zheyuan Hu (University Of California Berkeley), Sergey Levine (University Of California Berkeley)
Robotic IntelligenceConvolutional Neural NetworkReinforcement LearningImageSequential
🎯 What it does: This paper proposes the REBOOT system, which enables multi-fingered robot hands to autonomously learn contact-rich manipulation skills in the real world using reinforcement learning (RL), and accelerates learning by reusing data from previous tasks or different objects.
REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
Zeyi Liu (Columbia University), Shuran Song (Columbia University)
Explainability and InterpretabilityRobotic IntelligenceTransformerLarge Language ModelVision Language ModelVision-Language-Action ModelImageMultimodalityChain-of-ThoughtAudio
🎯 What it does: Proposes the REFLECT framework, which generates hierarchical robot experience summaries from multimodal perceptual data and uses large language models (LLMs) to progressively explain and correct failures.
Reinforcement Learning Enables Real-Time Planning and Control of Agile Maneuvers for Soft Robot Arms
Rianna Jitosho (Stanford University), Karen Liu (Stanford University)
Domain AdaptationRobotic IntelligenceRecurrent Neural NetworkReinforcement LearningTime SeriesSequential
🎯 What it does: Implement real-time planning and control on a soft robotic arm using deep reinforcement learning, achieving high-dynamic swinging actions.
Revisiting Depth-guided Methods for Monocular 3D Object Detection by Hierarchical Balanced Depth
Yi-Rong Chen (National Taiwan University), Winston H. Hsu (National Taiwan University)
Object DetectionAutonomous DrivingConvolutional Neural NetworkTransformerImageBenchmark
🎯 What it does: A new monocular 3D object detection framework called MonoHBD based on hierarchical balanced depth is proposed, which addresses the problem of loss imbalance caused by depth supervision and achieves real-time detection.
RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools
Haochen Shi (Stanford University), Jiajun Wu (Stanford University)
Robotic IntelligenceConvolutional Neural NetworkGraph Neural NetworkReinforcement LearningVideoPoint CloudGraph
🎯 What it does: Developed the RoboCook framework, utilizing a robotic arm and 15 printable tools to perform multi-tool operations on long-duration soft objects (e.g., dough) through perception, dynamics modeling, and closed-loop control, such as making dumplings and alphabet cookies.
RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning
Kevin Zakka (University Of California Berkeley), Pieter Abbeel (Google DeepMind)
Robotic IntelligenceReinforcement LearningSequentialBenchmarkAudio
🎯 What it does: The study uses deep reinforcement learning to train a simulated humanoid hand to play 150 piano pieces on a keyboard, demonstrating high-dimensional space control and fine coordination.
Robot Learning with Sensorimotor Pre-training
Ilija Radosavovic (University of California, Berkeley), Jitendra Malik (University of California, Berkeley)
Representation LearningRobotic IntelligenceTransformerSupervised Fine-TuningAuto EncoderImageMultimodalitySequential
🎯 What it does: Propose a self-supervised sensorimotor pre-training method called RPT, which uses Transformer to perform masked prediction on vision, joint states, and action sequences, thereby improving robot learning efficiency.
Robot Parkour Learning
Ziwen Zhuang, Hang Zhao
Depth EstimationKnowledge DistillationRobotic IntelligenceRecurrent Neural NetworkReinforcement LearningImageTabular
🎯 What it does: Proposed and implemented an end-to-end visual-driven robotic climbing system capable of performing obstacle navigation tasks such as climbing, jumping, crawling, and穿梭 on low-cost quadruped robots.
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Allen Z. Ren (Princeton University), Anirudha Majumdar (Google DeepMind)
Robotic IntelligenceTransformerLarge Language ModelText
🎯 What it does: Proposes the KNOWNO framework, which utilizes conformal prediction to calibrate the uncertainty of large language model planners, enabling robots to proactively request assistance when needed.
Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization
Yuan Zhang (University of Freiburg), Joschka Boedecker (University of Freiburg)
Robotic IntelligenceReinforcement Learning
🎯 What it does: Propose a robust reinforcement learning framework based on uncertainty set regularization (USR) to achieve robust policy learning for continuous control tasks;
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brianna Zitkovich (Google DeepMind), Kehang Han (Google DeepMind)
Robotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelVision-Language-Action ModelImageTextMultimodality
🎯 What it does: Fine-tune large-scale vision-language pre-trained models into robot control models, directly representing actions with text tokens to achieve closed-loop control and gain significant generalization and reasoning capabilities.
RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal (NVIDIA), Dieter Fox (NVIDIA)
Pose EstimationRobotic IntelligenceTransformerSupervised Fine-TuningVision-Language-Action ModelImagePoint Cloud
🎯 What it does: Proposed a 3D object manipulation method called RVT based on multi-view Transformer, which can predict the 3D pose of the end-effector and gripper state by jointly processing multi-view images in the robot's workspace, achieving multi-task object manipulation.
SA6D: Self-Adaptive Few-Shot 6D Pose Estimator for Novel and Occluded Objects
Ning Gao (Bosch Center for Artificial Intelligence), Gerhard Neumann (Karlsruhe Institute of Technology)
Pose EstimationConvolutional Neural NetworkContrastive LearningImagePoint Cloud
🎯 What it does: Propose an adaptive few-shot 6D pose estimation method SA6D, which can predict the 6D pose of unseen, occluded tabletop objects using only a few annotated RGB-D reference images.
Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
Katherine Metcalf (Apple), Barry-John Theobald (Apple)
Reinforcement Learning from Human FeedbackReinforcement LearningContrastive LearningTabularSequentialBenchmark
🎯 What it does: Integrate self-supervised temporal consistency tasks with preference learning to construct a reward function that encodes dynamics, significantly improving sample efficiency in preference-based reinforcement learning.
SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning
Krishan Rana (Queensland University of Technology), Niko Suenderhauf
Robotic IntelligenceTransformerLarge Language ModelPrompt EngineeringTextGraphChain-of-Thought
🎯 What it does: Propose SayPlan, a scalable task planning method that leverages 3D Scene Graphs (3DSG) and Large Language Models (LLM) in multi-level, multi-room environments.
SayTap: Language to Quadrupedal Locomotion
Yujin Tang (Google DeepMind), Tatsuya Harada (University of Tokyo)
Robotic IntelligenceTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: Developed an interactive quadruped robot control system that converts user-provided natural language instructions into low-level motion commands through foot contact patterns, enabling diverse gait execution.
Scalable Deep Kernel Gaussian Process for Vehicle Dynamics in Autonomous Racing
Jingyun Ning (University of Virginia), Madhur Behl (University of Virginia)
Autonomous DrivingTime SeriesSequential
🎯 What it does: Proposed and validated a model called DKL-SKIP that combines deep kernel learning with scalable Gaussian processes (SKIP-GP) to learn the error between the extended kinematic model of autonomous racing vehicles and real observations.
ScalableMap: Scalable Map Learning for Online Long-Range Vectorized HD Map Construction
Jingyi Yu (Wuhan University), Jizhang Sang (Wuhan University)
Autonomous DrivingTransformerSimultaneous Localization and MappingImage
🎯 What it does: Proposed an end-to-end pipeline for online long-range vectorized HD map construction based on a single monocular camera (ScalableMap).
SCALE: Causal Learning and Discovery of Robot Manipulation Skills using Simulation
Tabitha Edith Lee (Carnegie Mellon University), Oliver Kroemer (Carnegie Mellon University)
Robotic IntelligenceReinforcement Learning
🎯 What it does: This paper proposes the SCALE method, which discovers and learns a set of interpretable, compressed robotic manipulation skills through causal interventions in simulations;
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Huy Ha (Columbia University), Shuran Song (Google DeepMind)
Domain AdaptationKnowledge DistillationRobotic IntelligenceTransformerLarge Language ModelVision-Language-Action ModelDiffusion modelMultimodalityBenchmark
🎯 What it does: Propose a robot skill acquisition framework guided by large language models (LLMs), which can automatically generate and annotate multi-task visual-language-motion data, and distill it into robust multi-task language-conditioned visual-motor policies.
SCONE: A Food Scooping Robot Learning Framework with Active Perception
Yen-Ling Tai (National Yang Ming Chiao Tung University), Yi-Ting Chen (National Yang Ming Chiao Tung University)
Robotic IntelligenceConvolutional Neural NetworkRecurrent Neural NetworkTransformerVision-Language-Action ModelImageMultimodalitySequential
🎯 What it does: Proposed a food scooping robot learning framework called SCONE based on active perception, achieving implicit encoding of food properties through interaction encoding and state retrieval, thereby enhancing the generalization and stability of scooping strategies.
Seeing-Eye Quadruped Navigation with Force Responsive Locomotion Control
David DeFazio (Binghamton University), Shiqi Zhang (Binghamton University)
Robotic IntelligenceConvolutional Neural NetworkReinforcement LearningSimultaneous Localization and MappingPoint CloudTime Series
🎯 What it does: This study proposes a quadrupedal guide system for visually impaired individuals that can withstand human pulling forces and estimate the direction of external forces in real-time. The system includes: 1) a gait controller trained via reinforcement learning (PPO) that maintains stable locomotion under external disturbances; 2) a supervised learning-based external force estimator that estimates the magnitude and direction of pulling forces using joint encoder and IMU data; 3) a LIDAR-based local planner that selects the next navigation target based on the estimated force direction.
Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning
Archit Sharma (Stanford University), Chelsea Finn (Stanford University)
Robotic IntelligenceReinforcement Learning from Human FeedbackConvolutional Neural NetworkReinforcement LearningVision-Language-Action ModelImageBenchmark
🎯 What it does: Propose MEDAL++ to realize self-improving robots, achieving autonomous training with only a few demonstrations in non-episodic environments.
Semantic Mechanical Search with Large Vision and Language Models
Satvik Sharma (University of California Berkeley), Ken Goldberg (University of California Berkeley)
Object DetectionRobotic IntelligenceTransformerLarge Language ModelPrompt EngineeringVision Language ModelContrastive LearningImageTextMultimodality
🎯 What it does: This paper leverages large vision and language models (VLM/LLM) to first perform scene understanding, then uses LLM to generate object semantic affinity between objects, obtaining a pluggable semantic occupancy distribution, further enhancing the performance of mechanical search (including closed-world reasoning and open-world localization).
Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation
Yuanpei Chen (Stanford University), Karen Liu
Robotic IntelligenceTransformerReinforcement Learning
🎯 What it does: Designed and implemented the Sequential Dexterity system, leveraging reinforcement learning to cascade multiple dexterous wrist-hand manipulation strategies, enabling long-horizon operations involving multi-stage, distinct subtasks (e.g., block stacking and tool localization).
Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement
Anthony Simeonov (Massachusetts Institute of Technology), Dieter Fox (NVIDIA)
Pose EstimationRobotic IntelligenceTransformerDiffusion modelPoint Cloud
🎯 What it does: Proposed a Relational Pose Diffusion (RPDiff) based on iterative pose denoising diffusion, enabling multi-modal object-scene relationship rearrangement under unknown geometry, pose, and layout.
Simultaneous Learning of Contact and Continuous Dynamics
Bibit Bianchini (University of Pennsylvania), Michael Posa (University of Pennsylvania)
OptimizationRobotic IntelligenceSimultaneous Localization and MappingTime SeriesSequentialBenchmarkPhysics Related
🎯 What it does: Proposes a system identification method that simultaneously learns the contact dynamics and continuous dynamics of unknown objects. The method uses an implicit loss based on violation rate to train the physical model and incorporates residual neural networks in the continuous dynamics to compensate for unmodeled effects.
SLAP: Spatial-Language Attention Policies
Priyam Parashar, Chris Paxton
Robotic IntelligenceRecurrent Neural NetworkTransformerLarge Language ModelVision-Language-Action ModelPoint Cloud
🎯 What it does: Propose Spatial-Language Attention Policies (SLAP), a multi-task robotic control framework that integrates point clouds, language, and attention mechanisms, enabling the learning of continuous action policies from a few demonstrations in mobile manipulator and desktop scenarios.
Stabilize to Act: Learning to Coordinate for Bimanual Manipulation
Jennifer Grannen (Stanford University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceConvolutional Neural NetworkRecurrent Neural NetworkImageSequential
🎯 What it does: Designed and implemented the BUDS framework, which assigns one arm as the stable arm and the other as the execution arm in dual-arm tasks, achieving efficient sample learning through action space decomposition.
Stealthy Terrain-Aware Multi-Agent Active Search
Nikhil Angad Bakshi (Carnegie Mellon University), Jeff Schneider (Carnegie Mellon University)
OptimizationSafty and PrivacyImage
🎯 What it does: A STAR algorithm for covert multi-agent active search under known terrain maps is studied, balancing information gain and covertness, supporting distributed decision-making under communication failure and noisy observations.
STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience
Haresh Karnan (University of Texas at Austin), Peter Stone (University of Texas at Austin)
Representation LearningData-Centric LearningRobotic IntelligenceMultimodality
🎯 What it does: Propose an unsupervised and self-supervised terrain representation learning framework called STERLING, which learns terrain features required for visual navigation using unlabeled multimodal data collected by robots under non-expert operation, and applies these features in offline path planning tasks with preference alignment.
Stochastic Occupancy Grid Map Prediction in Dynamic Scenes
Zhanteng Xie (Temple University), Philip Dames (Temple University)
Autonomous DrivingRobotic IntelligenceRecurrent Neural NetworkAuto EncoderImage
🎯 What it does: This paper proposes two stochastic occupancy grid map prediction methods based on variational autoencoders, SOGMP and SOGMP++, achieving probabilistic prediction of future maps for mobile robots in dynamic environments.
STOW: Discrete-Frame Segmentation and Tracking of Unseen Objects for Warehouse Picking Robots
Yi Li (University of Washington), Dieter Fox (University of Washington)
Object TrackingSegmentationRobotic IntelligenceTransformerContrastive LearningImageVideo
🎯 What it does: Achieve instance segmentation and cross-frame tracking of unknown objects in a discrete frame environment to address perception challenges for warehouse picking robots under occlusion, rearrangement, and latency.
Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning
Manav Kulshrestha (Purdue University), Ahmed H Qureshi
OptimizationRobotic IntelligenceGraph Neural NetworkPoint Cloud
🎯 What it does: Proposed a structural concept learning method SCL based on graph attention networks, which automatically infers multi-level structural dependencies from RGB-D point clouds and generates optimal relocation plans.
Surrogate Assisted Generation of Human-Robot Interaction Scenarios
Varun Bhatt (University of Southern California), Stefanos Nikolaidis (University of Southern California)
Data SynthesisOptimizationRobotic Intelligence
🎯 What it does: Utilizing a deep neural network as a proxy model to predict human-robot interaction outcomes, combined with a differentiable quality diversity (DQD) algorithm to automatically generate diverse and challenging interaction scenarios;
Synthesizing Navigation Abstractions for Planning with Portable Manipulation Skills
Eric Rosen (Brown University), George Konidaris (Brown University)
Robotic IntelligenceReinforcement LearningSimultaneous Localization and MappingImage
🎯 What it does: By integrating manipulable skills with semantic maps, automatically generate navigation abstractions to achieve planning and execution for mobile manipulation tasks.
TactileVAD: Geometric Aliasing-Aware Dynamics for High-Resolution Tactile Control
Miquel Oller (University of Michigan), Nima Fazeli (University of Michigan)
OptimizationRobotic IntelligenceConvolutional Neural NetworkAuto EncoderImagePoint CloudBenchmark
🎯 What it does: Proposed TactileVAD, a generative decoder-only linear latent dynamics model for high-resolution tactile control, and introduced the tactile cartpole benchmark task.
Task Generalization with Stability Guarantees via Elastic Dynamical System Motion Policies
Tianyu Li (University of Pennsylvania), Nadia Figueroa (University of Pennsylvania)
Robotic IntelligencePoint CloudTime Series
🎯 What it does: Propose the Elastic-DS method, which utilizes task-parameterized Elastic-GMM to adaptively rewrite traditional LPV-DS, achieving zero-shot transfer from single demonstrations to new task instances while ensuring stability.
Task-Oriented Koopman-Based Control with Contrastive Encoder
Xubo Lyu (Simon Fraser University), Mo Chen (Simon Fraser University)
OptimizationExplainability and InterpretabilityRepresentation LearningRobotic IntelligenceReinforcement LearningContrastive LearningImage
🎯 What it does: Propose a task-oriented Koopman control framework that uses end-to-end reinforcement learning and contrastive encoders to simultaneously learn Koopman embeddings, linear operators, and corresponding LQR controllers.
Tell Me Where to Go: A Composable Framework for Context-Aware Embodied Robot Navigation
Harel Biggie (University of Colorado Boulder), Chris Heckman
Robotic IntelligenceLarge Language ModelPrompt EngineeringVision-Language-Action ModelSimultaneous Localization and MappingImageTextMultimodalityPoint Cloud
🎯 What it does: Developed the NavCon framework to interface LLM-generated code with mobile robot planning systems, enabling context-aware navigation based on natural language.
That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation
Abitha Thankaraj (New York University), Lerrel Pinto (New York University)
Representation LearningRobotic IntelligenceContrastive LearningTime SeriesAudio
🎯 What it does: Construct the AURL framework using audio self-supervised learning, predicting and generating dynamic, contact-rich robotic manipulation behaviors from 25k interaction audio-action pairs collected by contact microphones.
Topology-Matching Normalizing Flows for Out-of-Distribution Detection in Robot Learning
Jianxiang Feng (Technical University of Munich), Rudolph Triebel (Technical University of Munich)
Anomaly DetectionRobotic IntelligenceFlow-based ModelImage
🎯 What it does: Propose a topology-matched regularized flow (cRSB) trained with an information bottleneck for out-of-distribution (OOD) detection in robot vision.
Towards General Single-Utensil Food Acquisition with Human-Informed Actions
Ethan Kroll Gordon, Siddhartha Srinivasa
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningImageSequential
🎯 What it does: This paper collects trajectory data of humans acquiring food using a single utensil (e.g., a fork) and constructs a 26-dimensional action space, which is clustered into 11 discrete, executable action sets for robot-assisted feeding.
Towards Scalable Coverage-Based Testing of Autonomous Vehicles
James Tu (Waabi), Raquel Urtasun (Waabi)
Autonomous DrivingTabularBenchmark
🎯 What it does: Proposes a scalable coverage testing framework called GUARD based on Gaussian processes, which can estimate pass/fail probabilities in continuous parameterized scenarios and partition the parameter space into safe, dangerous, and unknown regions.
TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning
Weiwei Liu (Zhejiang University), Yong Liu (Zhejiang University)
Autonomous DrivingTransformerReinforcement Learning
🎯 What it does: Proposed a virtual traffic coordinator network called TraCo for multi-agent reinforcement learning to learn and issue team collaboration instructions, enabling each vehicle to follow collective demands in traffic scenarios.
Transforming a Quadruped into a Guide Robot for the Visually Impaired: Formalizing Wayfinding, Interaction Modeling, and Safety Mechanism
J. Taery Kim (Georgia Institute of Technology), Sehoon Ha (Georgia Institute of Technology)
Robotic IntelligenceReinforcement LearningSimultaneous Localization and MappingPoint CloudSequential
🎯 What it does: Transform a quadruped robot into a guide dog robot, achieving integration of navigation, interaction models, and safety mechanisms;
Tuning Legged Locomotion Controllers via Safe Bayesian Optimization
Daniel Widmer (ETH Zürich), Stelian Coros (ETH Zürich)
OptimizationRobotic Intelligence
🎯 What it does: Automatically adjust the feedback gains of a quadruped robot's controller through safe Bayesian optimization (GOSAFEOPT), addressing the discrepancy between the model and real hardware, and achieving safe and efficient parameter optimization under different gaits.
UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding
Han Xue (Shanghai Jiao Tong University), Cewu Lu (Shanghai Jiao Tong University)
Robotic IntelligenceConvolutional Neural NetworkTransformerSupervised Fine-TuningVideoPoint CloudBenchmark
🎯 What it does: Propose the UniFolding system, which utilizes UFONet to end-to-end unifiedly achieve garment unfolding and folding, supporting various styles such as long-sleeve and short-sleeve shirts.
ViNT: A Foundation Model for Visual Navigation
Dhruv Shah (University Of California Berkeley), Sergey Levine (University Of California Berkeley)
Robotic IntelligenceTransformerPrompt EngineeringDiffusion modelImage
🎯 What it does: Built and trained a Transformer-based visual navigation foundation model called ViNT, capable of achieving zero-shot navigation and rapid adaptation to downstream tasks across various robot platforms and environments.
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Wenlong Huang (Stanford University), Li Fei-Fei (Stanford University)
Robotic IntelligenceTransformerLarge Language ModelPrompt EngineeringVision Language ModelMultimodalityMesh
🎯 What it does: Propose a zero-shot framework VoxPoser, which leverages a large language model (LLM) to generate code for invoking a vision-language model (VLM) to construct a 3D value map. The framework directly plans robot trajectories under a model predictive control (MPC) framework, enabling various manipulation tasks such as daily grasping and opening doors.
Waypoint-Based Imitation Learning for Robotic Manipulation
Lucy Xiaoyang Shi (Stanford University), Chelsea Finn (Stanford University)
Robotic IntelligenceDiffusion model
🎯 What it does: This paper proposes an unsupervised preprocessing method called AWE for automatically extracting key waypoints, reducing decision length in behavioral cloning and mitigating cumulative error issues.
What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
Peide Huang (Carnegie Mellon University), Ding Zhao (Carnegie Mellon University)
Domain AdaptationRobotic IntelligenceGraph Neural NetworkReinforcement LearningTabularTime Series
🎯 What it does: Proposes a framework called COMPASS that automatically adjusts simulation environment parameters through differentiable causal discovery methods, thereby narrowing the gap between simulation and the real world, and achieving higher trajectory alignment and task success rates in dynamic multi-object interaction tasks (e.g., mini aerial hockey).
XSkill: Cross Embodiment Skill Discovery
Mengda Xu (Columbia University), Shuran Song (Columbia University)
Robotic IntelligenceTransformerDiffusion modelContrastive LearningVideo
🎯 What it does: Propose the XSkill framework, enabling the learning of reusable operational skills across different body types from unannotated and misaligned human-robot videos, and performing one-time imitation and combination on robots.