RSS 2024 Papers — Page 2
Robotics: Science and Systems · 134 papers
RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Multi-Modal Large Language Model Learning
Jianhao Yuan (University of Oxford), Matthew Gadd (University of Oxford)
Autonomous DrivingExplainability and InterpretabilityTransformerLarge Language ModelVision-Language-Action ModelVideoTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: Propose RAG-Driver, an end-to-end autonomous driving system that achieves explainability and generalization by leveraging retrieval-augmented multimodal large language models.
Real-Time Anomaly Detection and Reactive Planning with Large Language Models
Rohan Sinha (Stanford University), Marco Pavone (Stanford University)
Anomaly DetectionAutonomous DrivingOptimizationRobotic IntelligenceLarge Language ModelVision Language ModelTextMultimodalityChain-of-Thought
🎯 What it does: Proposed a two-stage runtime monitoring framework, AESOP, based on large language models (LLMs), for real-time anomaly detection and safe intervention decisions through slow generative reasoning after detecting anomalies, while coupling it with model predictive control (MPC) to achieve safe control for agile robots (e.g., drones, autonomous vehicles).
Reconciling Reality through Simulation: A Real-To-Sim-to-Real Approach for Robust Manipulation
Marcel Torne Villasevil (Massachusets Institute of Technology), Pulkit Agrawal (Massachusets Institute of Technology)
Knowledge DistillationRobotic IntelligenceReinforcement LearningPoint Cloud
🎯 What it does: Propose the RialTo system, leveraging fast scanning and digital twin scene construction, reverse distillation, reinforcement learning fine-tuning, and teacher-student distillation to achieve a robust control pipeline from real to simulation and back to real, significantly improving the success rate of home robots under disturbances and visual interference.
Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloning
Vitalis Vosylius (Dyson Robot Learning Lab), Stephen James (Dyson Robot Learning Lab)
Robotic IntelligenceTransformerDiffusion modelImage
🎯 What it does: Propose the Render and Diffuse method, unifying low-level robot actions and RGB observations into the image space, and iteratively updating the rendering actions through a learned diffusion denoising process to achieve behavior cloning.
Rethinking Robustness Assessment: Adversarial Attacks on Learning-based Quadrupedal Locomotion Controllers
Fan Shi (ETH Zurich), Stelian Coros (ETH Zurich)
Adversarial AttackRobotic IntelligenceReinforcement LearningTime SeriesSequential
🎯 What it does: A learning-based adversarial attack framework is proposed for quadrupedal robot locomotion controllers based on deep reinforcement learning, which can effectively discover and exploit vulnerabilities in both simulation and real robots;
Risk-Calibrated Human-Robot Interaction via Set-Valued Intent Prediction
Justin Lidard (Princeton University), Anirudha Majumdar (Princeton University)
Robotic IntelligenceReinforcement LearningImage
🎯 What it does: Proposes a risk-calibrated interactive planning framework (RCIP), which utilizes set-valued intent prediction and statistical risk calibration to automatically decide when to request human assistance when human intent is uncertain.
RL2AC: Reinforcement Learning-based Rapid Online Adaptive Control for Legged Robot Robust Locomotion
Shangke Lyu (Westlake University), Donglin Wang (Westlake University)
Robotic IntelligenceReinforcement LearningAuto Encoder
🎯 What it does: Propose a fast online adaptive control algorithm based on reinforcement learning (RL2AC), which achieves robust motion for legged robots under unknown loads, external disturbances, simulation-to-reality gaps, and varying terrain conditions by adding adaptive feedforward compensation to existing RL control policies.
RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots
Soroush Nasiriany, Yuke Zhu (The University Of Texas At Austin)
Robotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningDiffusion modelImageTextMultimodalityMeshSequential
🎯 What it does: Built an open-source large-scale simulation framework named RoboCasa for training general-purpose robots in everyday home environments, providing diverse kitchen scenarios, 3D assets, tasks, and datasets.
RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing
Bo Ai (Stanford University), Jiajun Wu (Stanford University)
OptimizationRobotic IntelligenceRecurrent Neural NetworkGraph Neural NetworkAuto EncoderImageMultimodalityPoint Cloud
🎯 What it does: Developed a robot dynamics modeling framework called RoboPack, which integrates visual and tactile information to address manipulation tasks with high occlusion and unknown physical properties, such as dense packing; and implemented model-based predictive control on a real robot.
RT-H: Action Hierarchies using Language
Suneel Belkhale (Google DeepMind), Dorsa Sadigh (Google DeepMind)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerLarge Language ModelVision-Language-Action ModelMultimodality
🎯 What it does: Propose the RT-H model, which utilizes a language action hierarchy (e.g., 'move arm forward') to first predict language motion under visual+language conditions, and then generate low-level robot actions based on this motion, thereby improving multi-task learning and interpretability.
RVT-2: Learning Precise Manipulation from Few Demonstrations
Ankit Goyal (NVIDIA), Dieter Fox (NVIDIA)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerVision-Language-Action ModelImagePoint Cloud
🎯 What it does: Studied a robot system called RVT-2 that can perform multi-task high-precision 3D manipulation using language instructions with only a small number of demonstrations.
Safe & Accurate at Speed with Tendons: A Robot Arm for Exploring Dynamic Motion
Simon Guist (Max Planck Institute for Intelligent Systems), Dieter Büchler
Robotic IntelligenceReinforcement LearningTabularTime Series
🎯 What it does: Designed and implemented a 4-degree-of-freedom cable-driven robotic arm named PAMY2, capable of achieving high-speed, precise, and safe dynamic motion.
Safe Planning for Articulated Robots Using Reachability-based Obstacle Avoidance With Spheres
Jonathan Michaux (University of Michigan), Ram Vasudevan (University of Michigan)
OptimizationRobotic Intelligence
🎯 What it does: Propose a real-time safe trajectory planning method (SPARROWS) based on reachability and spherical occupancy, enabling safe motion of multi-arm robots in crowded environments.
SAGE: Bridging Semantic and Actionable Parts for GEneralizable Articulated-Object Manipulation under Language Instructions
Haoran Geng (Stanford University), Leonidas Guibas
SegmentationPose EstimationRobotic IntelligenceLarge Language ModelVision Language ModelVision-Language-Action ModelImageTextPoint Cloud
🎯 What it does: SAGE is a general robotic framework for multi-category, natural language instruction-based articulated object manipulation, capable of first identifying object semantic parts, then mapping them to generalizable actionable parts (GAParts), and completing tasks through action program planning and execution trajectories, supporting interactive feedback to correct failures;
Scalable Distance-based Multi-Agent Relative State Estimation via Block Multiconvex Optimization
Tianyue Wu (Zhejiang University), Fei Gao (Zhejiang University)
OptimizationGraph
🎯 What it does: Proposed a distance-based relative state estimation method for large-scale multi-agent systems, based on a generic graph realization framework and designed a decomposable multi-convex optimization model.
ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection
Arpit Bahety (University of Texas at Austin), Roberto Martín-Martín (University of Texas at Austin)
OptimizationRobotic IntelligenceConvolutional Neural NetworkTransformerReinforcement LearningVideoPoint Cloud
🎯 What it does: A bimanual imitation learning framework named SCREWMIMIC based on helical axis projection is studied, which can learn bimanual collaborative manipulation from a single human video demonstration and refine execution policies through self-supervised interaction.
SEEK: Semantic Reasoning for Object Goal Navigation in Real World Inspection Tasks
Muhammad Fadhil Ginting, Ali-akbar Agha-mohammadi (Stanford University)
Robotic IntelligenceGraph Neural NetworkLarge Language ModelReinforcement LearningSimultaneous Localization and MappingImageTextPoint Cloud
🎯 What it does: This paper proposes the SEEK framework, which utilizes dynamic scene graphs and relation semantic networks to achieve object navigation based on semantic reasoning, significantly improving the inspection efficiency of robots in real environments.
Set It Up!: Functional Object Arrangement with Compositional Generative Models
Yiqing Xu (National University of Singapore), David Hsu (Massachusetts Institute of Technology)
GenerationRobotic IntelligenceTransformerLarge Language ModelDiffusion modelText
🎯 What it does: Propose the SetItUp framework, enabling robots to automatically generate functional desktop object arrangement plans based on vague human instructions. The framework first uses a large language model (LLM) to reason about abstract spatial relationships, then generates specific object poses satisfying these relationships through compositional diffusion models.
SpringGrasp: Synthesizing Compliant, Dexterous Grasps under Shape Uncertainty
Sirui Chen (Stanford University), Karen Liu (Stanford University)
OptimizationRobotic IntelligencePoint Cloud
🎯 What it does: Developed and implemented the SpringGrasp planner, which generates flexible grasping strategies from partial, noisy point clouds and was validated on real Allegro hands and Kuka robots.
Stein Variational Ergodic Search
Darrick Lee (University of Oxford), Ian Abraham (Yale University)
OptimizationRobotic Intelligence
🎯 What it does: This study proposes a 'Stein variational ergodic search' method based on Stein variational inference, which can simultaneously optimize multiple coverage paths in the continuous domain, enabling robots to perform parallel learning and online adaptation of different exploration strategies;
Tactile-Driven Non-Prehensile Object Manipulation via Extrinsic Contact Mode Control
Miquel Oller (University of Michigan), Nima Fazeli (University of Michigan)
OptimizationRobotic IntelligenceMultimodalityTabular
🎯 What it does: Design a non-grasping (non-prehensile) object manipulation algorithm based on contact patterns using high-resolution, flexible tactile sensors (Soft Bubbles) and a gradient-differentiable elastic model, enabling precise control of pose and force by driving other objects through object grasping on a plane.
Task Adaptation in Industrial Human-Robot Interaction: Leveraging Riemannian Motion Policies
Mike Allenspach (ETH Zurich), Roland Siegwart (ETH Zurich)
Robotic IntelligenceReinforcement LearningTime SeriesSequential
🎯 What it does: Proposed a fully autonomous motion control framework that utilizes Riemannian Motion Policies (RMPs) to achieve dynamic task planning and adaptation in industrial human-robot interaction, eliminating the need for manual robot control.
The Benefits of Sound Resound: An In-Person Replication of the Ability of Character-Like Robot Sound to Improve Perceived Social Warmth
Nnamdi Nwagwu (Oregon State University), Naomi T. Fitter (Oregon State University)
Robotic IntelligenceTextMultimodalityAudio
🎯 What it does: In a laboratory setting, the Hello Robot Stretch robot was used to perform LEGO assembly tasks, comparing the effects of three sound configurations (silent, functional sound, and characterized sound) on humans' perception of the robot's social attributes, localization awareness, and purchase value.
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Wilbert Pumacay (Universidad Católica San Pablo), Dieter Fox (University of Washington)
Robotic IntelligenceTransformerVision-Language-Action ModelImageTextMeshBenchmark
🎯 What it does: Propose THE COLOSSEUM, a large-scale bidirectional benchmark combining simulation and physical experiments, which includes 20 robotic manipulation tasks and 14-dimensional environmental perturbations (color, texture, size, lighting, camera pose, physical properties, etc.), along with corresponding 3D-printed physical experiments.
Tilde: Teleoperation for Dexterous In-Hand Manipulation Learning with a DeltaHand
Zilin Si (Carnegie Mellon University), Oliver Kroemer (Carnegie Mellon University)
Robotic IntelligenceReinforcement Learning from Human FeedbackConvolutional Neural NetworkReinforcement LearningDiffusion modelImage
🎯 What it does: Proposed a simulation learning system called Tilde based on a soft customizable DeltaHand, integrating a dual-link teleoperation interface TeleHand and a vision-conditioned diffusion strategy to complete seven high-difficulty in-hand manipulation tasks
Towards Tight Convex Relaxations for Contact-Rich Manipulation
Bernhard Paus Graesdal (Massachusetts Institute of Technology), Russ Tedrake (Massachusetts Institute of Technology)
OptimizationRobotic Intelligence
🎯 What it does: This paper proposes a method that utilizes compact convex relaxation to achieve global motion planning, capable of simultaneously handling discrete mode switching and continuous dynamics in contact-rich robotic systems;
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
Cheng Chi (Stanford University), Shuran Song (Stanford University)
Domain AdaptationRobotic IntelligenceReinforcement Learning from Human FeedbackDiffusion modelSimultaneous Localization and MappingVideoMultimodality
🎯 What it does: This paper proposes an end-to-end framework named Universal Manipulation Interface (UMI), which can collect human demonstrations using a GoPro-equipped handheld gripper and directly train visual-motor policies deployable across multiple robot platforms.
URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images
Qiuyu Chen, Abhishek Gupta (University Of Washington)
Data SynthesisDomain AdaptationRobotic IntelligenceTransformerReinforcement LearningVision-Language-Action ModelDiffusion modelImageTextMesh
🎯 What it does: This paper proposes the URDFormer pipeline, which can directly map real-world RGB images into URDF robotic arm scenes usable in simulations, achieving a zero-shot real-to-sim-to-real workflow from internet images to trainable robot control policies.
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
Vidhi Jain (Google DeepMind Robotics Carnegie Mellon University), Debidatta Dwibedi (Google DeepMind Robotics Carnegie Mellon University)
Robotic IntelligenceTransformerVision-Language-Action ModelContrastive LearningVideoText
🎯 What it does: Developed an end-to-end video-conditioned robot policy, Vid2Robot, which utilizes human demonstration videos to guide robots in performing the same tasks.
VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation
Wentao Zhao (Shandong University), Wei Zhang (Shandong University)
Robotic IntelligenceTransformerVision Language ModelVideoTextMultimodality
🎯 What it does: A new robotic manipulation framework called VLMPC is proposed, which combines vision-language models (VLM) and model predictive control (MPC) to enhance robotic operation capabilities in complex scenarios.
Who Plays First? Optimizing the Order of Play in Stackelberg Games with Many Robots
Haimin Hu (Princeton University), Jaime Fernández Fisac (Princeton University)
OptimizationRobotic Intelligence
🎯 What it does: This paper studies how to automatically find the community-optimal game order (i.e., the sequence of decision-making among robots) in multi-robot Stackelberg trajectory games and obtain the corresponding local Stackelberg equilibrium.
World Models for General Surgical Grasping
Hongbin Lin (Chinese University of Hong Kong)
SegmentationDepth EstimationCompressionRobotic IntelligenceReinforcement LearningWorld ModelImageVideoBiomedical Data
🎯 What it does: Designed and trained a deep reinforcement learning framework named GAS based on a world model, for pose-estimation-free grasping of unknown multiple surgical objects in robotic-assisted surgical environments.
Yell At Your Robot: Improving On-the-Fly from Language Corrections
Lucy Xiaoyang Shi (Stanford University), Chelsea Finn (Stanford University)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningVision-Language-Action ModelTextMultimodality
🎯 What it does: Proposes the YAY Robot framework, enabling robots to perform real-time adjustments and continuous learning through natural language corrections in long-term tasks.
You’ve Got to Feel It To Believe It: Multi-Modal Bayesian Inference for Semantic and Property Prediction
Parker Ewen (University of Michigan), Ram Vasudevan (University of Michigan)
Robotic IntelligenceImageMultimodalityPoint Cloud
🎯 What it does: This paper proposes a multimodal Bayesian inference framework that jointly estimates the semantic categories and physical attributes (such as friction coefficient) using visual and tactile information.