RSS 2024 Papers with AI Summaries
Robotics: Science and Systems · 134 papers
→ RSS 2024 papers with code (27)
Each paper below shows an AI-generated one-line summary. Get the full 6-part summary (innovation, method, data, results, limitations) and search all 134 RSS 2024 papers by keyword, author or institution —
free trial on arXivSub.
3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
Yanjie Ze (Shanghai Qizhi Institute), Huazhe Xu (Tsinghua University)
Robotic IntelligenceLarge Language ModelDiffusion modelPoint Cloud
🎯 What it does: Propose 3D Diffusion Policy (DP3), combining sparse point cloud encoding with diffusion policy to achieve low-sample visual imitation learning;
A Single Motor Nano Aerial Vehicle with Novel Peer-to-Peer Communication and Sensing Mechanism
Jingxian Wang (Northwestern University), Michael Rubenstein (University of Pennsylvania)
Robotic IntelligenceTime Series
🎯 What it does: Propose a single-motor, 20g MP3 drone equipped with a point-to-point communication and positioning system based on infrared light;
A Trajectory Tracking Algorithm for the LSMS Family of Cable-Driven Cranes
Javier Puig-Navarro (Analytical Mechannics Associates), B. Danette Allen (NASA Langley Research Center)
Robotic IntelligenceTime Series
🎯 What it does: Aiming at the lightweight surface manipulator system (LSMS) series cable-driven cranes, this paper proposes a control algorithm that achieves precise trajectory tracking, which was experimentally validated on the LSMS-L35 hardware platform.
AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation
Kaifeng Zhang (University of Illinois Urbana Champaign), Yunzhu Li (University of Illinois Urbana Champaign)
OptimizationRobotic IntelligenceGraph Neural NetworkPoint CloudTime SeriesSequentialPhysics Related
🎯 What it does: Propose a unified graph neural network dynamics framework called AdaptiGraph, which can online learn and predict deformation dynamics of various materials (ropes, granules, rigid boxes, fabrics) under unknown physical properties, and achieve adaptive control on real robots.
Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learning
Xinyang Gu (Robotera Technology Co Ltd), Jianyu Chen (Tsinghua University)
Robotic IntelligenceRecurrent Neural NetworkWorld ModelSequential
🎯 What it does: This paper proposes the Denoising World Model Learning (DWL) framework, achieving end-to-end reinforcement learning control. It successfully enables two sizes of humanoid robots to walk in complex real-world environments such as snow, slopes, stairs, and irregular terrains, achieving zero-shot simulation-to-real direct transfer.
Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion
Tairan He (Carnegie Mellon University), Guanya Shi (Carnegie Mellon University)
Robotic IntelligenceReinforcement LearningImage
🎯 What it does: Proposed a dual-strategy framework ABS, achieving high-speed and collision-free walking for quadruped robots in crowded environments
An abstract theory of sensor eventification
Yulin Zhang (Amazon Robotics), Dylan Shell (Texas A&M University)
OptimizationComputational Efficiency
🎯 What it does: Proposes the abstract theory of 'sensor eventification' and provides a formal framework based on relations and variators to determine whether traditional sensors can obtain sufficient information through eventification.
Any-point Trajectory Modeling for Policy Learning
Chuan Wen (UC Berkeley), Pieter Abbeel (UC Berkeley)
Robotic IntelligenceTransformerVision-Language-Action ModelDiffusion modelVideoTextMultimodalityBenchmark
🎯 What it does: Propose the Any-point Trajectory Modeling (ATM) framework, which achieves more efficient robot learning by pretraining trajectory prediction models on unlabeled videos and using predicted trajectories to guide policy learning with limited action demonstrations.
AnyFeature-VSLAM: Automating the Usage of Any Feature into Visual SLAM
Alejandro Fontan, Michael Milford (Queensland University of Technology)
Pose EstimationSimultaneous Localization and MappingImagePoint Cloud
🎯 What it does: Designed AnyFeature-VSLAM, a VSLAM system that can automatically switch between any visual features without manual parameter tuning.
AutoGPT+P: Affordance-based Task Planning using Large Language Models
Timo Birr (Karlsruhe Institute of Technology), Tamim Asfour (Karlsruhe Institute of Technology)
Robotic IntelligenceTransformerLarge Language ModelAgentic AIPrompt EngineeringImageTextMultimodality
🎯 What it does: Propose the AutoGPT+P system, which integrates affordance-based scene representation with large language models to achieve natural language task planning and support dynamic recovery and alternative solutions for missing objects.
AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries
Bingjie Tang (University of Southern California), Yashraj Narang (NVIDIA Corporation)
Data-Centric LearningRobotic IntelligenceReinforcement LearningMeshTime Series
🎯 What it does: Built the AutoMate framework and complete system, including a dataset of 100 collision-free, 3D-printable two-part assemblies, corresponding parallel simulation environments, and trained part-specific and generalist assembly strategies on this foundation. Finally, achieved zero-shot sim-to-real transfer, including perception-initialized assembly processes.
Blending Data-Driven Priors in Dynamic Games
Justin Lidard (Princeton University), Jaime Fernández Fisac (Princeton University)
Autonomous DrivingOptimizationReinforcement LearningSequential
🎯 What it does: Propose the KLGame framework, combining data-driven reference strategies with KL-regularized dynamic game planning to achieve multi-modal feedback Nash equilibrium strategies.
Broadcasting Support Relations Recursively from Local Dynamics for Object Retrieval in Clutters
Yitong Li (Peking University), Hao Dong (Peking University)
RetrievalConvolutional Neural NetworkGraph Neural NetworkAuto EncoderPoint Cloud
🎯 What it does: Propose a support relationship construction framework based on local dynamic recursive broadcasting for safe target object retrieval in cluttered stacks.
CLOSURE: Fast Quantification of Pose Uncertainty Sets
Yihuai Gao (Stanford University), Heng Yang (Harvard University)
Pose EstimationComputational EfficiencyImagePoint Cloud
🎯 What it does: Proposed and implemented a GPU-accelerated algorithm called CLOSURE to real-time quantify the minimal enclosing ball (MEGB) of the 6D pose uncertainty set (PURSE) under noisy observations generated by deep learning, thereby providing the worst-case error bound for pose estimation.
Collaborative Planar Pushing of Polytopic Objects with Multiple Robots in Complex Scenes
Zili Tang (Peking University), Meng Guo (Peking University)
OptimizationRobotic Intelligence
🎯 What it does: This paper proposes a complete framework for multi-robot collaborative pushing of arbitrary polygonal objects in complex obstacle environments; the core idea is to generate appropriate contact patterns through sparse optimization based on multi-direction feasibility estimation, and to obtain an executable hybrid plan via hierarchical keyframe-guided hybrid search (KG-HS). Subsequently, nonlinear model predictive control (NMPC) is utilized to achieve trajectory tracking and mode switching, and robustness is ensured during execution through event-triggered replanning.
Collision-Affording Point Trees: SIMD-Amenable Nearest Neighbors for Fast Motion Planning with Pointclouds
Clayton Ramsey, Lydia E Kavraki
Autonomous DrivingComputational EfficiencyRobotic IntelligencePoint Cloud
🎯 What it does: A data structure named Collision-Tolerant Point Tree (CAPT) was constructed for fast and accurate collision detection on perceived point clouds, with the implementation of SIMD parallel queries;
Computation-Aware Learning for Stable Control with Gaussian Process
Wenhan Cao (University of Manchester), Wei Pan (University of Manchester)
OptimizationComputational EfficiencyRobotic IntelligenceTabularTime Series
🎯 What it does: This paper proposes a 'computational-aware' learning and control framework tailored for robotic systems with limited computational resources. It first quantifies computational errors (computational uncertainty) in Gaussian process dynamics model learning, then utilizes this uncertainty for stability analysis (ROA estimation) and designs a controller with minimal input under the control Lyapunov function framework (SOCP or explicit control formulas).
Configuration Space Distance Fields for Manipulation Planning
Yiming Li (Idiap Research Institute), Sylvain Calinon (Idiap Research Institute)
OptimizationRobotic IntelligencePoint CloudMesh
🎯 What it does: Propose defining an unsigned Signed Distance Field (CDF) in the robot's configuration space and applying it to whole-body inverse kinematics and manipulation planning.
Conformalized Teleoperation: Confidently Mapping Human Inputs to High-Dimensional Robot Actions
Michelle D Zhao (Carnegie Mellon University), Andrea Bajcsy (Carnegie Mellon University)
Robotic IntelligenceTabular
🎯 What it does: Designed and implemented a low-dimensional to high-dimensional mapping model for assistive robotic arms based on adaptive conformal prediction, which can provide calibrated confidence intervals during deployment and real-time detection of high uncertainty states.
Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation
Aaditya Prasad (Stanford University), Jeannette Bohg (Stanford University)
Robotic IntelligenceDiffusion modelImage
🎯 What it does: Propose Consistency Policy, a low-latency visual motion control strategy derived from a pre-trained Diffusion Policy through self-consistency distillation.
Constraint-Aware Intent Estimation for Dynamic Human-Robot Object Co-Manipulation
Yifei Simon Shao (University of Pennsylvania), Nadia Figueroa (University of Pennsylvania)
Robotic IntelligenceTabularTime Series
🎯 What it does: This paper proposes a constraint-aware intention estimation and control framework for dynamic human-robot collaborative object manipulation;
Construction of a Multiple-DOF Underactuated Gripper with Force-Sensing via Deep Learning
Jihao Li (Zhejiang University), HUIXU DONG (Nanyang Technological University)
Robotic IntelligenceRecurrent Neural NetworkSupervised Fine-TuningTime Series
🎯 What it does: Designed and implemented a low-cost passive gripper GL-Robot driven by a single motor, utilizing a five-bar linkage structure to achieve parallel and enveloping grasping, and implementing closed-loop control without force sensing through LSTM + statistics.
ConTac: Continuum-Emulated Soft Skinned Arm with Vision-based Shape Sensing and Contact-aware Manipulation
Tuan Tai Nguyen (Japan Advanced Institute of Science and Technology), Van Ho (Hanoi University of Industry)
Domain AdaptationRobotic IntelligenceConvolutional Neural NetworkImage
🎯 What it does: Designed and implemented the ConTac platform, utilizing a single camera for shape reconstruction and contact detection on soft skin, and achieving zero-shot simulation-to-reality transfer through deep learning models, further applied to adaptive safety control and contact-based operations.
CraterGrader: Autonomous Robotic Terrain Manipulation for Lunar Site Preparation and Earthmoving
Ryan Lee (Carnegie Mellon University), William Whittaker (Carnegie Mellon University)
OptimizationRobotic IntelligenceSimultaneous Localization and MappingImagePoint CloudPhysics Related
🎯 What it does: Developed an autonomous robot named CraterGrader capable of performing earthwork tasks such as terrain leveling and crater filling in a sandbox environment simulating lunar soil, achieving full automation in terrain planning, perception, localization, and execution.
Decentralized Multi-Robot Line-of-Sight Connectivity Maintenance under Uncertainty
Yupeng Yang (University of North Carolina at Charlotte), Wenhao Luo (University of North Carolina at Charlotte)
OptimizationRobotic Intelligence
🎯 What it does: Proposes a decentralized control method to maintain line-of-sight (LOS) connectivity among multiple robots under Gaussian localization uncertainty.
Demonstrating Adaptive Mobile Manipulation in Retail Environments
Max Spahn (Delft University of Technology), Martijn Wisse (Delft University of Technology)
OptimizationRobotic IntelligenceTransformerLarge Language ModelVision Language ModelImageText
🎯 What it does: Developed and verified a mobile grasping robot platform suitable for retail environments, achieving a complete closed-loop system for order picking, navigation, perception, and teaching.
Demonstrating Agile Flight from Pixels without State Estimation
Ismail Geles (University of Zurich), Davide Scaramuzza (University of Zurich)
Autonomous DrivingReinforcement Learning from Human FeedbackTransformerReinforcement LearningImageVideo
🎯 What it does: Trained and deployed a pixel-based visual controller enabling quadrotor drones to perform high-speed racing on tracks without requiring state estimation;
Demonstrating Arena 3.0: Advancing Social Navigation in Collaborative and Highly Dynamic Environments
Linh Kästner (Technical University Berlin), Jens Lambrecht (Technical University Berlin)
Robotic IntelligenceWorld ModelBenchmark
🎯 What it does: Develop the Arena 3.0 platform, integrating realistic crowd simulation, social force models, dynamic task generation, robot navigation kits, and achieving cross-platform abstraction and unified API on three simulators: Flatland, Gazebo, and Unity;
Demonstrating CropFollow++: Robust Under-Canopy Navigation with Keypoints
Arun Narenthiran Sivakumar (University of Illinois Urbana Champaign), Girish Chowdhary (University of Illinois Urbana Champaign)
Pose EstimationAutonomous DrivingConvolutional Neural NetworkImageAgriculture Related
🎯 What it does: Proposed the CropFollow++ system, which utilizes RGB cameras to predict semantic keypoints for low-altitude crop row navigation in farmland;
Demonstrating Event-Triggered Investigation and Sample Collection for Human Scientists using Field Robots and Large Foundation Models
Tirthankar Bandyopadhyay (CSIRO Robotics, Data61), Stanislav Funiak (CSIRO Robotics, Data61)
Robotic IntelligenceTransformerLarge Language ModelVision-Language-Action ModelSimultaneous Localization and MappingImageTextPoint Cloud
🎯 What it does: An end-to-end event-triggered science exploration and sampling system based on a robot team, vibration sensor network, and large foundation model (LFM) was constructed and demonstrated in a simulated lunar surface sandbox, showcasing the robot's capability to perform exploration, localization, sampling, semantic mapping, and whole-body manipulation under natural language interaction.
Demonstrating HOUND: A Low-cost Research Platform for High-speed Off-road Underactuated Nonholonomic Driving
Sidharth Talia, Siddhartha Srinivasa
Autonomous DrivingOptimizationSimultaneous Localization and MappingBenchmark
🎯 What it does: Proposed a 1/10 scale, low-cost HOUND platform for high-speed off-road autonomous vehicle research and experimentation.
Demonstrating HumanTHOR: A Simulation Platform and Benchmark for Human-Robot Collaboration in a Shared Workspace
Chenxu Wang (Tsinghua University), Huaping Liu (Tsinghua University)
Robotic IntelligenceLarge Language ModelVision-Language-Action ModelImageTextMultimodalityBenchmark
🎯 What it does: Constructed the HumanTHOR simulation platform, supporting real-time VR interaction and multimodal communication in shared workspaces for human-robot collaboration, and benchmarked a series of daily tasks.
Demonstrating Language-Grounded Motion Controller
Ravi Tejwani (Massachusetts Institute of Technology), Haruhiko Asada
Robotic IntelligenceGraph Neural NetworkTime SeriesAudio
🎯 What it does: Proposed a language-grounded motion controller that synchronizes robot physical guidance with voice commands in real-time according to the user's level of cooperation, enhancing the naturalness of human-robot collaboration.
Demonstrating Learning from Humans on Open-Source Dexterous Robot Hands
Kenneth Shaw (Carnegie Mellon University), Deepak Pathak (Carnegie Mellon University)
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningVision Language ModelOptical FlowVideo
🎯 What it does: Presented three low-cost open-source humanoid robot hands (LEAP Hand, DASH Hand, LEAP Hand v2) at RSS 2024, and demonstrated learning methods based on human videos, optical capture, and reinforcement learning.
Demonstrating OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics
Peiqi Liu (New York University), Lerrel Pinto (New York University)
Robotic IntelligenceTransformerVision Language ModelSimultaneous Localization and MappingImageTextPoint Cloud
🎯 What it does: Built an open-source knowledge robot system called OK-Robot, which can complete the full pick-and-drop task of opening-recognizing-navigating-grasping-placing in new home environments through zero training.
Design and Control of a Bipedal Robotic Character
Ruben Grandia (Disney Research), Moritz Bächer
Robotic IntelligenceReinforcement LearningVideo
🎯 What it does: Designed and implemented a bipedal robot for entertainment applications, integrating reinforcement learning control, animation engine, and remote puppeteering interface to achieve expressive and dynamically stable motion performance.
Developing Design Guidelines for Older Adults with Robot Learning from Demonstration
Erin Hedlund-Botti (Georgia Institute of Technology), Matthew Gombolay (Georgia Institute of Technology)
Robotic IntelligenceImageVideo
🎯 What it does: Compare young adults and elderly adults by using Learning from Demonstration (LfD) to teach them to instruct Boston Dynamics Spot robots in two cleaning tasks, evaluate robot performance, usability, and participant experience, and develop design guidelines for the elderly based on the results.
DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
Chen Wang (Stanford University), Karen Liu
Pose EstimationRobotic IntelligenceDiffusion modelSimultaneous Localization and MappingPoint Cloud
🎯 What it does: Proposed DEXCAP, a portable hand motion capture system, and DEXIL, an imitation learning framework based on human hand MoCap data, enabling training of dual-arm robots without requiring robot hardware.
Differentiable Robust Model Predictive Control
Alex Oshin (Georgia Institute of Technology), Evangelos Theodorou (Georgia Institute of Technology)
OptimizationRobotic Intelligence
🎯 What it does: This paper proposes a differentiable tube model predictive control (DT-MPC) framework, which utilizes the implicit function theorem to achieve online adaptive adjustment of tube MPC parameters (e.g., cost weights, obstacle clause weights), thereby enabling robust safe control in environments with large disturbances.
Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learning
Xiaoyu Zhang (University of Illinois at Urbana-Champaign), Saurabh Gupta (University of Illinois at Urbana-Champaign)
Robotic IntelligenceReinforcement Learning from Human FeedbackDiffusion modelAuto EncoderVideo
🎯 What it does: Propose Diffusion Meets DAgger (DMD), which uses diffusion models to generate out-of-distribution states for data augmentation, improving sample efficiency in endoscopic arm imitation learning.
Don't Start From Scratch: Behavioral Refinement via Interpolant-based Policy Diffusion
Kaiqi Chen (National University of Singapore), Harold Soh (National University of Singapore)
Robotic IntelligenceDiffusion modelScore-based ModelAuto EncoderStochastic Differential Equation
🎯 What it does: Designed a BRIDGER interpolation diffusion method that leverages information-rich source strategies for imitation learning, improving upon the limitations of traditional Gaussian noise diffusion.
DrEureka: Language Model Guided Sim-To-Real Transfer
Yecheng Jason Ma (University of Pennsylvania), Dinesh Jayaraman (University of Pennsylvania)
Robotic IntelligenceTransformerLarge Language ModelReinforcement Learning
🎯 What it does: Utilize large language models (LLMs) to automatically generate reward functions and domain randomization configurations, enabling robots to achieve human-intervention-free transfer from simulation to real-world environments.
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Alexander Khazatsky (Stanford University), Chelsea Finn (Stanford University)
Data-Centric LearningRobotic IntelligenceConvolutional Neural NetworkTransformerVision-Language-Action ModelDiffusion modelImageVideoMultimodalityBenchmark
🎯 What it does: Proposed and released DROID, a large-scale robot manipulation dataset collected in real-world environments, containing 76k demonstration trajectories, 350 hours of interaction data, 564 scenes, 86 tasks, along with a complete hardware platform, data collection protocol, and reproducible training code.
Dynamic Adversarial Attacks on Autonomous Driving Systems
Amirhosein Chahe (Drexel University), Lifeng Zhou (Drexel University)
Image TranslationObject DetectionAutonomous DrivingAdversarial AttackConvolutional Neural NetworkImagePoint Cloud
🎯 What it does: Designed and implemented dynamically displayed adversarial patches on mobile vehicle screens to deceive the target detection models of autonomous driving systems into misclassifying traffic signs, thereby influencing their decision-making;
Dynamic On-Palm Manipulation via Controlled Sliding
William Yang (University of Pennsylvania), Michael Posa (University of Pennsylvania)
OptimizationRobotic Intelligence
🎯 What it does: Propose a framework for dynamic non-grasping operations on the palm through controlled sliding, achieving fast plate pickup and placement tasks on a 3D robotic arm.
Efficient Data Collection for Robotic Manipulation via Compositional Generalization
Jensen Gao (Stanford University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceSupervised Fine-TuningDiffusion modelImage
🎯 What it does: This study investigates the compositional generalization capability of visual imitation learning strategies in robot manipulation, and based on this, designs efficient domain-internal data collection strategies aimed at reducing data collection costs while improving the success rate of models in unseen environmental combinations.
Event-based Visual Inertial Velometer
Xiuyuan LU (Hong Kong University of Science and Technology), Shaojie Shen (Hong Kong University of Science and Technology)
Pose EstimationDepth EstimationOptimizationSimultaneous Localization and MappingMultimodality
🎯 What it does: Designed and implemented an event camera and IMU fusion velocity estimation system (velometer) that directly recovers linear velocity through event normal flow and depth estimation.
Evolution and learning in differentiable robots
Luke Strgar (Northwestern University), Sam Kriegman (Northwestern University)
OptimizationRobotic IntelligenceMesh
🎯 What it does: By evolving a massive (10,000) robot population in a differentiable physics simulation environment, and combining gradient descent to learn each robot's own feedforward neural network controller, the study generates and trains millions of robots with diverse morphologies and behaviors over thousands of generations.
Experience-based multi-agent path finding with narrow corridors
Rachel A Moan, Kris Hauser (University of Illinois at Urbana-Champaign)
OptimizationRobotic IntelligenceBenchmark
🎯 What it does: Propose an experience-driven multi-robot path planning algorithm that efficiently resolves multi-robot path conflicts in narrow passages (width of 1).
Explore until Confident: Efficient Exploration for Embodied Question Answering
Allen Z. Ren (Princeton University), Dorsa Sadigh (Stanford University)
Robotic IntelligencePrompt EngineeringVision Language ModelSimultaneous Localization and MappingMultimodality
🎯 What it does: This paper studies Embodied Question Answering (EQA), proposing to use large-scale vision-language models (VLMs) to construct semantic maps in unknown indoor environments and guide robots for efficient exploration. Ultimately, multi-step conformal prediction is employed for confidence calibration, determining when to stop exploration and provide answers.
Expressive Whole-Body Control for Humanoid Robots
Xuxin Cheng (UC San Diego), Xiaolong Wang (UC San Diego)
Robotic IntelligenceReinforcement LearningVideoText
🎯 What it does: Trained a control strategy that enables a humanoid robot (Unitree H1) to perform diverse, expressive full-body movements in real environments through deep reinforcement learning combined with large-scale human motion capture data.
FLAIR: Feeding via Long-Horizon AcquIsition of Realistic dishes
Rajat Kumar Jenamani (Cornell University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceTransformerLarge Language ModelPrompt EngineeringVision Language ModelVision-Language-Action ModelImageTextRetrieval-Augmented Generation
🎯 What it does: This paper proposes the FLAIR system, which enables robots to perform long-term, customizable food grasping and delivery according to user preferences.
From Compliant to Rigid Contact Simulation: a Unified and Efficient Approach
Justin Carpentier (Inria), Louis Montaut (Inria)
OptimizationRobotic IntelligenceBenchmarkPhysics Related
🎯 What it does: Designed and implemented a unified contact solver based on ADMM and proximal optimization, capable of simultaneously handling rigid and elastic contact without requiring physical relaxation.
Function Based Sim-to-Real Learning for Shape Control of Deformable Free-form Surfaces
Yingjun Tian (Chinese University of Hong Kong), Charlie C. L. Wang
Domain AdaptationRobotic IntelligencePoint CloudMeshSequential
🎯 What it does: This paper proposes a function-based simulation-to-real learning method for shape control of deformable free-form surfaces, integrating it with neural network forward and inverse kinematics pipelines.
Goal-Reaching Trajectory Design Near Danger with Piecewise Affine Reach-avoid Computation
Long Kiu Chung (Georgia Institute of Technology), Shreyas Kousik (Georgia Institute of Technology)
Autonomous DrivingOptimizationRobotic Intelligence
🎯 What it does: Propose a safety-guaranteed trajectory design method called PARC based on a piecewise affine model, which can achieve collision-free target arrival in scenarios close to danger.
GOAT: GO to Any Thing
Matthew Chang (Fair), Devendra Singh Chaplot (Meta)
Object DetectionSegmentationRobotic IntelligenceConvolutional Neural NetworkTransformerVision Language ModelSimultaneous Localization and MappingImageTextPoint Cloud
🎯 What it does: Propose a general-purpose navigation system called GOAT that can perform multimodal target navigation (category labels, images, language descriptions) in real home environments and has lifelong learning and cross-platform deployment capabilities.
GRaCE: Balancing Multiple Criteria to Achieve Stable, Collision-Free, and Functional Grasps
Tasbolat Taunyazov (National University of Singapore), Harold Soh (National University of Singapore)
OptimizationRobotic IntelligenceTransformerPoint CloudMesh
🎯 What it does: This paper proposes the GRACE framework, which balances multiple grasping criteria through hierarchical rules and an expected utility function, achieving executable, stable, collision-free, and functional grasps.
HACMan++: Spatially-Grounded Motion Primitives for Manipulation
Bowen Jiang (Carnegie Mellon University), David Held (Carnegie Mellon University)
Robotic IntelligenceReinforcement LearningPoint Cloud
🎯 What it does: Proposed the HACMan++ framework, utilizing spatial ground parameterized motion primitives to enable robots to complete long-term, precision spatial reasoning tasks by chaining multiple primitives.
Hamilton-Jacobi Reachability Analysis for Hybrid Systems with Controlled and Forced Transitions
Javier Borquez (University of Southern California), Somil Bansal (University of Southern California)
Autonomous DrivingOptimizationRobotic IntelligenceTime SeriesSequential
🎯 What it does: Proposed an extended Hamilton-Jacobi (HJ) reachability framework for handling hybrid systems with controlled and forced discrete switches, capable of computing backward reachable tubes (BRT) and providing optimal control strategies;
Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation
Abdelrhman Werby (University of Freiburg), Wolfram Burgard (University of Technology Nuremberg)
Robotic IntelligenceVision-Language-Action ModelSimultaneous Localization and MappingTextPoint Cloud
🎯 What it does: Propose HOV-SG, a hierarchical open-vocabulary 3D scene graph for language-based robot navigation.
Homotopic Path Set Planning for Robot Manipulation and Navigation
Jing Huang (Chinese University of Hong Kong)
OptimizationRobotic IntelligenceImagePoint Cloud
🎯 What it does: This paper proposes a homotopy path set planning method for robot manipulation and navigation, covering sparse channel detection, channel-aware optimal path planning, and path set generation based on path transfer.
HRP: Human affordances for Robotic Pre-training
Mohan Kumar Srirama (Carnegie Mellon University), Abhinav Gupta (Carnegie Mellon University)
Representation LearningRobotic IntelligenceTransformerSupervised Fine-TuningAuto EncoderContrastive LearningImageVideo
🎯 What it does: Leverage automatically extracted hand, object, and contact affordance annotations from internet human videos to perform semi-supervised pre-training on existing visual encoders, enhancing downstream manipulation performance in robot vision.
Human-oriented Representation Learning for Robotic Manipulation
Mingxiao Huo (University Of Berkeley), Wei Zhan (University Of Berkeley)
Representation LearningRobotic IntelligenceConvolutional Neural NetworkTransformerSupervised Fine-TuningVision-Language-Action ModelVideo
🎯 What it does: Propose a Task Fusion Decoder (TFD) module, which achieves more transferable robot manipulation visual representations by performing multi-task fine-tuning related to human perception on pre-trained visual encoders.
HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation
Carmelo Sferrazza (UC Berkeley), Pieter Abbeel (UC Berkeley)
Robotic IntelligenceReinforcement LearningBenchmark
🎯 What it does: Proposed HumanoidBench—a high-dimensional simulation humanoid robot benchmark containing 15 whole-body manipulation tasks and 12 locomotion tasks;
iHERO: Interactive Human-oriented Exploration and Supervision Under Scarce Communication
Zhuoli Tian (Peking University), Meng Guo (Peking University)
OptimizationRobotic IntelligenceSimultaneous Localization and MappingImage
🎯 What it does: Designed a discrete interaction-based multi-robot collaborative exploration and supervision framework (iHERO) under scarce communication conditions, enabling robots to actively return and update human-robot interaction information while maintaining low communication latency;
iMESA: Incremental Distributed Optimization for Collaborative Simultaneous Localization and Mapping
Daniel McGann (Carnegie Mellon University), Michael Kaess (Carnegie Mellon University)
OptimizationRobotic IntelligenceSimultaneous Localization and Mapping
🎯 What it does: Proposed and implemented iMESA, an incremental distributed backend for collaborative SLAM, capable of achieving real-time and accurate global state estimation under sparse communication conditions.
Imitation Bootstrapped Reinforcement Learning
Hengyuan Hu (Stanford University), Dorsa Sadigh (Stanford University)
Robotic IntelligenceTransformerReinforcement LearningImage
🎯 What it does: This study proposes a framework called 'Imitation Learning Bootstrapped Reinforcement Learning' (IBRL), which accelerates exploration and target estimation in reinforcement learning by utilizing action candidates provided by an independently trained imitation learning (IL) policy, thereby improving sample efficiency.
Implicit Graph Search for Planning on Graphs of Convex Sets
Ramkumar Natarajan (Carnegie Mellon University), Maxim Likhachev (Carnegie Mellon University)
OptimizationRobotic IntelligenceGraph
🎯 What it does: This paper proposes two implicit graph search methods (IxG and IxG*) for planning smooth, collision-avoiding motion trajectories on Graphs of Convex Sets (GCS), utilizing a search-optimization interleaving strategy to decompose the originally large-scale MICP solved across the entire GCS into a combination of graph search and local convex trajectory optimization.
INTERPRET: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning
Muzhi Han (University of California Los Angeles), Yuke Zhu (University of Texas at Austin)
Robotic IntelligenceReinforcement Learning from Human FeedbackLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: This study proposes the InterPreT framework, which utilizes natural language feedback provided by human non-experts during robot interaction. It leverages GPT-4 to generate and iteratively refine symbolic predicates in the form of Python functions, then learns symbolic operators and compiles them in real-time into PDDL domains, enabling robots to perform long-horizon, provably correct planning in unknown tasks.
JIGGLE: An Active Sensing Framework for Boundary Parameters Estimation in Deformable Surgical Environments
Nikhil Uday Shinde (University of California San Diego), Michael C. Yip (University of California San Diego)
Data SynthesisRobotic IntelligenceBiomedical Data
🎯 What it does: Propose the JIGGLE framework in deformable surgical environments for online probabilistic estimation of tissue boundary parameters and achieving safer tissue manipulation through active sensing to gain information;
Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics
Norman Di Palo (Imperial College London), Edward Johns (Imperial College London)
Robotic IntelligenceMeta LearningTransformerLarge Language ModelPrompt EngineeringVision-Language-Action ModelImageMultimodalityPoint Cloud
🎯 What it does: Propose the Keypoint Action Tokens (KAT) framework, which achieves few-shot imitation learning without additional training by leveraging pre-trained large language models (e.g., GPT-4 Turbo); it converts visual observations and robotic arm poses into 3D keypoints and three-point pose tokens that can be processed by text Transformers, enabling autoregressive generation of continuous actions from a single observation.
Khronos: A Unified Approach for Spatio-Temporal Metric-Semantic SLAM in Dynamic Environments
Lukas Schmid (Massachusetts Institute of Technology), Luca Carlone (Massachusetts Institute of Technology)
OptimizationRobotic IntelligenceSimultaneous Localization and MappingPoint CloudTime Series
🎯 What it does: This paper proposes the Khronos system, achieving unified spatiotemporal metric semantic SLAM capable of real-time construction of dense 4D maps containing short-term dynamics and long-term changes.
Language-Augmented Symbolic Planner for Open-World Task Planning
Guanqi Chen (The University of Hong Kong), Jia Pan (The University of Hong Kong)
TransformerLarge Language ModelText
🎯 What it does: Leverage large language models to assist symbolic planners in accomplishing complex long-term tasks in open-world environments, automatically diagnosing and repairing execution errors;
Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering
Snehal Jauhri (TU Darmstadt), Georgia Chalvatzaki (TU Darmstadt)
Pose EstimationRobotic IntelligenceConvolutional Neural NetworkNeural Radiance FieldImage
🎯 What it does: Propose NeuGraspNet, a full-implicit 6DoF robotic grasping approach based on neural surface rendering, which can achieve grasping from any perspective in cluttered scenes using only a single-view depth map.
Learning Manipulation by Predicting Interaction
Jia Zeng (TeleAI China Telecom Corp Ltd), Hongyang Li (TeleAI China Telecom Corp Ltd)
Robotic IntelligenceTransformerVision-Language-Action ModelContrastive LearningVideoTextMultimodality
🎯 What it does: Designed a pre-training framework called MPI for interaction, which utilizes keyframes and language instructions to predict unseen interaction frames and detect interactive objects, thereby enhancing robotic manipulation representations.
Learning to Learn Faster from Human Feedback with Language Model Predictive Control
Jacky Liang, Carolina Parada
Robotic IntelligenceMeta LearningReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Studied enhancing the teachability of robot code-writing LLMs by fine-tuning large language models through language model predictive control (LMPC) to improve rapid learning from human feedback.
Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration
Xinzhu Liu (Tsinghua University), Huaping Liu (Tsinghua University)
Robotic IntelligenceTransformerLarge Language ModelMultimodality
🎯 What it does: Propose a decentralized, heterogeneous, adaptive robot team collaboration framework based on a large language model, enabling newly joined robots without prior coordination to join at any time and location, thereby improving the team's efficiency in completing tidying tasks.
Linear-time Differential Inverse Kinematics: an Augmented Lagrangian Perspective
Bruce Wingo (Georgia Institute of Technology), Justin Carpentier (Inria)
OptimizationRobotic Intelligence
🎯 What it does: Proposed a differential inverse kinematics solver called LOIK with linear complexity that can handle linear equality and inequality constraints;
Logic-Skill Programming: An Optimization-based Approach to Sequential Skill Planning
Teng Xue (Idiap Research Institute), Sylvain Calinon (Idiap Research Institute)
OptimizationRobotic IntelligenceReinforcement LearningSequential
🎯 What it does: This paper proposes Logic-Skill Programming (LSP), an optimization-based framework for optimal sequential planning of task-agnostic robot manipulation skill libraries without relying on symbolic goal descriptions.
MIRAGE: Cross-Embodiment Zero-Shot Policy Transfer with Cross-Painting
Lawrence Yunliang Chen (UC Berkeley), Ken Goldberg (UC Berkeley)
Domain AdaptationRobotic IntelligenceConvolutional Neural NetworkRecurrent Neural NetworkDiffusion modelImageMesh
🎯 What it does: This paper studies zero-shot cross-body pose policy transfer on unseen robots, proposing the Mirage method which maps source robot policies to target robots through cross-drawing and forward dynamics models, bridging the visual and control domains.
Model Predictive Control for Aggressive Driving Over Uneven Terrain
Tyler Han (University of Washington), Byron Boots (University of Washington)
Autonomous DrivingOptimizationComputational EfficiencyImagePoint Cloud
🎯 What it does: Propose a physics-based discrete dynamic model for steep slopes and gullies with constraint-based dynamics, embedded into a parallel MPPI control framework, achieving safe aggressive driving for full-scale, full-speed off-road vehicles.
MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
Kuan Fang (University of California, Berkeley), Sergey Levine (University of California, Berkeley)
Knowledge DistillationRobotic IntelligencePrompt EngineeringVision Language ModelImage
🎯 What it does: Leverages a pre-trained vision-language model (GPT-4V) to generate a keypoint-based affordance representation by annotating candidate points, grids, and text prompts on images, mapping this representation to robot-executable trajectories, enabling zero-shot or few-shot open-world tasks such as grasping, tool use, and object rearrangement on desktop environments.
Motion Planning in Foliated Manifolds using Repetition Roadmap
Jiaming Hu (Contextual Robotics Institute, University of California, San Diego), Henrik I Christensen (Contextual Robotics Institute, University of California, San Diego)
Robotic Intelligence
🎯 What it does: Proposes the FoliatedRepMap framework, utilizing repeated roadmaps (GMM) and experience-based updates for motion planning on foliated manifolds.
MPCC++: Model Predictive Contouring Control for Time-Optimal Flight with Safety Constraints
Maria Krinner (ETH Zurich), Davide Scaramuzza (University of Zurich)
OptimizationRobotic IntelligenceTabularTime Series
🎯 What it does: This paper proposes the MPCC++ controller, combining track safety constraints, terminal sets, residual dynamics, and TuRBO parameter tuning, achieving high-speed quadrotor racing on tracks.
Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals
Moritz Reuss (Karlsruhe Institute of Technology), Rudolf Lioutikov (Karlsruhe Institute of Technology)
Robotic IntelligenceTransformerVision-Language-Action ModelDiffusion modelContrastive LearningMultimodality
🎯 What it does: Propose Multimodal Diffusion Transformer (MDT), a diffusion strategy framework that can learn multi-task long-horizon operations from multimodal targets (images and language);
Natural Language Can Help Bridge the Sim2Real Gap
Albert Yu (University of Texas at Austin), Roberto Martín-Martín (University of Texas at Austin)
Domain AdaptationRobotic IntelligenceReinforcement LearningVision-Language-Action ModelContrastive LearningImageTextSequential
🎯 What it does: This paper proposes a method that leverages natural language descriptions to bridge visual representations from simulation to real-world scenarios, enabling visual imitation learning in low-data settings.
NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
Jiazhao Zhang (Peking University), He Wang (Peking University)
Robotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningVision Language ModelVision-Language-Action ModelVideoTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: Propose NaVid, a navigation agent based on a large video-language model that accomplishes Vision-and-Language Navigation (VLN) tasks using only RGB video streams, without requiring maps, odometry, or depth inputs.
Octo: An Open-Source Generalist Robot Policy
Dibya Ghosh (University of California Berkeley), Sergey Levine (University of California Berkeley)
Robotic IntelligenceConvolutional Neural NetworkTransformerLarge Language ModelSupervised Fine-TuningVision-Language-Action ModelDiffusion modelMultimodality
🎯 What it does: Proposes Octo, an open-source Transformer foundation model capable of achieving zero-shot control across various robots and supporting efficient fine-tuning.
Octopi: Object Property Reasoning with Large Tactile-Language Models
Samson Yu (National University of Singapore), Harold Soh (National University of Singapore)
RecognitionRobotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringContrastive LearningVideoText
🎯 What it does: Explored combining GelSight tactile vision with large language models to predict object hardness, roughness, and protrusion, and perform physical reasoning, proposing the PHYSICLEAR dataset and the OCTOPI framework.
Offline Imitation Learning Through Graph Search and Retrieval
Zhao-Heng Yin (University of California Berkeley), Pieter Abbeel (University of California Berkeley)
Representation LearningRobotic IntelligenceDiffusion modelImageGraph
🎯 What it does: Developed an offline imitation learning method called GSR, which constructs a graph structure using pre-trained visual representations, calculates action values through graph search, assigns weights to each state via retrieval, and finally learns a better robot control policy using behavioral cloning.
One-Shot Imitation Learning with Invariance Matching for Robotic Manipulation
Xinyu Zhang (Rutgers University), Abdeslam Boularias (Rutgers University)
Robotic IntelligenceMeta LearningGraph Neural NetworkTransformerPoint Cloud
🎯 What it does: Proposed a one-shot imitation learning framework called IMOP that directly infers the robot end-effector pose through task-invariant region matching.
Optimal Non-Redundant Manipulator Surface Coverage with Rank-Deficient Manipulability Constraints
Tong Yang (Zhejiang University), Rong Xiong (Zhejiang University)
Robotic IntelligencePoint CloudMesh
🎯 What it does: Propose a non-redundant coverage path planning (SNCPP) algorithm based on verifiable singular configurations, enabling continuous surface coverage on non-redundant robotic arms.
Parallel and Proximal Linear-Quadratic Methods for Real-Time Constrained Model-Predictive Control
Wilson Jallet (University of Toulouse), Justin Carpentier (Inria)
OptimizationRobotic IntelligenceTime SeriesBenchmark
🎯 What it does: Proposes a parallelized quadratic homogeneous constraint LQR solving algorithm that can rapidly solve large-scale nonlinear MPC problems under dual regularization.
Partially Observable Task and Motion Planning with Uncertainty and Risk Awareness
Aidan Curtis (MIT Computer Science and Artificial Intelligence Laboratory), Leslie Pack Kaelbling (MIT Computer Science and Artificial Intelligence Laboratory)
OptimizationRobotic IntelligenceReinforcement LearningSimultaneous Localization and Mapping
🎯 What it does: Proposes the TAMPURA framework, capable of performing long-term task and motion planning in partially observable, uncertain, and risky environments.
POAM: Probabilistic Online Attentive Mapping for Efficient Robotic Information Gathering
Weizhe Chen (Indiana University), Roni Khardon (Indiana University)
Robotic IntelligenceSimultaneous Localization and MappingImage
🎯 What it does: Proposed a framework called POAM for online sparse Gaussian processes, enabling probabilistic, online, and attention-aware mapping for robot information collection in resource-constrained large-scale environments.
PoCo: Policy Composition from and for Heterogeneous Robot Learning
Lirui Wang (MIT), Russ Tedrake (MIT)
Domain AdaptationRobotic IntelligenceDiffusion modelScore-based ModelImageVideoMultimodalityPoint Cloud
🎯 What it does: Proposes the PoCo (Policy Composition) framework, which achieves efficient transfer and generalization of robotic manipulation policies across different domains (simulation, human videos, real robots) and sensors (RGB, point cloud, tactile) without retraining, by probabilistically multiplying combinations of multi-source, multi-modal, and multi-task diffusion models during inference.
POLICEd RL: Learning Closed-Loop Robot Control Policies with Provable Satisfaction of Hard Constraints
Jean-Baptiste Bouvier (University of California Berkeley), Negar Mehr (University of California Berkeley)
OptimizationSafty and PrivacyRobotic IntelligenceReinforcement Learning
🎯 What it does: Proposes POLICEd RL, a reinforcement learning framework that achieves closed-loop control using deep networks in black-box robotic environments, ensuring that the closed-loop system always satisfies given linear hard constraints.
Practice Makes Perfect: Planning to Learning Skill Parameter Policies
Nishanth Kumar, Jennifer L. Barry (AI Institute)
Robotic IntelligenceReinforcement Learning
🎯 What it does: This paper proposes a framework based on planning to implement skill parameter policies, enabling robots to autonomously select and practice skills in online learning environments without resets, thereby rapidly improving their parameter policies.
Pushing the Limits of Cross-Embodiment Learning for Manipulation and Navigation
Jonathan Heewon Yang (Stanford University), Sergey Levine (Stanford University)
Robotic IntelligenceConvolutional Neural NetworkTransformerDiffusion modelImage
🎯 What it does: Train a cross-embodiment unified goal-conditioned policy capable of simultaneously controlling multiple types of robots, including robotic arms, quadrupedal robots, drones, and mobile bases.
RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes
Kyle Stachowicz (University of California Berkeley), Sergey Levine (University of California Berkeley)
Autonomous DrivingReinforcement Learning
🎯 What it does: Propose a RL framework (RACER) that integrates CVaR risk-sensitive objective, distributed critics, and adaptive action constraints to achieve safe and efficient learning in real-world high-speed off-road driving tasks.