These 27 RSS 2024 papers come with a code repository. Each shows an AI one-line summary below — get the verified repo link + the full 6-part summary (innovation, method, data, results, limitations) and search every RSS 2024 paper, free trial on arXivSub.
3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
Yanjie Ze (Shanghai Qizhi Institute), Huazhe Xu (Tsinghua University)
CodeRobotic IntelligenceLarge Language ModelDiffusion modelPoint Cloud
🎯 What it does: Propose 3D Diffusion Policy (DP3), combining sparse point cloud encoding with diffusion policy to achieve low-sample visual imitation learning;
🎯 What it does: A data structure named Collision-Tolerant Point Tree (CAPT) was constructed for fast and accurate collision detection on perceived point clouds, with the implementation of SIMD parallel queries;
Demonstrating Arena 3.0: Advancing Social Navigation in Collaborative and Highly Dynamic Environments
Linh Kästner (Technical University Berlin), Jens Lambrecht (Technical University Berlin)
CodeRobotic IntelligenceWorld ModelBenchmark
🎯 What it does: Develop the Arena 3.0 platform, integrating realistic crowd simulation, social force models, dynamic task generation, robot navigation kits, and achieving cross-platform abstraction and unified API on three simulators: Flatland, Gazebo, and Unity;
Demonstrating Event-Triggered Investigation and Sample Collection for Human Scientists using Field Robots and Large Foundation Models
Tirthankar Bandyopadhyay (CSIRO Robotics, Data61), Stanislav Funiak (CSIRO Robotics, Data61)
CodeRobotic IntelligenceTransformerLarge Language ModelVision-Language-Action ModelSimultaneous Localization and MappingImageTextPoint Cloud
🎯 What it does: An end-to-end event-triggered science exploration and sampling system based on a robot team, vibration sensor network, and large foundation model (LFM) was constructed and demonstrated in a simulated lunar surface sandbox, showcasing the robot's capability to perform exploration, localization, sampling, semantic mapping, and whole-body manipulation under natural language interaction.
Demonstrating OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics
Peiqi Liu (New York University), Lerrel Pinto (New York University)
CodeRobotic IntelligenceTransformerVision Language ModelSimultaneous Localization and MappingImageTextPoint Cloud
🎯 What it does: Built an open-source knowledge robot system called OK-Robot, which can complete the full pick-and-drop task of opening-recognizing-navigating-grasping-placing in new home environments through zero training.
🎯 What it does: Designed a BRIDGER interpolation diffusion method that leverages information-rich source strategies for imitation learning, improving upon the limitations of traditional Gaussian noise diffusion.
🎯 What it does: Designed and implemented dynamically displayed adversarial patches on mobile vehicle screens to deceive the target detection models of autonomous driving systems into misclassifying traffic signs, thereby influencing their decision-making;
From Compliant to Rigid Contact Simulation: a Unified and Efficient Approach
Justin Carpentier (Inria), Louis Montaut (Inria)
CodeOptimizationRobotic IntelligenceBenchmarkPhysics Related
🎯 What it does: Designed and implemented a unified contact solver based on ADMM and proximal optimization, capable of simultaneously handling rigid and elastic contact without requiring physical relaxation.
🎯 What it does: Propose a safety-guaranteed trajectory design method called PARC based on a piecewise affine model, which can achieve collision-free target arrival in scenarios close to danger.
🎯 What it does: This paper proposes the GRACE framework, which balances multiple grasping criteria through hierarchical rules and an expected utility function, achieving executable, stable, collision-free, and functional grasps.
🎯 What it does: This paper proposes a homotopy path set planning method for robot manipulation and navigation, covering sparse channel detection, channel-aware optimal path planning, and path set generation based on path transfer.
iMESA: Incremental Distributed Optimization for Collaborative Simultaneous Localization and Mapping
Daniel McGann (Carnegie Mellon University), Michael Kaess (Carnegie Mellon University)
CodeOptimizationRobotic IntelligenceSimultaneous Localization and Mapping
🎯 What it does: Proposed and implemented iMESA, an incremental distributed backend for collaborative SLAM, capable of achieving real-time and accurate global state estimation under sparse communication conditions.
Khronos: A Unified Approach for Spatio-Temporal Metric-Semantic SLAM in Dynamic Environments
Lukas Schmid (Massachusetts Institute of Technology), Luca Carlone (Massachusetts Institute of Technology)
CodeOptimizationRobotic IntelligenceSimultaneous Localization and MappingPoint CloudTime Series
🎯 What it does: This paper proposes the Khronos system, achieving unified spatiotemporal metric semantic SLAM capable of real-time construction of dense 4D maps containing short-term dynamics and long-term changes.
Language-Augmented Symbolic Planner for Open-World Task Planning
Guanqi Chen (The University of Hong Kong), Jia Pan (The University of Hong Kong)
CodeTransformerLarge Language ModelText
🎯 What it does: Leverage large language models to assist symbolic planners in accomplishing complex long-term tasks in open-world environments, automatically diagnosing and repairing execution errors;
🎯 What it does: Designed a pre-training framework called MPI for interaction, which utilizes keyframes and language instructions to predict unseen interaction frames and detect interactive objects, thereby enhancing robotic manipulation representations.
Learning to Learn Faster from Human Feedback with Language Model Predictive Control
Jacky Liang, Carolina Parada
CodeRobotic IntelligenceMeta LearningReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Studied enhancing the teachability of robot code-writing LLMs by fine-tuning large language models through language model predictive control (LMPC) to improve rapid learning from human feedback.
Linear-time Differential Inverse Kinematics: an Augmented Lagrangian Perspective
Bruce Wingo (Georgia Institute of Technology), Justin Carpentier (Inria)
CodeOptimizationRobotic Intelligence
🎯 What it does: Proposed a differential inverse kinematics solver called LOIK with linear complexity that can handle linear equality and inequality constraints;
MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
Kuan Fang (University of California, Berkeley), Sergey Levine (University of California, Berkeley)
CodeKnowledge DistillationRobotic IntelligencePrompt EngineeringVision Language ModelImage
🎯 What it does: Leverages a pre-trained vision-language model (GPT-4V) to generate a keypoint-based affordance representation by annotating candidate points, grids, and text prompts on images, mapping this representation to robot-executable trajectories, enabling zero-shot or few-shot open-world tasks such as grasping, tool use, and object rearrangement on desktop environments.
Octopi: Object Property Reasoning with Large Tactile-Language Models
Samson Yu (National University of Singapore), Harold Soh (National University of Singapore)
CodeRecognitionRobotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringContrastive LearningVideoText
🎯 What it does: Explored combining GelSight tactile vision with large language models to predict object hardness, roughness, and protrusion, and perform physical reasoning, proposing the PHYSICLEAR dataset and the OCTOPI framework.
🎯 What it does: Proposes a parallelized quadratic homogeneous constraint LQR solving algorithm that can rapidly solve large-scale nonlinear MPC problems under dual regularization.
CodeRobotic IntelligenceSimultaneous Localization and MappingImage
🎯 What it does: Proposed a framework called POAM for online sparse Gaussian processes, enabling probabilistic, online, and attention-aware mapping for robot information collection in resource-constrained large-scale environments.
RVT-2: Learning Precise Manipulation from Few Demonstrations
Ankit Goyal (NVIDIA), Dieter Fox (NVIDIA)
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackTransformerVision-Language-Action ModelImagePoint Cloud
🎯 What it does: Studied a robot system called RVT-2 that can perform multi-task high-precision 3D manipulation using language instructions with only a small number of demonstrations.
CodeRobotic IntelligenceTransformerVision Language ModelVideoTextMultimodality
🎯 What it does: A new robotic manipulation framework called VLMPC is proposed, which combines vision-language models (VLM) and model predictive control (MPC) to enhance robotic operation capabilities in complex scenarios.
Who Plays First? Optimizing the Order of Play in Stackelberg Games with Many Robots
Haimin Hu (Princeton University), Jaime Fernández Fisac (Princeton University)
CodeOptimizationRobotic Intelligence
🎯 What it does: This paper studies how to automatically find the community-optimal game order (i.e., the sequence of decision-making among robots) in multi-robot Stackelberg trajectory games and obtain the corresponding local Stackelberg equilibrium.