NeurIPS 2024 Papers — Page 41
Conference on Neural Information Processing Systems · 4035 papers
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
Liwei Jiang (University of Washington), Nouha Dziri (Allen Institute for Artificial Intelligence)
Safty and PrivacyAdversarial AttackTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: The WILDTEAMING framework is proposed to automatically mine jailbreak techniques from real user interactions with chatbots and combine them to generate diverse adversarial attacks, thereby constructing a large-scale secure training dataset called WILDJAILBREAK.
Wings: Learning Multimodal LLMs without Text-only Forgetting
Yi-Kai Zhang (Nanjing University), Han-Jia Ye (Nanjing University)
TransformerLarge Language ModelVision Language ModelTextMultimodality
🎯 What it does: This paper proposes a multimodal large language model structure named WINGS, which utilizes low-rank residual attention learners (LoRRA) on both visual and textual sides, along with an attention-shift-based soft routing mechanism, to address the issue of forgetting tasks that are text-only during multimodal training.
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Peng Wang (Zhejiang University), Huajun Chen (Zhejiang University)
OptimizationKnowledge DistillationTransformerLarge Language ModelText
🎯 What it does: The WISE method is proposed to achieve lifelong model editing for LLMs, allowing the model to remain reliable, generalizable, and not interfere with existing knowledge after multiple updates.
WizardArena: Post-training Large Language Models via Simulated Offline Chatbot Arena
Haipeng Luo (Tsinghua University), Weizhu Chen (Microsoft Corporation)
TransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: An offline simulated chat arena called WizardArena is built based on large language models to replace human evaluation and is used for model fine-tuning.
WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment
Hao Tang (Cornell University), Kevin Ellis (Cornell University)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningAgentic AIWorld ModelText
🎯 What it does: This paper proposes a model-driven intelligent agent called WorldCoder based on LLM, which utilizes LLM to construct a world model of the environment by writing Python programs and executes actions through planning.
Wormhole Loss for Partial Shape Matching
Amit Bracha (Technion Israel Institute of Technology), Ron Kimmel (Technion Israel Institute of Technology)
Diffusion modelPoint CloudMesh
🎯 What it does: The research focuses on partial shape matching and proposes a wormhole loss function based on consistent points.
Worst-Case Offline Reinforcement Learning with Arbitrary Data Support
Kohei Miyaguchi (IBM Research)
Reinforcement Learning
🎯 What it does: A worst-case offline reinforcement learning framework is proposed that does not require any data support (concentrability) assumptions, along with the corresponding algorithm WMRL;
Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
Avelina Asada Hadji-Kyriacou (University of St Andrews), Ognjen Arandjelovic (University of St Andrews)
TransformerLarge Language ModelReinforcement LearningText
🎯 What it does: This paper proposes Direct Preference Heads (DPH), which align the language model by scoring candidate outputs through an additional reward head during inference, avoiding direct modification of the generation distribution.
X-Ray: A Sequential 3D Representation For Generation
Tao Hu, Gim Hee Lee
GenerationData SynthesisDiffusion modelVideoMesh
🎯 What it does: A method for X-Ray serialized 3D representation is proposed, and the generation of 3D meshes from single-view images is achieved through a video diffusion model.
xLSTM: Extended Long Short-Term Memory
Maximilian Beck (Johannes Kepler University Linz), Sepp Hochreiter (Johannes Kepler University Linz)
Recurrent Neural NetworkText
🎯 What it does: An extended LSTM model called xLSTM is proposed, which combines exponential gating and matrix memory to address the storage, correction, and parallelism limitations of traditional LSTMs.
XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation
Ziyi Wang (Tsinghua University), Jiwen Lu (Tsinghua University)
SegmentationVision Language ModelDiffusion modelPoint Cloud
🎯 What it does: Proposes the XMask3D framework, which utilizes cross-modal mask inference to achieve open vocabulary 3D semantic segmentation.
xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Julius Hense (Berlin Institute for the Foundations of Learning and Data), Klaus Robert Muller
Explainability and InterpretabilityBiomedical Data
🎯 What it does: This paper proposes the xMIL framework and implements xMIL-LRP to generate interpretable heatmaps for the predictions of multi-instance learning (MIL) models in digital pathology.
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Xin Cheng (Peking University), Dongyan Zhao (Peking University)
RetrievalCompressionComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: This paper proposes an extreme context compression method called xRAG, which utilizes retrieval embeddings as multimodal features to compress an entire document into a single token input for a language model.
Yo'LLaVA: Your Personalized Language and Vision Assistant
Thao Nguyen (University of Wisconsin Madison), Yong Jae Lee (University of Wisconsin Madison)
RecognitionRetrievalTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: A method is proposed for personalizing large multimodal models using a small number of image pairs, enabling the model to recognize and engage in conversations with users about specific subjects.
YOLOv10: Real-Time End-to-End Object Detection
Ao Wang (Tsinghua University), Guiguang Ding (Tsinghua University)
Object DetectionConvolutional Neural NetworkImage
🎯 What it does: A real-time end-to-end YOLOv10 object detection framework has been developed, eliminating NMS and comprehensively optimizing the model structure.
You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning
Théo Moutakanni (Meta), Piotr Bojanowski (Meta)
SegmentationDepth EstimationRepresentation LearningTransformerContrastive LearningImage
🎯 What it does: The study investigates whether manual data augmentation is necessary in large-scale self-supervised joint embedding architecture (JEA), proposing and validating that using only cropping (without random resizing) and occlusion can achieve performance comparable to or even exceeding that of traditional strong data augmentation.
You Only Cache Once: Decoder-Decoder Architectures for Language Models
Yutao Sun (Tsinghua University), Furu Wei (Microsoft Research)
TransformerLarge Language ModelText
🎯 What it does: Proposes the YOCO decoder-decoder architecture, which caches KV only once, significantly reducing GPU memory usage and pre-fill latency for long sequence inference.
You Only Look Around: Learning Illumination-Invariant Feature for Low-light Object Detection
MingboHong, Shuaicheng Liu (University of Electronic Science and Technology of China)
Object DetectionConvolutional Neural NetworkImage
🎯 What it does: The YOLA framework and Illumination-Invariant Module (IIM) are proposed to enhance object detection performance under low-light conditions by learning illumination-invariant features based on the Lambertian model.
YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals
Sandeep Mishra (University of Texas at Austin), Alan Bovik
GenerationData SynthesisPose EstimationLarge Language ModelDiffusion modelScore-based ModelNeural Radiance FieldImageText
🎯 What it does: By combining text prompts with 3D skeletons (poses), TetraPose ControlNet based on Stable Diffusion achieves controllable high-quality 3D generation of animal forms, supporting both known and fictional animals.
Your contrastive learning problem is secretly a distribution alignment problem
Zihao Chen (Georgia Institute of Technology), Eva L Dyer
Domain AdaptationRepresentation LearningContrastive LearningImage
🎯 What it does: This paper proposes Generalized Contrastive Alignment (GCA), which reinterprets contrastive learning as a distribution alignment problem and constructs a customizable alignment loss using optimal transport, allowing for fine-grained control of positive and negative sample relationships in self-supervised learning.
Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training
Yunshu Wu (University of California Riverside), Greg Ver Steeg (University of California Riverside)
RestorationGenerationData SynthesisDiffusion modelContrastive LearningImage
🎯 What it does: A self-supervised Contrastive Diffusion Loss (CDL) is proposed, which enhances the model's denoising performance in out-of-distribution (OOD) regions by treating the diffusion model as a noise classifier, thereby improving and accelerating the sequential and parallel sampling processes.
Zero-Shot Event-Intensity Asymmetric Stereo via Visual Prompting from Image Domain
Hanyue Lou (Peking University), Boxin Shi (Peking University)
Depth EstimationAutonomous DrivingPrompt EngineeringImage
🎯 What it does: A zero-training event-intensity asymmetric stereo matching framework called ZEST is proposed, which aligns visual cues from events and frames, allowing image domain pre-trained stereo matching and monocular depth estimation models to be directly applied to event-frame scenes.
Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
Jieren Deng (Institute of Automation Chinese Academy of Sciences), Yunkuan Wang (Institute of Automation Chinese Academy of Sciences)
Object DetectionTransformerVision Language ModelImageMultimodality
🎯 What it does: This paper proposes the Incremental Visual Language Object Detection (IVLOD) task and designs an incremental learning method that maintains zero-shot generalization.
Zero-shot Image Editing with Reference Imitation
Xi Chen, Hengshuang Zhao
Image TranslationImage HarmonizationConvolutional Neural NetworkDiffusion modelImageVideo
🎯 What it does: This paper proposes a zero-shot image editing framework called MimicBrush, where users only need to mark the area to be edited in the source image and provide a reference image. The model automatically locates the corresponding part in the reference image and completes the editing in the source image.
Zero-Shot Reinforcement Learning from Low Quality Data
Scott Jeen (University of Cambridge), Jonathan Cullen
Reinforcement LearningTabular
🎯 What it does: This paper studies methods for zero-shot reinforcement learning on low-quality data and proposes conservative regularization to alleviate outrageous state-action estimation.
Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly
Junsheng Zhou (Tsinghua University), Zhizhong Han (Wayne State University)
Object DetectionSegmentationGenerationPose EstimationDepth EstimationOptimizationDiffusion modelImagePoint Cloud
🎯 What it does: A deep prior assembly framework is proposed to achieve zero-shot scene reconstruction from a single image.
Zero-Shot Tokenizer Transfer
Benjamin Minixhofer (University of Cambridge), Ivan Vulić (University of Edinburgh)
TransformerLarge Language ModelSupervised Fine-TuningTextMultimodalityBenchmark
🎯 What it does: A method is proposed to transfer pre-trained language models to any new tokenizer using zero-shot tokenizer transfer (ZeTT) without training a new model.
Zero-Shot Transfer of Neural ODEs
Tyler Ingebrand (University of Texas at Austin), ufuk topcu
OptimizationRobotic IntelligenceTime SeriesOrdinary Differential Equation
🎯 What it does: This study investigates a zero-shot transfer method for learning neural ODE basis functions through functional encoders, enabling adaptive control systems to quickly identify new dynamics without retraining.
Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering
Ido Sobol (Technion), Or Litany (NVIDIA)
GenerationData SynthesisDiffusion modelImage
🎯 What it does: A zero-shot new perspective synthesis enhancement method called Zero-to-Hero is proposed, which improves the generation quality and consistency of the existing Zero-1-to-3 model.
ZeroMark: Towards Dataset Ownership Verification without Disclosing Watermark
Junfeng Guo (University of Maryland), Heng Huang (University of Maryland)
Adversarial AttackConvolutional Neural NetworkImage
🎯 What it does: A method called ZeroMark is designed to verify dataset ownership in a black-box scenario without exposing the watermark. It utilizes adversarial attacks to generate samples near the decision boundary and completes the verification through gradient cosine similarity and hypothesis testing.
Zeroth-Order Sampling Methods for Non-Log-Concave Distributions: Alleviating Metastability by Denoising Diffusion
Ye He (Georgia Institute of Technology), Molei Tao (Georgia Institute of Technology)
OptimizationDiffusion modelMultimodality
🎯 What it does: This paper proposes a sampling framework based on zero-order queries, utilizing Denoising Diffusion Monte Carlo (DDMC) combined with rejection sampling to achieve a gradient-free sampling algorithm ZOD-MC, which can efficiently sample from non-log-convex distributions.
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
Yefei He (Zhejiang University), Bohan Zhuang (Monash University)
GenerationRetrievalCompressionComputational EfficiencyTransformerLarge Language ModelText
🎯 What it does: This paper proposes ZipCache, which performs accurate and efficient mixed-precision quantization compression for KV cache in large language models, significantly reducing memory usage and inference latency.
Zipfian Whitening
Sho Yokoi (Tohoku University / RIKEN), Hidetoshi Shimodaira (Kyoto University / RIKEN)
Text
🎯 What it does: This paper proposes a whitening method (Zipfian whitening) that uses actual word frequency (Zipfian frequency) for expectation calculation in the word vector space, and provides corresponding symmetry evaluation metrics. By applying this method to post-process pre-trained word vectors such as GloVe, Word2Vec, and fastText, it significantly improves the performance of downstream tasks like sentence similarity.
Zipper: Addressing Degeneracy in Algorithm-Agnostic Inference
Geng Chen (Nankai University), Changliang Zou (Nankai University)
Tabular
🎯 What it does: This paper proposes a new testing framework for the assessment of prediction quality that is independent of algorithms/models.
ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving
Tao MA, Hongsheng Li (Chinese University of Hong Kong)
Object DetectionSegmentationAutonomous DrivingMultimodalityPoint Cloud
🎯 What it does: A zero-shot offline panoramic perception framework ZOPP is proposed, utilizing multimodal (multi-view cameras + LiDAR) to automatically generate 3D semantic, instance, detection, and occupancy labels;