NeurIPS 2024 Papers — Page 40
Conference on Neural Information Processing Systems · 4035 papers
Verified Safe Reinforcement Learning for Neural Network Dynamic Models
Junlin Wu (Washington University in St. Louis), Yevgeniy Vorobeychik (Washington University in St. Louis)
Autonomous DrivingSafty and PrivacyReinforcement LearningTime Series
🎯 What it does: A safety reinforcement learning framework based on finite-step reachability verification is proposed, capable of training control policies that can operate within specified safety boundaries on neural network dynamic models.
VeXKD: The Versatile Integration of Cross-Modal Fusion and Knowledge Distillation for 3D Perception
JI Yuzhe, Xinhu Zheng (Hong Kong University of Science and Technology)
Object DetectionSegmentationAutonomous DrivingKnowledge DistillationTransformerMultimodalityPoint Cloud
🎯 What it does: Proposes the VeXKD framework, which combines cross-modal fusion and knowledge distillation to enhance the accuracy and real-time performance of single-modal 3D perception;
VFIMamba: Video Frame Interpolation with State Space Models
Guozhen Zhang (Nanjing University), Limin Wang (Nanjing University)
RestorationGenerationConvolutional Neural NetworkOptical FlowVideo
🎯 What it does: An efficient video frame interpolation method VFIMamba utilizing the S6 state space model is proposed.
Video Diffusion Models are Training-free Motion Interpreter and Controller
Zeqi Xiao (Nanyang Technological University), Xingang Pan (Nanyang Technological University)
GenerationData SynthesisDiffusion modelVideo
🎯 What it does: A training-free video motion control framework is proposed, which achieves control over motion direction and point dragging by extracting motion features (MOFT) from video diffusion models.
Video Token Merging for Long Video Understanding
Seon-Ho Lee (Korea University), Xinyu Li (Amazon)
Computational EfficiencyTransformerVideo
🎯 What it does: For the task of long video understanding, a Learnable Video Token Merging (VTM) method based on Transformer is proposed to reduce token redundancy and improve model efficiency.
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
Shiwei Wu (University of Science and Technology of China), Mike Zheng Shou (National University of Singapore)
Computational EfficiencyRepresentation LearningTransformerMixture of ExpertsVision Language ModelVideoMultimodality
🎯 What it does: This paper proposes the implementation of visual computation pruning in online video large language models through Mixture-of-Depth's LayerExpert, maintaining high resolution while significantly reducing costs.
VideoTetris: Towards Compositional Text-to-Video Generation
Ye Tian (Peking University), Bin CUI
GenerationData SynthesisLarge Language ModelDiffusion modelOptical FlowVideoText
🎯 What it does: The VideoTetris framework is proposed to achieve text-based composable video generation, supporting spatial positioning and temporal changes of multiple objects.
VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation
Youpeng Wen (Shenzhen Campus of Sun Yat-Sen University), Xiaodan Liang (Peng Cheng Laboratory)
Robotic IntelligenceTransformerDiffusion modelVideo
🎯 What it does: The VidMan framework is proposed, which utilizes a video diffusion model for robot action prediction through a two-stage training process.
Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels
Yikai Wang (Tsinghua University), Jun Zhu (Tsinghua University)
GenerationData SynthesisDiffusion modelGaussian SplattingVideo
🎯 What it does: Vidu4D achieves the automated generation of high-quality dynamic 3D content from video by recovering high-fidelity 4D (time-series 3D) representations from a single generated video.
Virtual Scanning: Unsupervised Non-line-of-sight Imaging from Irregularly Undersampled Transients
Xingyu Cui (Tianjin University), Jingyu Yang (Tianjin University)
RestorationImageVideo
🎯 What it does: An unsupervised framework is proposed that utilizes virtual scanning and a SURE-based denoiser to reconstruct non-line-of-sight scenes from irregularly under-sampled time-delay signals.
VISA: Variational Inference with Sequential Sample-Average Approximations
Heiko Zimmermann (University of Amsterdam), Jan-Willem van de Meent (University of Amsterdam)
OptimizationTabular
🎯 What it does: A variational inference method named VISA is proposed, which utilizes Sequential Sample Average Approximation (SAA) for forward KL inference in non-differentiable and computationally expensive models, significantly reducing the number of model evaluations.
Vision Foundation Model Enables Generalizable Object Pose Estimation
Kai Chen (Chinese University of Hong Kong), Qi Dou (UC Berkeley)
Pose EstimationTransformerContrastive LearningImagePoint Cloud
🎯 What it does: A two-stage framework VFM-6D based on visual foundation models is proposed for generalized pose estimation of arbitrary object categories.
Vision Mamba Mender
Jiacong Hu (Zhejiang University), Mingli Song (Zhejiang University)
ClassificationRecognitionImage
🎯 What it does: This paper proposes Vision Mamba Mender, which utilizes post-state correlation analysis to identify and repair external and internal state defects in Mamba models for visual tasks, thereby enhancing model performance.
Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Chenyu Yang (Tsinghua University), Jifeng Dai (Tsinghua University)
GenerationRetrievalRepresentation LearningTransformerContrastive LearningImageTextMultimodality
🎯 What it does: A visual model pre-training method utilizing Layer Compression Learning (LCL) is proposed, which can train the visual encoder from scratch on mixed image-text sequences.
Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights
Sy-Tuyen Ho (Singapore University of Technology and Design), Ngai-man Cheung
Domain AdaptationNeural Architecture SearchTransformerImageBenchmark
🎯 What it does: The first NAS benchmark for Vision Transformer (ViT) in Out-of-Distribution (OoD) generalization (OoD-ViT-NAS) is proposed, and a systematic evaluation and analysis of over 3,000 ViT architectures on 8 major OoD datasets is conducted using this benchmark.
Vision-Language Models are Strong Noisy Label Detectors
Tong Wei (Southeast University), Min-Ling Zhang (Southeast University)
ClassificationAnomaly DetectionTransformerPrompt EngineeringVision Language ModelImageMultimodality
🎯 What it does: The DEFT framework is proposed, which detects noisy labels by learning positive and negative dual text prompts on a pre-trained vision-language model, and uses the filtered clean samples to complete the final model fine-tuning.
Vision-Language Navigation with Energy-Based Policy
Rui Liu (Zhejiang University), Yi Yang (Zhejiang University)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerReinforcement LearningVision-Language-Action ModelMultimodality
🎯 What it does: This paper proposes an energy-based navigation strategy (ENP) that guides the decision-making of visual-language navigation (VLN) agents by learning the state-action joint distribution from expert demonstrations;
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
Jiannan Wu (OpenGVLab), Jifeng Dai (Tsinghua University)
RecognitionSegmentationGenerationPose EstimationTransformerLarge Language ModelVision Language ModelDiffusion modelImageTextMultimodality
🎯 What it does: Proposes VisionLLM v2, constructing an end-to-end general multimodal large model that unifies visual perception, understanding, and generation, supporting hundreds of visual-visual language tasks;
VisMin: Visual Minimal-Change Understanding
Rabiul Awal (Mila Quebec AI Institute Université de Montréal), Aishwarya Agrawal (Mila Quebec AI Institute Université de Montréal)
ClassificationRecognitionRetrievalTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelDiffusion modelImageTextMultimodalityBenchmark
🎯 What it does: The VisMin benchmark is proposed to evaluate the fine-grained understanding ability of visual-language models in matching images and texts with minimal changes.
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Shenyuan Gao (Hong Kong University of Science and Technology), Hongyang Li (University of Hong Kong)
GenerationAutonomous DrivingDiffusion modelWorld ModelVideo
🎯 What it does: A general world model called Vista has been constructed, capable of generating realistic future driving scenes with high spatiotemporal resolution and supporting multimodal action control;
Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model
Haogeng Liu (Institute of Automation, Chinese Academy of Sciences), Hongxia Yang (Institute of Automation, Chinese Academy of Sciences)
Computational EfficiencyRepresentation LearningTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: The concept of visual anchors is proposed, and based on this, the Anchor Former (AcFormer) is designed as a visual-language connector for multimodal large language models, capable of significantly reducing the number of visual tokens and computational costs while maintaining or even improving accuracy.
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
Keyu Tian (Peking University), Liwei Wang (Peking University)
GenerationTransformerAuto EncoderImage
🎯 What it does: A new visual autoregressive framework VAR is proposed, which achieves efficient image generation through multi-scale 'next-scale' prediction.
Visual Data Diagnosis and Debiasing with Concept Graphs
Rwiddhi Chakraborty (UiT Arctic University of Norway), Fernando De la Torre (Carnegie Mellon University)
ClassificationData-Centric LearningSupervised Fine-TuningDiffusion modelImage
🎯 What it does: Construct a conceptual knowledge graph of visual data, diagnose the graph to discover concept co-occurrence biases, and achieve data debiasing by generating images of missing concept combinations, ultimately enhancing the generalization ability of downstream classifiers.
Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion
Dongyang Li (Southern University of Science and Technology), Quanying Liu (Southern University of Science and Technology)
ClassificationRestorationGenerationRetrievalTransformerDiffusion modelContrastive LearningMultimodalityTime SeriesMagnetic Resonance Imaging
🎯 What it does: A zero-shot visual decoding and reconstruction framework based on EEG has been developed, which includes an ATM EEG encoder and a two-stage EEG-guided diffusion generation method.
Visual Fourier Prompt Tuning
Runjia Zeng (Rochester Institute of Technology), Dongfang Liu (Rochester Institute of Technology)
ClassificationDomain AdaptationExplainability and InterpretabilityTransformerPrompt EngineeringImage
🎯 What it does: This paper proposes a Visual Fourier Prompt Tuning (VFPT) method that embeds 2D Fast Fourier Transform (FFT) into visual prompts, allowing for efficient transfer across different datasets with only a small number of parameters fine-tuned.
Visual Perception by Large Language Model’s Weights
Feipeng Ma (University of Science and Technology of China), Xiaoyan Sun (University of Science and Technology of China)
RecognitionComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodality
🎯 What it does: VLoRA is proposed, which achieves multimodal perception by converting visual features into low-rank model weights (perceptual weights) and integrating them with LLM weights without introducing visual tokens.
Visual Pinwheel Centers Act as Geometric Saliency Detectors
Haixin Zhong (Fudan University), yuguo yu
Spiking Neural NetworkImage
🎯 What it does: This paper proposes a two-dimensional self-adaptive synaptic plasticity brain-machine network (SESNN), which simulates the topological evolution from salt-and-pepper organization to tear-drop structure in the V1 layer through training on natural images, and demonstrates that the pinwheel center is a precursor processing unit for geometric saliency detection.
Visual Prompt Tuning in Null Space for Continual Learning
Yue Lu (Northwestern Polytechnical University), Yanning Zhang (Xidian University)
ClassificationRecognitionOptimizationTransformerPrompt EngineeringImage
🎯 What it does: To address the catastrophic forgetting problem in visual prompt tuning within ViT, an orthogonal projection of the prompt gradient in the null space is proposed to achieve interference-free updates for learned tasks.
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
Yushi Hu (University of Washington), Ranjay Krishna
Object DetectionSegmentationGenerationOptimizationTransformerLarge Language ModelVision Language ModelMultimodalityBenchmarkChain-of-Thought
🎯 What it does: Proposes the Visual SKETCHPAD framework, allowing multimodal language models to generate and utilize intermediate visual sketches during the reasoning process.
Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing
Hao Fei (National University of Singapore), Shuicheng YAN
Object DetectionSegmentationGenerationTransformerLarge Language ModelVision Language ModelImageVideoText
🎯 What it does: VITRON is proposed, a unified pixel-level visual large language model capable of simultaneously handling understanding, generation, segmentation, and editing tasks for both images and videos.
Vivid-ZOO: Multi-View Video Generation with Diffusion Model
Bing Li (King Abdullah University of Science and Technology), Bernard Ghanem (King Abdullah University of Science and Technology)
GenerationData SynthesisDiffusion modelVideoText
🎯 What it does: A diffusion model named Vivid-ZOO has been developed, capable of generating high-quality multi-view videos based on text descriptions.
VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance
Divyansh Srivastava (University of California San Diego), Tsui-Wei Weng (University of California San Diego)
ClassificationObject DetectionConvolutional Neural NetworkLarge Language ModelImage
🎯 What it does: A Vision-Language Guided Concept Bottleneck Model (VLG-CBM) is constructed, which generates candidate concepts through LLM and uses open-domain location-based object detectors (such as Grounding-DINO) to produce visually recognizable and locatable concept annotations, thereby achieving end-to-end visual and language guidance during the training of CBM.
VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought
Gabriel Herbert Sarch (Carnegie Mellon University), Katerina Fragkiadaki (Carnegie Mellon University)
Knowledge DistillationRobotic IntelligenceTransformerVision Language ModelVideoMultimodalityRetrieval-Augmented Generation
🎯 What it does: This paper proposes a framework named ICAL, which utilizes visual-language models to abstract and correct noisy demonstrations, automatically generating reusable multimodal memories to enhance task reasoning and decision-making.
VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions
Guangyan Chen (Beijing Institute of Technology), Yufeng Yue (Beijing Institute of Technology)
Robotic IntelligenceReinforcement Learning from Human FeedbackTransformerReinforcement LearningVision Language ModelVideoMultimodality
🎯 What it does: Proposes the VLMimic framework, which utilizes visual language models to learn fine-grained action skills directly from a small number of human videos and adaptively updates in new environments through an iterative comparison strategy.
VMamba: Visual State Space Model
Yue Liu (University of Chinese Academy of Sciences), Yunfan Liu (University of Chinese Academy of Sciences)
ClassificationObject DetectionSegmentationConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: A visual state space model named VMamba is proposed, achieving linear time complexity for visual backbone networks.
Voila-A: Aligning Vision-Language Models with User's Gaze Attention
Kun Yan (Beihang University), Shuai Ma (Beihang University)
RetrievalExplainability and InterpretabilityTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: Voila-A is proposed, which aligns visual language models using user gaze information to enhance their interpretability and effectiveness in real-world scenarios.
Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection
Guowen Zhang (Hong Kong Polytechnic University), Lei Zhang (Hong Kong Polytechnic University)
Object DetectionAutonomous DrivingPoint Cloud
🎯 What it does: A group-free voxel 3D detection backbone network based on State Space Model (SSM) is proposed—Voxel Mamba.
Voxel Proposal Network via Multi-Frame Knowledge Distillation for Semantic Scene Completion
Lubo Wang (Tianjin University), Ping Li (Hong Kong Polytechnic University)
SegmentationAutonomous DrivingKnowledge DistillationConvolutional Neural NetworkPoint Cloud
🎯 What it does: This paper proposes the Voxel Proposal Network (VPNet), which achieves semantic scene completion through BEV and a 3D dual-branch, confidence voxel proposals, and multi-frame knowledge distillation.
VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization
Yiwei Zhang (University of Chinese Academy of Sciences), Weiming Hu (Stony Brook University)
SegmentationGenerationAutonomous DrivingTransformerAuto EncoderPoint Cloud
🎯 What it does: Utilizing a VQ-VAE-like generative model to convert BEV maps into discrete tokens, constructing a codebook as a prior; aligning camera perspective (PV) features to the discrete BEV space through a dedicated token decoder, and subsequently generating high-quality BEV semantic maps using the codebook, thereby addressing challenges such as occlusion and low resolution.
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models
Jinghan Jia (Michigan State University), Sijia Liu (IBM Research)
OptimizationSafty and PrivacyTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper proposes a weight attribution-based gradual forgetting framework for large language models (WAGLE), which enhances the forgetting effect of large language models while maintaining their original functionality by identifying the model weights that have the most significant impact on the forgetting task.
Warm-starting Push-Relabel
Sami Davies (University of California Berkeley), Yuyan Wang (Google Research)
SegmentationOptimizationImage
🎯 What it does: A Push-Relabel algorithm utilizing predicted flow for warm-start in the maximum flow problem is proposed, with a theoretical runtime upper bound of O(η n²);
Warm-up Free Policy Optimization: Improved Regret in Linear Markov Decision Processes
Asaf Cassel (Tel Aviv University), Aviv Rosenberg (Google Research)
OptimizationReinforcement Learning
🎯 What it does: A linear MDP policy optimization algorithm (CFPO) is proposed that does not require a pure exploration warm-up phase. It maintains bounded estimates of Q-values through a feature compression mechanism, achieving more efficient learning.
Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Giannis Daras (University of Texas at Austin), Arash Vahdat (NVIDIA)
RestorationSuper ResolutionDiffusion modelOptical FlowVideo
🎯 What it does: Proposes the Warped Diffusion framework, which utilizes image diffusion models to address video inverse problems through noise distortion and equivariant self-guidance, ensuring temporal consistency.
Wasserstein convergence of Cech persistence diagrams for samplings of submanifolds
Charles Arnal (Universite Paris Saclay), Vincent Divol (CREST ENSAE)
Point Cloud
🎯 What it does: Under the assumption of sampling from submanifolds, the Wasserstein convergence of Cech persistent diagrams (PD) is studied, providing a quadratic improvement on the bottleneck distance, a consistency law and limit law for α-total persistence, as well as necessary and sufficient conditions for p-Wasserstein convergence requiring p > m.
Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
Jiaming Lv (Dalian University of Technology), Peihua Li (Dalian University of Technology)
Object DetectionKnowledge DistillationGaussian SplattingImage
🎯 What it does: This paper proposes a knowledge distillation method based on Wasserstein distance, implementing distillation at the logits layer (WKD-L) and the feature layer (WKD-F);
Wasserstein Distributionally Robust Optimization through the Lens of Structural Causal Models and Individual Fairness
Ahmad Reza Ehyaei, Samira Samadi (Max Planck Institute for Intelligent Systems)
OptimizationTabular
🎯 What it does: A distributionally robust optimization framework that combines causal structure and sensitive attributes (Causally Fair DRO) is proposed to address individual fairness issues.
Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning
Takuo Matsubara (University of Edinburgh)
ClassificationOptimizationTabular
🎯 What it does: This paper proposes a gradient boosting framework based on Wasserstein gradient flow (WGBoost), which can predict the probability distribution of input (a set of particles) and achieve distribution prediction of tree models in evidential learning.
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Wenkai Yang (Renmin University of China), Xu Sun (Peking University)
Adversarial AttackRobotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningAgentic AIText
🎯 What it does: This paper studies and experiments with backdoor attacks on agents driven by large language models, proposing a general attack framework divided into three types: Query-Attack, Observation-Attack, and Thought-Attack.
Watermarking Makes Language Models Radioactive
Tom Sander (Meta), Teddy Furon (Inria)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: The study investigates whether watermark text used for LLM fine-tuning leaves detectable watermark traces (i.e., 'radioactivity') in the model.
WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off
Eva Giboulot (Inria), Teddy Furon (Inria)
GenerationTransformerLarge Language ModelTextBenchmark
🎯 What it does: The WaterMax watermarking scheme is proposed, achieving high detectability and almost no loss in text quality by selecting the most easily detectable text when generating multiple segments of text.
WATT: Weight Average Test Time Adaptation of CLIP
David OSOWIECHI, Christian Desrosiers (École de Technologie Supérieure)
ClassificationDomain AdaptationTransformerPrompt EngineeringVision Language ModelImage
🎯 What it does: This paper proposes the WATT method, which achieves adaptation during CLIP testing through multi-template text prompts and weight averaging, improving zero-shot classification performance in cross-domain tasks.
WaveAttack: Asymmetric Frequency Obfuscation-based Backdoor Attacks Against Deep Neural Networks
Jun Xia (East China Normal University), Mingsong Chen (University of Notre Dame)
ClassificationAdversarial AttackConvolutional Neural NetworkGenerative Adversarial NetworkImage
🎯 What it does: A backdoor attack method called WaveAttack based on Discrete Wavelet Transform (DWT) is proposed, which generates covert triggers using high-frequency components and employs asymmetric frequency obfuscation with different amplitudes during training and inference phases to balance attack effectiveness and sample quality.
Weak Supervision Performance Evaluation via Partial Identification
Felipe Maia Polo (University of Michigan), Yuekai Sun (University of Michigan)
OptimizationData-Centric LearningLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: In weakly supervised learning, a model evaluation method without true labels is proposed, transforming the evaluation into a Fréchet bounds problem through part recognition, thereby providing observable upper and lower bounds for metrics such as accuracy, precision, recall, and F1 score.
Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles
Qi Chen (Australian Institute for Machine Learning, University of Adelaide), Qi Wu (Australian Institute for Machine Learning, University of Adelaide)
TransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: The SPLAT benchmark is proposed to evaluate and stimulate the lateral thinking ability of large language models through situational puzzles.
Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
Zhanhui Zhou (Shanghai Artificial Intelligence Laboratory), Yu Qiao (Shanghai Artificial Intelligence Laboratory)
GenerationOptimizationTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: Without training large language models, alignment with large language models is achieved through greedy search of the log probability differences between small fine-tuned models and un-tuned models during inference.
Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach
Mathilde Caron (Google DeepMind), Ahmet Iscen (Google DeepMind)
RecognitionRetrievalTransformerLarge Language ModelPrompt EngineeringVision Language ModelImageTextMultimodalityBenchmarkRetrieval-Augmented Generation
🎯 What it does: Utilizing multimodal large language models (LLMs) to validate, correct, and generate explanatory notes and question-answer pairs for existing entity retrieval datasets, thereby constructing a high-quality large-scale visual entity recognition training set.
Weight decay induces low-rank attention layers
Seijin Kobayashi (ETH Zurich), Johannes von Oswald (Google)
TransformerImageText
🎯 What it does: This paper combines theoretical and experimental research to study the impact of using L2 regularization (weight decay) on matrix rank in the factorized parameterization of Transformers (such as key-query and value-projection). It proves that this is equivalent to nuclear norm regularization and shows exponential fast convergence in the gradient flow; experiments validate the performance of low-rank induction in language models, Vision Transformers, and pre-trained LLAMA2 weights.
Weight Diffusion for Future: Learn to Generalize in Non-Stationary Environments
Mixue Xie (Beijing Institute of Technology), Chengwei Zhu (Tencent)
Domain AdaptationDiffusion modelTabular
🎯 What it does: In the domain incremental setting, the evolution of domain generalization is studied, and the Weight Diffusion (W-Diff) framework is proposed. It utilizes a conditional diffusion model to learn the evolution patterns of classifier parameters and generates customized classifiers for future domains; simultaneously, it learns a domain-shared feature encoder to suppress overfitting; during inference, robustness is enhanced through a weight ensemble.
Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML
Tehila Dahan (Technion), Kfir Yehuda Levy
OptimizationFederated LearningImage
🎯 What it does: This study investigates robust training against Byzantine attacks in asynchronous distributed machine learning and proposes a weighted robust aggregation framework combined with dual momentum μ²-SGD, achieving optimal convergence rates for the first time in an asynchronous Byzantine environment.
WeiPer: OOD Detection using Weight Perturbations of Class Projections
Maximilian Granz (Free University of Berlin), Tim Landgraf (Free University of Berlin)
Domain AdaptationAnomaly DetectionConvolutional Neural NetworkTransformerSupervised Fine-TuningImage
🎯 What it does: This paper proposes a method to expand feature representation by applying random perturbations (WeiPer) to the weights of the final fully connected layer in the network, and implements post-hoc OOD detection using MSP or KL divergence scores.
Weisfeiler and Leman Go Loopy: A New Hierarchy for Graph Representational Learning
Raffaele Paolino (LMU Munich), Gitta Kutyniok (LMU Munich)
Representation LearningDrug DiscoveryGraph Neural NetworkGraph
🎯 What it does: Proposes the r-loopy Weisfeiler-Leman (r-WL) random hierarchy and the corresponding r-MPNN/GNN architecture, which can count cycles of length up to r+2 and all cactus graphs with edges within r+2.
What do Graph Neural Networks learn? Insights from Tropical Geometry
Tuan Anh Pham (University of Edinburgh), Vikas Garg (Aalto University)
Graph Neural NetworkGraph
🎯 What it does: This study investigates the class of functions that ReLU message passing GNNs (MPNNs) can learn and proves that it is equivalent to the continuous piecewise linear functions (CPLM) learned by ReLU feedforward networks (FNN).
What does guidance do? A fine-grained analysis in a simple setting
Muthu Chidambaram (Duke University), Jianfeng Lu (Duke University)
GenerationData SynthesisDiffusion modelImageOrdinary Differential Equation
🎯 What it does: A fine-grained theoretical analysis of the guidance mechanism in diffusion models is conducted, proving that guidance does not sample from a skewed distribution, and revealing its dynamic behavior under different distributions (tight support mixture distributions and Gaussian mixtures), pointing out that excessive guidance can lead to samples deviating from the support.
What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration
Libo Qin (Central South University), Wanxiang Che (Harbin Institute of Technology)
GenerationRetrievalTransformerPrompt EngineeringVision Language ModelImageTextMultimodality
🎯 What it does: Systematic experiments on the core steps of multimodal context learning (MM-ICL) (retrieval, ranking, construction) explored the impact of 20 strategies on 4 tasks (image-text generation, visual question answering, image classification, chain reasoning) across 6 VLLMs.
What If the Input is Expanded in OOD Detection?
Boxuan Zhang (Wuhan University), Bo Han (Hong Kong Baptist University)
Domain AdaptationAnomaly DetectionConvolutional Neural NetworkTransformerVision Language ModelImage
🎯 What it does: The CoVer framework is proposed, which expands the input dimensions using common image corruptions (90 types) and averages the multi-dimensional confidence to enhance the detection capability for OOD samples.
What Is Missing For Graph Homophily? Disentangling Graph Homophily For Graph Neural Networks
Yilun Zheng (Nanyang Technological University), Lihui Chen (Mila - Quebec Artificial Intelligence Institute)
Graph Neural NetworkGraph
🎯 What it does: The paper studies and dissects graph homophily into three dimensions: labels, structure, and features, proposing a new composite metric called Tri-Hom.
What is my quantum computer good for? Quantum capability learning with physics-aware neural networks
Daniel Hothem (Sandia National Laboratories), Timothy Proctor (Sandia National Laboratories)
Graph Neural NetworkGraphPhysics Related
🎯 What it does: A neural network architecture based on quantum physics knowledge (qpa-NN) is proposed to predict the performance (success rate or process fidelity) of quantum computers on different quantum circuits.
What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Samyak Jain (Five AI), Puneet K. Dokania (Five AI)
Data SynthesisSafty and PrivacyAdversarial AttackTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: A synthetic data generation framework is proposed for systematically studying the mechanisms of safe fine-tuning and its adversarial attacks;
What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights
Xin Wen (University of Hong Kong), XIAOJUAN QI
ClassificationRepresentation LearningTransformerContrastive LearningImageText
🎯 What it does: Conduct systematic controlled experiments on the robustness of CLIP with long-tail pre-training data, exploring the impact of language supervision, pretext tasks, data distribution, scale, and open-world concepts, and transferring findings to supervised and self-supervised learning.
What Makes Partial-Label Learning Algorithms Effective?
Jiaqi Lv (Southeast University), Xin Geng (Southeast University)
ClassificationData-Centric LearningContrastive LearningImageBenchmark
🎯 What it does: The system analyzes and summarizes effective design principles of partial label learning (PLL) methods, proving that the most critical factor is mini-batch PL purification. Based on this, it proposes an instance-dependent warm-up strategy called StreamPurify and a minimal working algorithm named SASM.
What makes unlearning hard and what to do about it
Kairan Zhao (University of Warwick), Peter Triantafillou (University of Warwick)
ClassificationConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: This paper conducts a systematic study on the ease of unlearning in machine learning models, analyzing the mixing degree of the forget set and the retain set in the embedding space, as well as the memorization degree of the forget set on different algorithms. It proposes a refined RUM framework to enhance unlearning performance.
What Matters in Graph Class Incremental Learning? An Information Preservation Perspective
Jialu Li (Tianjin University), Qinghua Hu (Tianjin University)
Graph Neural NetworkGraph
🎯 What it does: This paper proposes a Graph Space Information Preservation framework (GSIP) for graph-based incremental learning, which aligns the low-frequency local-global information and high-frequency neighbor similarity from the outputs of the old model to reduce semantic and structural shifts, thereby alleviating catastrophic forgetting.
What matters when building vision-language models?
Hugo Laurençon (Hugging Face), Victor Sanh (Hugging Face)
RecognitionGenerationRetrievalTransformerSupervised Fine-TuningVision Language ModelImageTextMultimodality
🎯 What it does: This paper investigates and systematically evaluates the core design choices of Visual Language Models (VLM) and develops an 8B parameter Idefics2 VLM based on experimental results.
What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information
Yiting Chen (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Analyzed the impact of RoPE weight vector angles on attention, revealing that non-orthogonal weights are sensitive to basic grammatical information, while orthogonal weights are sensitive to advanced semantic information; based on this, proposed the Angle Weight Masking (AWM) method, which updates only the orthogonal weight pairs during fine-tuning, significantly reducing the number of trainable parameters while maintaining or improving performance.
What type of inference is planning?
Miguel Lazaro-Gredilla, Dileep George (Google Deepmind)
OptimizationReinforcement Learning from Human FeedbackReinforcement LearningTabular
🎯 What it does: This paper rephrases the planning task as a probabilistic inference problem, deriving a variational formula for planning, and proposes Value Belief Propagation (VBP) and scalable VI-LP approximation methods for sparse state factor MDPs.
What Variables Affect Out-of-Distribution Generalization in Pretrained Models?
Md Yousuf Harun (Rochester Institute of Technology), Christopher Kanan (University of Rochester)
Domain AdaptationRepresentation LearningConvolutional Neural NetworkTransformerImage
🎯 What it does: This study investigates the transferability of pre-trained deep networks under different conditions and their generalization ability to OOD (out-of-distribution) data. It systematically evaluates the impact of variables such as network architecture, training data diversity, image resolution, and data augmentation on the 'tunnel effect' and OOD performance.
When are dynamical systems learned from time series data statistically accurate?
Jeongjin Park, Nisha Chandramoorthy (University of Chicago)
Recurrent Neural NetworkSupervised Fine-TuningTime SeriesPhysics RelatedStochastic Differential EquationOrdinary Differential Equation
🎯 What it does: This paper studies the generality and statistical accuracy issues of learning dynamical systems from time series data, proposing a novel generalization definition based on ergodic theory, and theoretically and experimentally verifying that Jacobian matching training can enhance the statistical accuracy of the model.
When does perceptual alignment benefit vision representations?
Shobhita Sundaram, Phillip Isola
SegmentationDepth EstimationRetrievalTransformerSupervised Fine-TuningImageRetrieval-Augmented Generation
🎯 What it does: This study enhances the performance of large visual models (such as CLIP, DINO, DINOv2, etc.) on various visual downstream tasks by conducting secondary pre-training (alignment) on the NIGHTS human similarity judgment dataset.
When is an Embedding Model More Promising than Another?
Maxime DARRIN (McGill University), Pablo Piantanida
Representation LearningDrug DiscoveryTextBenchmark
🎯 What it does: This paper proposes a unified, label-free embedding model evaluation method that constructs comparable metrics using the concepts of sufficiency, informativeness, and defects from information theory.
When Is Inductive Inference Possible?
Zhou Lu (Princeton University)
🎯 What it does: This paper proposes a non-uniform online learning framework and proves that the sufficient and necessary condition for inference to be completed with finite errors under both realizable and unrealizable settings is that the hypothesis class is a countable union of finite Littlestone dimension classes.
When is Multicalibration Post-Processing Necessary?
Dutch Hansen (University of Southern California), Vatsal Sharan (University of Southern California)
ClassificationOptimizationConvolutional Neural NetworkTransformerImageTextTabular
🎯 What it does: This paper evaluates the performance of the multicalibration post-processing method across various models and datasets through large-scale experiments and provides practical guidance.
When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search
Xuan Chen (Purdue University), Xiangyu Zhang (Purdue University)
Adversarial AttackTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: A black-box jailbreak attack method based on deep reinforcement learning, called RLbreaker, is proposed for automatically generating and optimizing Jailbreaking Prompts.
When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding
Marah Ghoummaid (Technion), Uri Shalit (Technion)
Tabular
🎯 What it does: Designed and implemented the CARED method to learn a causal decision-making strategy that can be deferred to experts based on observational data with hidden confounding.
When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL
Lenart Treven (ETH Zurich), Andreas Krause (ETH Zurich)
Robotic IntelligenceReinforcement LearningStochastic Differential Equation
🎯 What it does: This paper proposes a Time-Adaptive Control and Perception Framework (TACOS), which transforms continuous-time RL problems (including interaction costs or interaction count limitations) into equivalent discrete-time MDPs, thereby utilizing existing RL algorithms to achieve fewer system interactions.
When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
Leon Lang (University of Amsterdam), Scott Emmons (University of California Berkeley)
Reinforcement Learning from Human FeedbackReinforcement Learning
🎯 What it does: This study investigates the potential risks associated with using Reinforcement Learning from Human Feedback (RLHF) in partially observable environments, providing corresponding theoretical analysis and experimental evidence.
Where Do Large Learning Rates Lead Us?
Ildus Sadrtdinov (Constructor University), Dmitry Vetrov (Constructor University)
OptimizationConvolutional Neural NetworkTransformerImage
🎯 What it does: In a controlled experimental setup, the impact of different initial learning rates on the final solutions of neural networks was systematically studied. It was proposed that a learning rate within the 'sub-interval 2A', slightly above the convergence threshold, can locate low valleys of high-quality solutions during the pre-training phase, and achieve optimal generalization performance after fine-tuning or weight averaging. Additionally, this learning rate range encourages the network to learn sparser and more targeted features. The experiments were extended to practical training settings, validating the universality of the conclusions.
Where does In-context Learning Happen in Large Language Models?
Suzanna Sia (Johns Hopkins University), Kevin Duh (Johns Hopkins University)
Computational EfficiencyAI Code AssistantTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This study investigates at which layer of large language models task recognition occurs during in-context learning;
Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval
Dvir Samuel (Bar-Ilan University), Gal Chechik (NVIDIA Research)
SegmentationRetrievalDiffusion modelImageVideo
🎯 What it does: This paper proposes PDM, a method for personalized retrieval and segmentation that utilizes intermediate features from a pre-trained text-to-image diffusion model without requiring additional training.
Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)
Michael Saxon, William Yang Wang
GenerationComputational EfficiencyVision Language ModelImageTextBenchmark
🎯 What it does: A T2IScoreScore benchmark is proposed, utilizing Semantic Error Graphs (SEG) and three meta-metrics (rank, sep, delta) for objective evaluation of text-image (prompt-faithfulness) metrics.
Who's asking? User personas and the mechanics of latent misalignment
Asma Ghandeharioun (Google Research), Lucas Dixon (Google Research)
Safty and PrivacyAdversarial AttackTransformerPrompt EngineeringText
🎯 What it does: The study investigates the rejection behavior of secure training models when faced with malicious queries and finds that user personas significantly influence whether the model leaks harmful information; by manipulating personas through activation steering and prompt prefix methods, the impact on rejection and hidden representations is explored.
Who’s Gaming the System? A Causally-Motivated Approach for Detecting Strategic Adaptation
Trenton Chang (University of Michigan), Jenna Wiens (University of Michigan)
Anomaly DetectionOptimizationTabular
🎯 What it does: A framework based on causal inference is proposed to rank strategic game behaviors (such as upcoding) in a multi-agent environment and identify the agents most likely to cheat.
Why are Visually-Grounded Language Models Bad at Image Classification?
Yuhui Zhang (Stanford University), Serena Yeung-Levy (Stanford University)
ClassificationTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageText
🎯 What it does: Evaluating and enhancing the performance of Visual Language Models (VLM) in image classification tasks.
Why Do We Need Weight Decay in Modern Deep Learning?
Francesco D'Angelo (Ecole Polytechnique Federale de Lausanne), Nicolas Flammarion (Ecole Polytechnique Federale de Lausanne)
OptimizationConvolutional Neural NetworkTransformerLarge Language ModelImageText
🎯 What it does: This paper explores the role of weight decay in different training paradigms (overfitting and underfitting) in modern deep learning through experimental and theoretical analysis, revealing that it is not merely explicit regularization, but rather enhances performance by altering optimization dynamics.
Why Go Full? Elevating Federated Learning Through Partial Network Updates
Haolin Wang (Beihang University), Shaojie Tang (University at Buffalo)
Federated LearningComputational EfficiencyConvolutional Neural NetworkTransformerImageText
🎯 What it does: This paper studies the hierarchical mismatch problem caused by full model updates in federated learning and proposes the FedPart method, which updates only part of the network layers in each round and designs a layer selection strategy, significantly improving model convergence speed and accuracy.
Why the Metric Backbone Preserves Community Structure
Maximilien Dreveton (École Polytechnique Fédérale de Lausanne), Patrick Thiran (École Polytechnique Fédérale de Lausanne)
Graph Neural NetworkGraph
🎯 What it does: This study investigates the retention of community structure by the metric backbone (the union of all shortest paths) in the weighted random block model, and based on this, proposes practical methods for graph sparsification and construction using the metric backbone.
Why Transformers Need Adam: A Hessian Perspective
Yushun Zhang (Chinese University of Hong Kong), Zhi-Quan Luo (Chinese University of Hong Kong)
OptimizationConvolutional Neural NetworkTransformerImageText
🎯 What it does: This paper conducts a fine-grained analysis of the Hessian block spectrum of Transformer and CNN, proposing that the block heterogeneity in Transformers is the reason for the performance gap between SGD and Adam.
Why Warmup the Learning Rate? Underlying Mechanisms and Improvements
Dayal Singh Kalra (University of Maryland), Maissam Barkeshli (University of Maryland)
OptimizationConvolutional Neural NetworkTransformerImageText
🎯 What it does: This paper systematically studies the mechanism of learning rate warmup, revealing that its main function is to allow the network to tolerate higher learning rates, and proposes a more efficient learning rate initialization and an improved Adam (GI-Adam).
Wide Two-Layer Networks can Learn from Adversarial Perturbations
Soichiro Kumano (University of Tokyo), Toshihiko Yamasaki (University of Tokyo)
Adversarial AttackTabular
🎯 What it does: This paper theoretically proves that in sufficiently wide two-layer networks, perturbation learning can successfully learn discriminative features from mislabelled adversarial samples that are the same as those from the original clean samples, and it demonstrates that adversarial perturbations actually encompass the class features of the entire training set.
Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections
Jiacong Xu (Johns Hopkins University), Vishal M. Patel (Johns Hopkins University)
GenerationData SynthesisComputational EfficiencyGaussian SplattingImage
🎯 What it does: The Wild-GS method is proposed, adapting 3D Gaussian Splatting to unconstrained photo collections for real-time novel view synthesis.
WildGaussians: 3D Gaussian Splatting In the Wild
Jonas Kulhanek (Czech Technical University in Prague), Torsten Sattler (Czech Technical University in Prague)
RestorationGenerationContrastive LearningGaussian SplattingPoint Cloud
🎯 What it does: We propose WildGaussians, which combines 3D Gaussian Splatting with DINO-based uncertainty prediction to achieve high-quality real-time rendering in wild scenes with occlusion and lighting variations.