ICLR 2024 Papers — Page 22
International Conference on Learning Representations · 2260 papers
Training Diffusion Models with Reinforcement Learning
Kevin Black (University of California), Sergey Levine (University of California)
GenerationOptimizationReinforcement LearningVision Language ModelDiffusion modelImage
🎯 What it does: A DDPO framework based on reinforcement learning is proposed, modeling the denoising process of diffusion models as an MDP using multi-step decision trees, directly optimizing downstream objectives (such as image compressibility, aesthetic quality, and prompt-image alignment) and achieving automatic alignment without human annotations.
Training Graph Transformers via Curriculum-Enhanced Attention Distillation
Yisong Huang (Fuzhou University), Yang-Geng Fu (Fuzhou University)
Knowledge DistillationGraph Neural NetworkTransformerGraph
🎯 What it does: A curriculum learning-based attention distillation framework is proposed, where a Local Graph Transformer (LGT) teacher guides a Global Graph Transformer (GGT) student to perform semi-supervised node classification.
Training Socially Aligned Language Models on Simulated Social Interactions
Ruibo Liu (Dartmouth College), Soroush Vosoughi (Dartmouth College)
Large Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: Utilizing multi-agent interactions in a simulated society (SANDBOX) to train language models for alignment with social values.
Training Unbiased Diffusion Models From Biased Dataset
Yeongmin Kim (Korea Advanced Institute of Science and Technology), Il-chul Moon
GenerationData SynthesisDiffusion modelScore-based ModelImage
🎯 What it does: This paper proposes a time-dependent importance reweighting (TIW) diffusion model that corrects dataset bias by estimating the time-varying density ratio and achieves unbiased score matching.
Training-free Multi-objective Diffusion Model for 3D Molecule Generation
Xu Han (Tufts University), Dongsheng Li (Microsoft Research)
GenerationData SynthesisDrug DiscoveryDiffusion modelGraphStochastic Differential Equation
🎯 What it does: A training-free multi-objective 3D molecular generation method MUDM is proposed, utilizing a pre-trained unconditional diffusion model and a time-independent attribute prediction function for controllable generation.
Trajeglish: Traffic Modeling as Next-Token Prediction
Jonah Philion (NVIDIA), Sanja Fidler (NVIDIA)
Autonomous DrivingTransformerLarge Language ModelTime SeriesSequentialBenchmark
🎯 What it does: Discretize the trajectories of multiple agents such as vehicles, pedestrians, and bicycles into a small number of discrete action symbols, and use a GPT-like autoregressive Transformer to model these discrete actions to generate realistic roll-outs of multi-agent traffic scenarios given an initial state.
TRAM: Bridging Trust Regions and Sharpness Aware Minimization
Tom Sherborne (University of Edinburgh), Hao Peng (University of Illinois Urbana-Champaign)
Domain AdaptationOptimizationSupervised Fine-TuningImageText
🎯 What it does: A new fine-tune optimization algorithm TRAM is proposed, which combines sharpness-aware minimization with trust region regularization to enhance the model's generalization ability on out-of-distribution (OOD) data.
Transferring Labels to Solve Annotation Mismatches Across Object Detection Datasets
Yuan-Hong Liao (NVIDIA), Sanja Fidler (Georgia Institute of Technology)
Object DetectionDomain AdaptationAutonomous DrivingImage
🎯 What it does: In target detection, the issue of annotation mismatch caused by different dataset labeling protocols is addressed by proposing the concept of label transfer and developing the Label-Guided Pseudo-Labeling (LGPL) method at the data center;
Transferring Learning Trajectories of Neural Networks
Daiki Chijiwa (NTT Corporation)
OptimizationConvolutional Neural NetworkImage
🎯 What it does: A learning transfer method is proposed and implemented, which transfers the learning trajectory of a trained network from one initialization to another, resulting in parameters that can be used directly or significantly accelerate training.
Transformer Fusion with Optimal Transport
Moritz Imfeld (ETH Zurich), Sidak Pal Singh (ETH Zurich)
TransformerImageText
🎯 What it does: Integrate multiple Transformer models and use Optimal Transport to achieve parameter alignment and fusion.
Transformer-Modulated Diffusion Models for Probabilistic Multivariate Time Series Forecasting
Yuxin Li (Xidian University), Mingyuan Zhou (University of Texas at Austin)
TransformerDiffusion modelTime Series
🎯 What it does: A Transformer-based Diffusion Model (TMDM) is proposed for probabilistic forecasting of multivariate time series, capable of simultaneously outputting predicted values and uncertainty confidence intervals.
Transformer-VQ: Linear-Time Transformers via Vector Quantization
Lucas Dax Lingle
GenerationCompressionComputational EfficiencyTransformerImageText
🎯 What it does: This paper proposes Transformer-VQ, a decoder Transformer that utilizes vector quantization (VQ) to achieve linear time self-attention.
Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
Licong Lin (University of California), Song Mei (University of California)
TransformerSupervised Fine-TuningReinforcement LearningSequential
🎯 What it does: This paper studies and proves that using supervised pre-trained Transformers can perform reinforcement learning in new environments (ICRL) and can approximate classic algorithms such as LinUCB, Thompson sampling, and UCB-VI.
Transformers can optimally learn regression mixture models
Reese Pathak (University of California, Berkeley), Abhimanyu Das (Google Research)
TransformerTabular
🎯 What it does: This paper studies the learning effectiveness of Transformer in mixed linear regression models, proving that Transformer can achieve the Bayesian optimal posterior mean predictor and nearly reach oracle performance in experiments.
Transport meets Variational Inference: Controlled Monte Carlo Diffusions
Francisco Vargas (University of Cambridge), Nikolas Nüsken (King's College London)
OptimizationDiffusion modelTabularBenchmarkStochastic Differential Equation
🎯 What it does: This paper proposes a framework that unifies optimal transport and variational inference, and based on this, designs the Controlled Monte Carlo Diffusion (CMCD) sampler, which can simultaneously learn the forward and backward diffusion processes.
Traveling Waves Encode The Recent Past and Enhance Sequence Learning
T. Anderson Keller (Kempner Institute for the Study of Natural and Artificial Intelligence Harvard University), Max Welling (Amsterdam Machine Learning Lab University of Amsterdam)
Recurrent Neural NetworkSequential
🎯 What it does: A novel recurrent neural network architecture called wRNN is proposed, which allows the hidden state to exhibit reversible propagating fluctuations for effectively storing recent sequence information.
Treatment Effects Estimation By Uniform Transformer
Ruoqi Yu (University of Illinois Urbana-Champaign), Shulei Wang (University of Illinois Urbana-Champaign)
TransformerTabular
🎯 What it does: A new weighted framework (WUNT) is proposed, which uses an adaptive uniform transformer to map the covariate distribution of the control group to a uniform distribution. Weights are then obtained through kernel density or projection estimation, and the treatment effect is directly estimated in the form of U statistics.
Tree Cross Attention
Leo Feng (Mila - Universite de Montreal), Mohamed Osama Ahmed (Borealis AI)
RetrievalOptimizationTransformerReinforcement LearningMultimodality
🎯 What it does: Proposes Tree Cross Attention (TCA) and ReTreever architecture to achieve logarithmic token retrieval and efficient inference through cross-attention;
Tree Search-Based Policy Optimization under Stochastic Execution Delay
David Valensi (Technion), Gal Dalal (Nvidia Research)
OptimizationReinforcement LearningSequential
🎯 What it does: A framework for MDPs with stochastic execution delays (SED-MDP) has been designed and implemented, and within this framework, the DEZ algorithm is proposed, which utilizes delay queues and effective decision time to achieve state-agnostic policy learning and inference.
Tree-Planner: Efficient Close-loop Task Planning with Large Language Models
Mengkang Hu (University of Hong Kong), Ping Luo (University of Hong Kong)
TransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes TREE-PLANNER, an LLM approach that divides task planning into three stages: plan sampling, action tree construction, and environment-based decision-making.
True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning
Weihao Tan (Nanyang Technological University), Bo An (Nanyang Technological University)
Robotic IntelligenceTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: This study proposes the TWOSOME framework, which aligns large language models with embodied environments online through reinforcement learning to efficiently complete decision-making tasks.
Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
Bingchen Zhao (University of Edinburgh), Cihang Xie (University of California Santa Cruz)
Domain AdaptationComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodality
🎯 What it does: During the fine-tuning process of multimodal large language models (MLLM), only the LayerNorm parameters in the attention blocks are tuned;
Turning large language models into cognitive models
Marcel Binz (Max Planck Institute for Biological Cybernetics), Eric Schulz (Max Planck Institute for Biological Cybernetics)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: By fine-tuning the final layer of the large language model LLaMA and generating text embeddings using psychological experimental data, an interpretable cognitive model called CENTaUR was constructed to simulate human decision-making behavior.
TUVF: Learning Generalizable Texture UV Radiance Fields
An-Chieh Cheng (University of California San Diego), Xiaolong Wang (NVIDIA)
GenerationData SynthesisDiffusion modelAuto EncoderGenerative Adversarial NetworkPoint CloudMesh
🎯 What it does: This paper proposes Texture UV Radiance Fields (TUVF), which decouples texture from 3D shapes by generating textures in a learnable UV spherical space, enabling texture editing and cross-shape transfer.
Two-stage LLM Fine-tuning with Less Specialization and More Generalization
Yihan Wang (University of California Los Angeles), Sanjiv Kumar (Google)
TransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: This paper discusses the issue of over-specialization leading to a decline in generalization during the single-task fine-tuning process of large language models (LLMs), and proposes a two-stage fine-tuning framework called ProMoT, which first learns the task format using soft prompts (Prompt tuning), then freezes the soft prompts to fine-tune the model, ultimately achieving the goal of reducing format specialization and enhancing generalization.
Two-timescale Extragradient for Finding Local Minimax Points
Jiseok Chae (Korea Advanced Institute of Science and Technology), Donghwan Kim (Korea Advanced Institute of Science and Technology)
OptimizationOrdinary Differential Equation
🎯 What it does: This paper studies the convergence properties of the two-timescale extragradient method in non-convex non-concave minimax problems. It proves that this method is more robust compared to the traditional gradient descent ascent (GDA) method, capable of converging to local minimax points without requiring the Hessian of the maximization variable to be non-degenerate, and almost certainly avoids strict non-minimax points. It also provides theoretical conditions for global convergence.
UC-NERF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving
Kai Cheng (University of Science and Technology of China), Xuejin Chen (University of Science and Technology of China)
GenerationPose EstimationDepth EstimationAutonomous DrivingNeural Radiance FieldImage
🎯 What it does: This paper proposes UC-NeRF, which addresses the issue of reduced rendering quality in multi-camera autonomous driving scenarios caused by color inconsistencies and pose errors due to inaccurate camera calibration in NeRF.
Un-Mixing Test-Time Normalization Statistics: Combatting Label Temporal Correlation
Devavrat Tomar (École Polytechnique Fédérale de Lausanne), Behzad Bozorgtabar (École Polytechnique Fédérale de Lausanne)
ClassificationObject TrackingDomain AdaptationImageVideo
🎯 What it does: A test-time normalization layer named UnMix-TNS is proposed, which can achieve adaptive normalization in label time-dependent non-i.i.d. test streams, enhancing model robustness.
Unbalancedness in Neural Monge Maps Improves Unpaired Domain Translation
Luca Eyring (Tübingen AI Center), Fabian J Theis
Image TranslationDomain AdaptationDrug DiscoveryGenerative Adversarial NetworkImageBiomedical Data
🎯 What it does: The study investigates how to introduce imbalance into arbitrary neural Monge mappers within the framework of Unbalanced Optimal Transport (UOT) to enhance the performance of unpaired domain translation tasks.
Unbiased Watermark for Large Language Models
Zhengmian Hu (University of Maryland), Heng Huang (University of Pittsburgh)
GenerationTransformerLarge Language ModelText
🎯 What it does: A bias-free watermarking framework has been designed and validated, which can embed watermarks in large language models without affecting the quality of generation.
Uncertainty Quantification via Stable Distribution Propagation
Felix Petersen (Stanford University), Mikhail Yurochkin (MIT IBM Watson AI Lab)
Anomaly DetectionComputational EfficiencyConvolutional Neural NetworkImageTabularStochastic Differential Equation
🎯 What it does: By propagating α-stable distributions such as Gaussian and Cauchy through Stable Distribution Propagation (SDP) without the need for sampling, the impact of input uncertainty on prediction results is quantified, enabling confidence interval predictions and out-of-distribution (OOD) selective predictions.
Uncertainty-aware Constraint Inference in Inverse Constrained Reinforcement Learning
Sheng Xu (Chinese University of Hong Kong), Guiliang Liu (Chinese University of Hong Kong)
Autonomous DrivingOptimizationReinforcement LearningTabularSequential
🎯 What it does: This paper proposes an uncertainty-aware inverse constraint reinforcement learning framework (UAICRL) that can infer safety constraints from expert demonstrations and learn policies that adhere to these constraints.
Uncertainty-aware Graph-based Hyperspectral Image Classification
Linlin Yu (University of Texas at Dallas), Feng Chen (University of Texas at Dallas)
ClassificationGraph Neural NetworkImage
🎯 What it does: The paper proposes a method for uncertainty quantification in hyperspectral image classification based on Graph Convolutional Networks (GCN). It models pixels using Evidential GCN and Graph Posterior Network, and incorporates Unmixing Regularization (UR) based on a physical mixing model and Total Variation Regularization (TV) based on edge preservation during training to enhance the detection capability for unknown classes (OOD) and misclassifications.
Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learning
James Chapman (University College London), Ana Lawry Aguila (University College London)
OptimizationRepresentation LearningContrastive LearningImageMultimodalityBiomedical Data
🎯 What it does: This paper proposes a general GEP (Generalized Eigenvalue Problem) framework based on the Eckhart-Young unconstrained objective, unifying linear, deep, and multi-view CCA into a form suitable for optimization using stochastic gradient descent. Based on this, efficient algorithms such as CCA-EY, PLS-EY, DCCA-EY, and SSL-EY are designed.
Understanding Addition in Transformers
Philip Quirke (Apart Research), Fazl Barez (University of Oxford)
TransformerTabular
🎯 What it does: Research and reverse engineer the internal algorithm of a single-layer Transformer in the integer addition task.
Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks
Hao Chen (Carnegie Mellon University), Bhiksha Raj (Carnegie Mellon University)
ClassificationDomain AdaptationConvolutional Neural NetworkTransformerSupervised Fine-TuningVision Language ModelImageText
🎯 What it does: This study investigates the impact of label noise in pre-training data on downstream tasks and proposes a lightweight black-box tuning method, NMTune, to mitigate the negative effects of noise.
Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression
Runtian Zhai (Carnegie Mellon University), Pradeep Kumar Ravikumar
Representation LearningImageText
🎯 What it does: This paper provides a theoretical analysis of augmented-driven self-supervised representation learning from the perspective of kernel spaces and presents non-parametric generalization bounds for any encoder.
Understanding Catastrophic Forgetting in Language Models via Implicit Inference
Suhas Kotha (Carnegie Mellon University), Aditi Raghunathan (Carnegie Mellon University)
TransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: Through experiments on linear regression simulations and real large language models, this study investigates the catastrophic forgetting caused by fine-tuning and proposes a Conjugate Prompting strategy to recover the suppressed pre-training capabilities.
Understanding Certified Training with Interval Bound Propagation
Yuhao Mao (ETH Zürich), Martin Vechev (ETH Zürich)
OptimizationAdversarial AttackConvolutional Neural NetworkReinforcement LearningImage
🎯 What it does: Theoretical and experimental analysis of the provably robust network training mechanism based on Interval Bound Propagation (IBP) is conducted, and a measure of propagation tightness is proposed.
Understanding Convergence and Generalization in Federated Learning through Feature Learning Theory
Wei Huang (RIKEN AIP), Taiji Suzuki (RIKEN AIP)
Federated LearningConvolutional Neural NetworkImage
🎯 What it does: This paper analyzes the convergence and generalization of federated learning (FedAvg) on a two-layer convolutional neural network through the theory of feature learning, and proposes a weighted aggregation method based on client feature similarity to alleviate the problem of heterogeneous data.
Understanding Domain Generalization: A Noise Robustness Perspective
Rui Qiao (National University of Singapore), Bryan Kian Hsiang Low (National University of Singapore)
Domain AdaptationConvolutional Neural NetworkImage
🎯 What it does: This paper explores whether domain generalization (DG) algorithms outperform traditional empirical risk minimization (ERM) in the presence of label noise, providing both theoretical and experimental analysis.
Understanding Expressivity of GNN in Rule Learning
Haiquan Qiu (Tsinghua University), quanming yao
Graph Neural NetworkGraph
🎯 What it does: This study investigates the expressive power of state-of-the-art graph neural networks (such as NBFNet and RED-GNN) in learning rules for knowledge graph reasoning, and proposes enhancing expressiveness through labeling high-degree nodes (EL-GNN).
Understanding In-Context Learning from Repetitions
Jianhao Yan (Zhejiang University), Yue Zhang (Westlake Institute for Advanced Study)
TransformerLarge Language ModelReinforcement LearningText
🎯 What it does: Proposes a surface repetition-based word co-occurrence reinforcement mechanism to explain the behavior of large language models in context learning and the reasons for their successes and failures.
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions
Satwik Bhattamishra (University of Oxford), Varun Kanade (University of Oxford)
TransformerLarge Language ModelSequential
🎯 What it does: This study investigates the contextual learning capabilities of Transformer and other architectures within an autoregressive framework for Boolean functions; it explores the enhancement of sample efficiency through teaching sequences and the potential of pre-trained LLMs to implement learning algorithms.
Understanding prompt engineering may not require rethinking generalization
Victor Akinwande (Carnegie Mellon University), J Zico Kolter
TransformerLarge Language ModelPrompt EngineeringVision Language ModelImage
🎯 What it does: The research highlights the generalization performance of prompt engineering in visual-language models, and provides a tight generalization upper bound using the PAC-Bayes framework by constructing a discrete prompt space with a prior from the language model.
Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation
Noel Loo (Massachusetts Institute of Technology), Daniela Rus (Massachusetts Institute of Technology)
Anomaly DetectionKnowledge DistillationAdversarial AttackConvolutional Neural NetworkNeural Radiance FieldImage
🎯 What it does: A more powerful dataset reconstruction attack is proposed, utilizing the Neural Tangent Kernel (NTK) and KKT conditions to recover all training samples from the trained network parameters, and this attack is associated with dataset distillation (KIP).
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Robert Kirk (University College London), Roberta Raileanu (Meta)
GenerationReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: This paper systematically evaluates three fine-tuning methods for large language models (SFT, RLHF, BoN) in terms of outlier distribution generalization and output diversity in summarization and instruction-following tasks, revealing that RLHF outperforms SFT in generalization but significantly reduces diversity, and proposes a multi-dimensional evaluation framework.
Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift
Yihao Xue (University of California), Baharan Mirzasoleiman (University of California)
ClassificationDomain AdaptationRepresentation LearningContrastive LearningImageTextMultimodality
🎯 What it does: This study investigates the robustness of multimodal contrastive learning (MMCL) under distribution shifts, providing theoretical analysis and validating two mechanisms—within-class contrast and cross-class feature sharing—as well as the impact of caption richness on robustness.
Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks
Nguyen Hung-Quang, Khoa D Doan
Adversarial AttackConvolutional Neural NetworkTransformerGaussian SplattingImage
🎯 What it does: A lightweight defense method is proposed to inject random noise into intermediate layer features during the inference phase to reduce the success rate of query-based adversarial attacks.
Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
Zixiang Chen (University of California), Quanquan Gu (Carnegie Mellon University)
Representation LearningTransformerContrastive LearningImageText
🎯 What it does: This paper provides a systematic theoretical analysis of the transferable representation learning and zero-shot transfer mechanism of CLIP, and based on this, proposes a new regularization method to enhance the model's margin and zero-shot performance.
Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates
Nicholas Corrado, Josiah P. Hanna (University of Wisconsin)
Reinforcement Learning
🎯 What it does: This paper evaluates the impact of using dynamic invariant data augmentation on model-agnostic reinforcement learning in sparse reward tasks through experiments, exploring the roles of state-action coverage, reward density, and augmentation replay ratio.
Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
Kun LEI, Huazhe Xu (Tsinghua University)
Reinforcement LearningTabular
🎯 What it does: A unified offline and online deep reinforcement learning framework, Uni-O4, is designed to achieve seamless two-stage switching using the on-policy objective of PPO, and to implement multi-step policy improvement through behavior cloning ensembles and model-based OPE.
Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback
Yifu Yuan (Tianjin University), YAN ZHENG
Autonomous DrivingReinforcement Learning from Human FeedbackTransformerReinforcement LearningSequentialBenchmark
🎯 What it does: This paper proposes Uni-RLHF, a unified RLHF platform that includes annotation interfaces for multiple types of human feedback, a scalable crowdsourced annotation pipeline, and reproducible offline RLHF baselines.
Uni3D: Exploring Unified 3D Representation at Scale
Junsheng Zhou (Tsinghua University), Xinlong Wang (Beijing Academy of Artificial Intelligence)
ClassificationRecognitionSegmentationRetrievalTransformerContrastive LearningImageTextMultimodalityPoint Cloud
🎯 What it does: We propose Uni3D, a scalable 3D foundation model with up to 1 billion parameters, which learns a unified 3D representation by transplanting the ViT structure as a point cloud encoder and aligning across modalities (point clouds, images, text) within a pre-training framework.
UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
Haoyu Lu (Renmin University of China), Mingyu Ding (University of California)
RetrievalTransformerVision Language ModelImageVideoTextMultimodality
🎯 What it does: A parameter-efficient cross-modal transfer learning framework named UniAdapter is designed, which can adapt to various cross-modal downstream tasks (such as retrieval, question answering, captioning, etc.) by fine-tuning only a small number of parameters on a frozen large-scale vision-language pre-trained model.
Unified Generative Modeling of 3D Molecules with Bayesian Flow Networks
Yuxuan Song (Institute of AI Industry Research), Wei-Ying Ma (Institute of AI Industry Research)
GenerationDrug DiscoveryGraph Neural NetworkFlow-based ModelGraph
🎯 What it does: Proposes GeoBFN, which uses Bayesian flow networks to generate 3D molecular geometries.
Unified Human-Scene Interaction via Prompted Chain-of-Contacts
Zeqi Xiao (Shanghai AI Laboratory), Jiangmiao Pang (Shanghai AI Laboratory)
Robotic IntelligenceTransformerLarge Language ModelReinforcement LearningPrompt EngineeringTextMultimodality
🎯 What it does: Proposes the UniHSI framework, which utilizes an LLM planner to convert natural language commands into a Chain of Contacts plan, executed step-by-step by a unified controller, enabling long-term human-scene interactions with multiple objects and multiple steps.
Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization
Yang Jin (Peking University), Yadong MU
GenerationData SynthesisRetrievalTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: A unified multimodal foundation model LaVIT is designed and trained, which converts images into discrete tokens that can be processed in parallel with text through a dynamic discrete visual tokenizer, predicting the next token in an autoregressive manner within the LLM, achieving understanding and generation of image-text pairs.
Unified Projection-Free Algorithms for Adversarial DR-Submodular Optimization
Mohammad Pedramfar (Purdue University), Vaneet Aggarwal (Purdue University)
Optimization
🎯 What it does: A series of unified projection-free Frank-Wolfe type algorithms are proposed for the online adversarial continuous optimization of DR submodular functions, covering monotonic/non-monotonic, full information/semi-bandit/bandit feedback, vector/value queries, and lower closed or general convex constraints.
Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence
Sunghwan Hong (Korea University), Stephen Lin (Microsoft Research Asia)
TransformerImage
🎯 What it does: A unified feature and cost aggregation network (UFC) based on Transformer is proposed for dense visual and semantic matching tasks.
UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science
Yazheng Yang (University of Hong Kong), Qi Liu (Beijing Academy of Artificial Intelligence)
ClassificationOptimizationRecurrent Neural NetworkTransformerContrastive LearningTabularFinance Related
🎯 What it does: This paper studies a universal table pre-training framework called UniTabE, which utilizes fine-grained cell processing, a lightweight shallow decoder, and free prompts to achieve pre-training and downstream fine-tuning of table data.
Universal Backdoor Attacks
Benjamin Schneider (University of Waterloo), Florian Kerschbaum (University of Waterloo)
ClassificationAdversarial AttackConvolutional Neural NetworkImage
🎯 What it does: This paper proposes a 'Universal Backdoor Attack' that can misclassify any input into any target category during inference by contaminating only a very small proportion of samples (0.15%) in the training set.
Universal Guidance for Diffusion Models
Arpit Bansal (University of Maryland), Tom Goldstein (University of Maryland)
Object DetectionSegmentationGenerationDiffusion modelImage
🎯 What it does: A universal guiding algorithm is proposed, allowing diffusion models to be controlled by any guiding modality without the need to retrain any specific components.
Universal Humanoid Motion Representations for Physics-Based Control
Zhengyi Luo (Reality Labs Research), Weipeng Xu (Reality Labs Research)
Robotic IntelligenceReinforcement LearningVideoPhysics Related
🎯 What it does: A universal humanoid robot motion representation (PULSE) is proposed, capable of reproducing and generating a wide range of human actions under physical-based control;
Universal Jailbreak Backdoors from Poisoned Human Feedback
Javier Rando (ETH Zurich), Florian Tramèr (ETH Zurich)
Adversarial AttackReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: This paper studies the implantation of a universal 'jailbreak' backdoor by poisoning training data during the Reinforcement Learning from Human Feedback (RLHF) training process, allowing attackers to use hidden trigger words (such as SUDO) in any prompt to induce the language model to generate harmful content.
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition
Wenxuan Zhou (University of Southern California), Hoifung Poon (Microsoft Research)
RecognitionKnowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: In this paper, the authors distilled knowledge of open-domain named entity recognition (NER) from ChatGPT into a smaller, more cost-effective UniversalNER model using a mission-focused instruction tuning approach, achieving targeted distillation of large language models for open NER tasks.
Unknown Domain Inconsistency Minimization for Domain Generalization
Seungjae Shin (Korea Advanced Institute of Science and Technology), Il-chul Moon
Domain AdaptationOptimizationImage
🎯 What it does: This paper proposes the Unknown Domain Inconsistency Minimization (UDIM) method, which optimizes the consistency of the loss landscape between the source domain and the simulated unknown domain by perturbing both the parameter space and the data space, thereby enhancing domain generalization performance.
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
Hongtao Wu (ByteDance Research), Tao Kong (ByteDance Research)
GenerationRobotic IntelligenceTransformerVision Language ModelVideo
🎯 What it does: This paper proposes the GR-1 model, which utilizes a large-scale video generation pre-trained GPT-style transformer to achieve end-to-end language-conditioned visual robot manipulation.
Unleashing the Potential of Fractional Calculus in Graph Neural Networks with FROND
Qiyu Kang (Nanyang Technological University), Wee Peng Tay (Nanyang Technological University)
ClassificationGraph Neural NetworkGraph
🎯 What it does: Proposes the FROND framework, which implements long-term memory characteristics in graph neural networks using Caputo fractional derivatives, improving node feature updates.
Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning
Ruizhe Shi (Tsinghua University), Huazhe Xu (Shanghai AI Lab)
TransformerLarge Language ModelReinforcement LearningSequential
🎯 What it does: In offline reinforcement learning, the decision transformer is initialized using a pre-trained language model (GPT-2) and its decision-making performance on sparse and dense reward tasks is enhanced through LoRA fine-tuning, MLP embedding, and language prediction auxiliary loss.
Unlocking the Power of Representations in Long-term Novelty-based Exploration
Alaa Saade (Google Deepmind), Bilal Piot (Google Deepmind)
Representation LearningTransformerReinforcement LearningVideo
🎯 What it does: This paper proposes a new non-parametric exploration method called RECODE, which utilizes clustering adaptive online density estimation to achieve long-term counting rewards, while introducing a CASM transformer-style inverse dynamics representation to enhance the quality of state embeddings.
Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models
Zhaowei Zhu (Docta), Yang Liu (University of California)
ClassificationData-Centric LearningTransformerLarge Language ModelText
🎯 What it does: Evaluate and clean the label errors in publicly available datasets for training harmless language models, enhancing data credibility and downstream classification performance.
Unpaired Image-to-Image Translation via Neural Schrödinger Bridge
Beomsu Kim (KAIST), Jong Chul Ye (KAIST)
Image TranslationGenerationDomain AdaptationGenerative Adversarial NetworkImageStochastic Differential Equation
🎯 What it does: This paper proposes an unpaired image translation framework called UNSB based on the Schrödinger Bridge (SB), capable of performing multi-step unsupervised domain transfer at high resolution.
Unprocessing Seven Years of Algorithmic Fairness
André Cruz, Moritz Hardt (Max Planck Institute for Intelligent Systems)
OptimizationTabularBenchmark
🎯 What it does: This paper evaluates and compares all major algorithms regarding error rate fairness over the past seven years, concluding that post-processing only on the optimal predictor can achieve the accuracy-fairness Pareto frontier attainable by all methods.
UNR-Explainer: Counterfactual Explanations for Unsupervised Node Representation Learning Models
Hyunju Kang (Sungkyunkwan University), Hogun Park (Sungkyunkwan University)
Explainability and InterpretabilityRepresentation LearningGraph Neural NetworkGraph
🎯 What it does: This paper proposes a counterfactual explanation method for unsupervised node representation learning models—UNR-Explainer, which can find subgraphs that significantly change the k-nearest neighbor set of the target node in the embedding space by minimizing perturbation edges.
Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space
Yufei Gu (Fudan University), Tomaso Aste (University College London)
ClassificationRepresentation LearningConvolutional Neural NetworkImage
🎯 What it does: This paper conducts experiments on the learning process of deep networks with different widths on a noisy dataset, studying how noisy data affects the double descent phenomenon, and uses k-NN to evaluate the 'isolation' degree of noisy samples in the learned feature space.
Unraveling the Key Components of OOD Generalization via Diversification
Harold Luc Benoit, Amir Zamir (Swiss Federal Institute of Technology)
Domain AdaptationConvolutional Neural NetworkImage
🎯 What it does: This paper systematically studies the key factors of diversification methods in out-of-distribution (OOD) generalization, evaluating the impact of unlabeled data distribution, learning algorithms, and diversification loss on performance.
Unsupervised Order Learning
Seon-Ho Lee (Korea University), Chang-Su Kim (Korea University)
ClassificationRecognitionRepresentation LearningConvolutional Neural NetworkImageSequential
🎯 What it does: An unsupervised sequence learning method (UOL) is proposed, capable of achieving ordered clustering and ranking estimation on unlabeled data.
Unsupervised Pretraining for Fact Verification by Language Model Distillation
Adrián Bazaga (University of Cambridge), Gos Micklem (University of Cambridge)
Knowledge DistillationRepresentation LearningGraph Neural NetworkTransformerLarge Language ModelContrastive LearningTextGraph
🎯 What it does: An unsupervised pre-training framework SFAVEL is proposed, which achieves high-quality alignment features between text claims and knowledge graph facts by distilling the semantic representations of pre-trained language models into a knowledge graph attention network.
Unveiling and Manipulating Prompt Influence in Large Language Models
Zijian Feng (Nanyang Technological University), Kezhi Mao (Nanyang Technological University)
GenerationExplainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: A method based on Token Distribution Dynamics (TDD) is proposed to explain and manipulate the influence of prompts on the generation results of large language models.
Unveiling Options with Neural Network Decomposition
Mahdi Alikhasi (University of Alberta), Levi Lelis
Reinforcement Learning
🎯 What it does: An algorithm is proposed to decompose a trained neural network policy into sub-policies and package them as options to achieve knowledge transfer across tasks.
Unveiling the Pitfalls of Knowledge Editing for Large Language Models
Zhoubo Li (Zhejiang University), Huajun Chen (Zhejiang University)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper systematically studies the potential side effects that may occur during knowledge editing in large language models, proposing and quantifying two main issues: knowledge conflict (contradictory edits) and knowledge distortion (edits leading to irreversible distortion of knowledge structure). It experimentally verifies the shortcomings of existing editing methods in these two aspects and proposes a multi-label editing (MLE) strategy to mitigate knowledge distortion.
Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels
Zahra Babaiee (Austria and Massachusetts Institute of Technology), Radu Grosu (Austria)
Explainability and InterpretabilityRepresentation LearningConvolutional Neural NetworkAuto EncoderImage
🎯 What it does: This paper conducts large-scale visualization and clustering of millions of convolution kernels from various depthwise separable convolution networks, revealing that the weights after training exhibit interpretable differential Gaussian (DoG) patterns and their first and second derivative patterns across all layers.
USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields
Moyang Li (ETH Zurich), Peidong Liu (Westlake University)
RestorationPose EstimationOptimizationNeural Radiance FieldImageVideo
🎯 What it does: Utilizing neural radiance fields (NeRF) to jointly optimize the continuous time motion trajectory of rolling shutter cameras, thereby learning a distortion-free 3D scene representation from rolling shutter images and recovering the camera trajectory;
V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection
Yichao Shen (Xi'an Jiaotong University), Baining Guo (Microsoft Research Asia)
Object DetectionTransformerPoint Cloud
🎯 What it does: This paper proposes a 3D object detection framework based on DETR called V-DETR, and introduces 3D Vertex Relative Position Encoding (3DV-RPE) to enhance detection performance.
ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation
Kim-Celine Kahl (German Cancer Research Center), Paul F Jaeger
SegmentationDomain AdaptationAnomaly DetectionConvolutional Neural NetworkImageBiomedical DataComputed Tomography
🎯 What it does: Proposes the VALUES framework to systematically validate the effectiveness and practicality of uncertainty estimation in semantic segmentation.
Vanishing Gradients in Reinforcement Finetuning of Language Models
Noam Razin (Apple), Etai Littwin (Apple)
OptimizationTransformerSupervised Fine-TuningReinforcement LearningText
🎯 What it does: The study investigates the gradient vanishing problem during the reinforcement learning fine-tuning (RFT) process of language models and proposes alleviation through a small amount of supervised fine-tuning (SFT).
Variance Reduced Halpern Iteration for Finite-Sum Monotone Inclusions
Xufeng Cai (University of Wisconsin Madison), Jelena Diakonikolas (University of Wisconsin Madison)
Optimization
🎯 What it does: This paper proposes two variance-reduced algorithms based on Halpern iteration to solve finite-sum monotone inclusion problems, covering applications such as variational inequalities (VI) and convex-concave minimization.
Variance-aware Regret Bounds for Stochastic Contextual Dueling Bandits
Qiwei Di (University of California), Quanquan Gu (University of California)
Reinforcement LearningTabular
🎯 What it does: This paper addresses the contextual two-armed bandit problem and proposes a variance-aware algorithm VACDB based on generalized linear models, providing its variance-aware cumulative regret upper bound.
Variance-enlarged Poisson Learning for Graph-based Semi-Supervised Learning with Extremely Sparse Labeled Data
Xiong Zhou (Harbin Institute of Technology), Xiangyang Ji (Tsinghua University)
ClassificationOptimizationGraph Neural NetworkGraph
🎯 What it does: This paper proposes a Variational Poisson Learning (VPL) framework for graph-structured semi-supervised learning, particularly suitable for extremely sparse label scenarios, and presents efficient implementations of the V-Laplace and V-Poisson algorithms, as well as the V-GPN model applicable to graph neural networks.
Variational Bayesian Last Layers
James Harrison (Google DeepMind), Jasper Snoek (Vector Institute)
ClassificationOptimizationImageTextTabular
🎯 What it does: A variational Bayesian last layer (VBLL) training framework is proposed, achieving Bayesian networks with no sampling and single forward inference.
Variational Inference for SDEs Driven by Fractional Noise
Rembert Daems (Ghent University), Tolga Birdal (Imperial College London)
GenerationData SynthesisOptimizationVideoStochastic Differential Equation
🎯 What it does: A variational inference framework for stochastic differential equations (SDEs) driven by fractal Brownian motion with Markov approximation is proposed, and it is applied to video prediction using neural SDEs.
VBH-GNN: Variational Bayesian Heterogeneous Graph Neural Networks for Cross-subject Emotion Recognition
Chenyu Liu (Nanyang Technological University), Yang Liu (Nanyang Technological University)
RecognitionDomain AdaptationGraph Neural NetworkMultimodalityBiomedical Data
🎯 What it does: This study focuses on cross-subject emotion recognition, utilizing multimodal physiological signals and achieving relationship distribution adaptation through a variational Bayesian heterogeneous graph neural network.
VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections
Dongqi Fu (University of Illinois Urbana-Champaign), Bo Long (Meta AI)
Graph Neural NetworkTransformerGraph
🎯 What it does: This paper proposes a graph Transformer that can be used under small-batch training conditions—VCR-Graphormer. It generates a personalized PageRank (PPR) sampling token list for each node and learns node representations through self-attention in the Transformer.
VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models
Zihao Zhu (Chinese University of Hong Kong), Baoyuan Wu (Chinese University of Hong Kong)
Anomaly DetectionTransformerLarge Language ModelVision Language ModelImageMultimodality
🎯 What it does: This paper proposes a general dirty sample detection framework called VDC, which utilizes a multimodal large language model (MLLM) to assess the semantic inconsistency between images and labels, thereby identifying backdoor attack samples, noisy labels, and mixed dirty samples.
VDT: General-purpose Video Diffusion Transformers via Mask Modeling
Haoyu Lu (Renmin University of China), Mingyu Ding (University of California)
GenerationData SynthesisTransformerDiffusion modelVideo
🎯 What it does: This paper proposes the Video Diffusion Transformer (VDT), a novel video diffusion model based on Transformer that can unify various tasks such as unconditional generation, video prediction, interpolation, animation, and completion.
VeRA: Vector-based Random Matrix Adaptation
Dawid Jan Kopiczko, Yuki M Asano
TransformerSupervised Fine-TuningImageText
🎯 What it does: We propose VeRA, a parameter-efficient fine-tuning method that utilizes shared random low-rank matrices and trainable scaling vectors, significantly reducing the number of trainable parameters.
VersVideo: Leveraging Enhanced Temporal Diffusion Models for Versatile Video Generation
Jinxi Xiang (Tencent AI Lab), Yang Wei (Sun Yat-sen University)
GenerationData SynthesisMixture of ExpertsDiffusion modelVideoTextMultimodality
🎯 What it does: This work proposes a multimodal control video diffusion model, VersVideo, which can generate high-quality, temporally and spatially consistent videos based on various conditions such as text, images, depth, edges, and styles.
VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks
Zhaomin Wu (National University of Singapore), Bingsheng He (National University of Singapore)
Federated LearningImageTabularBenchmark
🎯 What it does: Constructed VertiBench, a comprehensive benchmark for vertical federated learning, providing synthetic data generation methods based on feature importance and correlation, real image-to-image VFL datasets, and systematic evaluations of various mainstream VFL algorithms.
VFLAIR: A Research Library and Benchmark for Vertical Federated Learning
Tianyuan Zou (Institute for AI Industry Research Tsinghua University), Ya-Qin Zhang (Institute for AI Industry Research Tsinghua University)
Federated LearningSafty and PrivacyComputational EfficiencyAdversarial AttackImageTextTabularBenchmark
🎯 What it does: This paper proposes and implements a lightweight, scalable vertical federated learning framework called VFLAIR, and conducts a unified evaluation of 11 types of attacks and 8 types of defense methods under this framework, providing benchmarks for communication/computation efficiency, model performance, and defense capabilities.