ICLR 2024 Papers — Page 11
International Conference on Learning Representations · 2260 papers
INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
Chao Chen (Alibaba Cloud), Jieping Ye (Alibaba Cloud)
GenerationTransformerLarge Language ModelText
🎯 What it does: This work proposes the INSIDE framework, which utilizes the internal states of LLMs (intermediate layer hidden vectors) to detect knowledge hallucinations; it measures semantic consistency by calculating the eigenvalues of the covariance of the embedding space of multiple generated answer sentences and introduces Feature Clipping to suppress overconfident hallucinations.
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
Xingchao Liu (University of Texas), qiang liu
GenerationKnowledge DistillationDiffusion modelRectified FlowImageText
🎯 What it does: In this paper, the authors adapt Stable Diffusion into a one-click text-to-image generation model, proposing a new text-conditioned Rectified Flow pipeline, and achieve single-step high-quality image generation through Reflow and knowledge distillation.
Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model
Jiahao Li (Adobe Research), Sai Bi (Adobe Research)
GenerationData SynthesisTransformerDiffusion modelNeural Radiance FieldMesh
🎯 What it does: Through a two-stage process, four consistent views are generated at once using a fine-tuned 2D text-to-image diffusion model, and then a Transformer-based sparse view reconstruction model quickly synthesizes NeRF 3D assets.
InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists
Yulu Gan (Peking University), Ahmed Alaa
Object DetectionSegmentationGenerationDepth EstimationTransformerLarge Language ModelSupervised Fine-TuningDiffusion modelImageMultimodality
🎯 What it does: By mapping text instructions to image generation tasks, a unified visual task framework based on natural language was achieved using the stable diffusion model;
InstructDET: Diversifying Referring Object Detection with Generalized Instructions
Ronghao Dang (Tongji University), Yibing Song (Alibaba DAMO Academy)
Object DetectionTransformerLarge Language ModelPrompt EngineeringVision Language ModelImageTextMultimodality
🎯 What it does: This paper proposes the InstructDET method, which automatically generates diverse detection instructions based on large visual-language models, thereby elevating reference object detection (ROD) to a level more aligned with actual user needs.
Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions
Taehyeon Kim (Korea Advanced Institute of Science and Technology), Se-Young Yun (Korea Advanced Institute of Science and Technology)
GenerationOptimizationTransformerLarge Language ModelSupervised Fine-TuningContrastive LearningText
🎯 What it does: Instructive Decoding (ID) is proposed, which compares the log probabilities of the original instruction and its noisy variants, and adjusts the logits of the next word based on this comparison to enhance the adherence of instruction-tuned language models during generation.
InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image
Jianhui Li (Tsinghua University), Jun Zhu (Tsinghua University)
Image TranslationGenerationTransformerLarge Language ModelDiffusion modelNeural Radiance FieldImageText
🎯 What it does: An end-to-end Diffusion+NeRF framework called InstructPix2NeRF is proposed, which can perform 3D-consistent portrait editing based on a single image and natural language instructions.
InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior
Chenguo Lin (Peking University), Yadong MU
GenerationData SynthesisTransformerLarge Language ModelDiffusion modelPoint CloudGraph
🎯 What it does: A two-stage 3D indoor scene generation framework called INSTRUCTSCENE is proposed, based on semantic graph priors and a layout decoder.
Integrating Planning and Deep Reinforcement Learning via Automatic Induction of Task Substructures
Jung-Chun Liu (National Taiwan University), Tian-Li Yu (National Taiwan University)
Reinforcement LearningSequential
🎯 What it does: A framework that combines deep reinforcement learning with classical planning is proposed, which uses genetic programming to automatically induce task substructures from a small number of demonstrations, and provides internal rewards through a key action network to enhance the learning efficiency of sparse reward tasks.
Intelligent Switching for Reset-Free RL
Darshan Patil (Mila), Sarath Chandar (Mila)
Robotic IntelligenceReinforcement Learning
🎯 What it does: A reset-free reinforcement learning algorithm named RISC is proposed, which can intelligently switch between the forward task controller and the rollback controller without frequently resetting the environment, thereby collecting experiences more efficiently and improving learning speed.
Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning
Yun-Hin Chan (University of Hong Kong), Edith C. H. Ngai
Federated LearningConvolutional Neural NetworkTransformerImage
🎯 What it does: A method called InCo Aggregation is proposed, which utilizes internal cross-layer gradients in federated learning to enhance the similarity of deep client models, significantly improving model performance in heterogeneous device environments.
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Yi Wang (OpenGVLab, Shanghai AI Laboratory), Yu Qiao (OpenGVLab, Shanghai AI Laboratory)
RecognitionGenerationRetrievalTransformerLarge Language ModelVision Language ModelDiffusion modelContrastive LearningVideoTextMultimodality
🎯 What it does: A large-scale multimodal dataset, InternVid, was constructed, which has a high correspondence between video and text. The ViCLIP vision-language model was trained on this dataset and further applied to tasks such as action recognition, video retrieval, text-to-video generation, and video dialogue.
InterpGNN: Understand and Improve Generalization Ability of Transdutive GNNs through the Lens of Interplay between Train and Test Nodes
Jiawei Sun (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
ClassificationGraph Neural NetworkGraph
🎯 What it does: In the semi-supervised transductive node classification task, we first derive the PAC-Bayesian upper bound of L-hop interaction and GNN generalization error; then we propose the Graph Global Workspace (InterpGNN-GW) module, which uses key-value attention to embed important nodes into a global memory and broadcasts them to all nodes during training and inference, thereby enhancing interaction among remote nodes.
Interpretable Diffusion via Information Decomposition
Xianghao Kong (University of California Riverside), Greg Ver Steeg (University of California Riverside)
Image TranslationRestorationExplainability and InterpretabilityDiffusion modelImageOrdinary Differential Equation
🎯 What it does: This paper views the denoising diffusion model as a Gaussian noise channel, utilizing the error of the MMSE denoiser to directly compute mutual information and conditional mutual information, thereby achieving point-level and pixel-level interpretable information decomposition, which in turn evaluates the model's compositional understanding, the influence of words on pixels, and the response to image editing.
Interpretable Meta-Learning of Physical Systems
Matthieu Blanke (Inria Paris), Marc Lelarge (Inria Paris)
Explainability and InterpretabilityComputational EfficiencyRobotic IntelligenceMeta LearningTime SeriesSequentialPhysics Related
🎯 What it does: This paper proposes an interpretable meta-learning framework suitable for learning physical systems in multiple environments (CAMEL), which achieves rapid adaptation and physical parameter identification through task-related low-dimensional linear context parameters.
Interpretable Sparse System Identification: Beyond Recent Deep Learning Techniques on Time-Series Prediction
Xiaoyi Liu (Southeast University), Wenwu Yu (Southeast University)
OptimizationExplainability and InterpretabilityComputational EfficiencyTime Series
🎯 What it does: A gradient-free learning model GLIP based on Fourier basis functions and sparse recognition is proposed for long-term time series prediction on CPU.
Interpreting CLIP's Image Representation via Text-Based Decomposition
Yossi Gandelsman (University of California Berkeley), Jacob Steinhardt (University of California Berkeley)
SegmentationExplainability and InterpretabilityTransformerContrastive LearningImage
🎯 What it does: Analyze the internal structure of the CLIP ViT image encoder, breaking it down into layers, attention heads, and image positions, and use the text space to explain the semantics of each component;
Interpreting Robustness Proofs of Deep Neural Networks
Debangshu Banerjee (University of Illinois Urbana-Champaign), Gagandeep Singh (VMware Research)
Explainability and InterpretabilityConvolutional Neural NetworkImage
🎯 What it does: This paper proposes the ProFIt method for analyzing the robustness proofs of deep neural networks, helping humans understand the content of the proofs.
Interventional Fairness on Partially Known Causal Graphs: A Constrained Optimization Approach
Aoqi Zuo (University of Melbourne), Mingming Gong (University of Melbourne)
OptimizationTabularFinance Related
🎯 What it does: A constraint optimization framework is proposed to achieve interventional fairness on partially known causal graphs (MPDAG).
Intriguing Properties of Data Attribution on Diffusion Models
Xiaosen Zheng (Singapore Management University), Min Lin (Sea AI Lab)
GenerationData-Centric LearningDiffusion modelImage
🎯 What it does: This paper studies how to perform data attribution on diffusion models (DDPM and Stable Diffusion), proposing an improved version of the TRAK method called D-TRAK, and conducts large-scale experiments on various datasets.
Intriguing Properties of Generative Classifiers
Priyank Jaini (Google DeepMind), Robert Geirhos (Google DeepMind)
ClassificationGenerationDiffusion modelImageText
🎯 What it does: The study evaluates the performance of text-to-image diffusion models and autoregressive models as zero-shot classifiers on 17 challenging OOD datasets.
Invariance-based Learning of Latent Dynamics
Kai Lagemann (German Center for Neurodegenerative Diseases), Sach Mukherjee (University of Cambridge)
TransformerAuto EncoderTime SeriesPhysics RelatedOrdinary Differential Equation
🎯 What it does: The LaDID framework is proposed, which learns the latent dynamics of high-dimensional observations through variational autoencoders and Transformers, supporting continuous time prediction and cross-system generalization.
Inverse Approximation Theory for Nonlinear Recurrent Neural Networks
Shida Wang (National University of Singapore), Qianxiao Li (National University of Singapore)
Recurrent Neural NetworkTextSequential
🎯 What it does: This paper proves that for nonlinear recurrent neural networks (RNNs) to stably approximate sequence-to-sequence relationships, the objective function must possess an exponentially decaying memory structure;
Investigating the Benefits of Projection Head for Representation Learning
Yihao Xue (University of California), Baharan Mirzasoleiman (University of California)
Representation LearningContrastive LearningImage
🎯 What it does: This study investigates the impact of adding a projection head on representation learning in self-supervised contrastive learning, supervised contrastive learning, and supervised learning, providing a theoretical explanation.
INViTE: INterpret and Control Vision-Language Models with Text Explanations
Haozhe Chen (Columbia University), Chengzhi Mao
Explainability and InterpretabilityAdversarial AttackTransformerVision Language ModelImage
🎯 What it does: This paper proposes the INViTE framework, which disables self-attention in the Transformer and retains only local operations, mapping each potential token to the CLS layer, and then provides natural language explanations using CLIP's text retrieval.
IRAD: Implicit Representation-driven Image Resampling against Adversarial Attacks
Yue Cao (Agency for Science Technology and Research), Qing Guo (Agency for Science Technology and Research)
Adversarial AttackConvolutional Neural NetworkDiffusion modelImage
🎯 What it does: This paper proposes an image resampling method based on implicit continuous representation (IRAD), which utilizes SampleNet to automatically predict pixel-level offsets to counter adversarial attacks during the inference phase.
Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability
Ivan Lee (University of California), Taylor Berg-Kirkpatrick (University of California)
Convolutional Neural NetworkRecurrent Neural NetworkTransformerLarge Language ModelPrompt EngineeringImageText
🎯 What it does: This paper evaluates 13 different architectures of causal language models through training on a series of synthetic in-context learning tasks (associative recall, linear regression, multi-class classification, image classification, language modeling) to explore the relationship between architecture and ICL capabilities.
Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video
Shashanka Venkataramanan (Inria), Yannis Avrithis (Institute of Advanced Research on Artificial Intelligence)
ClassificationObject DetectionObject TrackingSegmentationKnowledge DistillationRepresentation LearningTransformerContrastive LearningVideo
🎯 What it does: A powerful image encoder with strong generalization capabilities is obtained through self-supervised learning from a long, unlabelled first-person video.
Is Self-Repair a Silver Bullet for Code Generation?
Theo X. Olausson (Massachusetts Institute of Technology), Armando Solar-Lezama (Massachusetts Institute of Technology)
GenerationAI Code AssistantTransformerLarge Language ModelText
🎯 What it does: This paper investigates the feasibility of improving performance in Python code generation tasks through self-repair by large language models, evaluating the self-debugging capabilities of CodeLlama-13b, GPT-3.5, and GPT-4.
Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching
Aleksandar Makelov (SERI MATS), Neel Nanda
Explainability and InterpretabilityTransformerLarge Language ModelText
🎯 What it does: This study investigates the phenomenon of interpretative hallucinations that may occur during the process of subspace activation patching, and validates this phenomenon through mathematical examples, indirect object identification tasks (IOI), and factual recall experiments. It also demonstrates that under reasonably designed subspaces, patches can reliably locate features and provide explanations.
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition
CHEN CHEN, Chao-Han Huck Yang (NVIDIA Research)
RecognitionTransformerLarge Language ModelMultimodalityAudio
🎯 What it does: In automatic speech recognition, a generative error correction (GER) framework based on large language models is proposed, which significantly reduces the word error rate by combining acoustic information with LLM output through post-hoc dynamic fusion (UADF).
Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting
Aleksei Ustimenko (ShareChat), Aleksandr Beznosikov (Innopolis University)
OptimizationStochastic Differential Equation
🎯 What it does: A general Ito chain is studied, and an upper bound on the Wasserstein-2 distance error with respect to the corresponding SDE is provided.
iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
Yong Liu (Tsinghua University), Mingsheng Long (Tsinghua University)
TransformerTime Series
🎯 What it does: A method called 'iTransformer' is proposed, which captures multivariate correlations using attention in the Transformer structure by reversing the dimensions of time series data (treating the entire sequence of each variable as a token) and learning sequence representations with a feedforward network, while keeping the original Transformer components unchanged.
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models
Erfan Shayegani (University of California, Riverside), Nael Abu-Ghazaleh (University of California, Riverside)
Adversarial AttackTransformerVision Language ModelImageTextMultimodality
🎯 What it does: Developed a cross-modal adversarial attack based on embedding space, utilizing a visual encoder to generate seemingly normal images combined with ordinary text prompts, successfully breaching the security barrier of multimodal language models.
Jointly Training Large Autoregressive Multimodal Models
Emanuele Aiello (Politecnico di Torino), Barlas Oguz (Meta AI)
GenerationRetrievalTransformerLarge Language ModelVision Language ModelImageTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: The JAM framework is proposed, which integrates pre-trained text generation LLMs and text-image autoregressive models into a unified multimodal autoregressive model, achieving interwoven generation of text and images through instruction tuning.
Jointly-Learned Exit and Inference for a Dynamic Neural Network
florence regol, Mark Coates (McGill University)
ClassificationOptimizationComputational EfficiencyConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: A dynamic neural network that jointly learns early exit gating and intermediate reasoning modules (JEI-DNN) is proposed to address the training-testing mismatch problem of traditional methods.
JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling
Jingyang Zhang (Apple), Yao Yao (Apple)
GenerationData SynthesisDepth EstimationDiffusion modelImageTextMultimodality
🎯 What it does: This paper proposes JointNet, an architecture that extends a pre-trained text-to-image diffusion model for jointly modeling the distribution of RGB images and dense labels (such as depth and normals).
JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention
Yuandong Tian (Meta), Simon Shaolei Du
TransformerLarge Language ModelText
🎯 What it does: The JoMA framework is proposed, which combines the self-attention of Transformers with the training dynamics of MLP layers, deriving the implicit dynamics of self-attention and explaining the mechanism by which multi-layer Transformers learn hierarchical structures.
Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAX
Clément Bonnet (InstaDeep), Alexandre Laterre (InstaDeep)
OptimizationConvolutional Neural NetworkTransformerReinforcement LearningTabularBenchmark
🎯 What it does: We propose Jumanji, a suite of 22 JAX-based reinforcement learning environments covering routing, packing, and logic problems, and providing customizable initial state generators and baseline A2C training code.
Kalman Filter for Online Classification of Non-Stationary Data
Michalis Titsias, Jorg Bornschein
ClassificationOptimizationImageTime SeriesStochastic Differential Equation
🎯 What it does: This paper proposes an online learning framework based on Kalman filtering to adaptively handle parameter drift in non-stationary data streams, and on this basis, implements online fine-tuning of classification tasks and deep features.
Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies
Haanvid Lee (KAIST), Kee-Eung Kim (KAIST)
Reinforcement Learning
🎯 What it does: This study proposes a framework for offline evaluation of deterministic policies in continuous action spaces, achieving in-sample TD learning without out-of-domain samples by applying Gaussian kernel relaxation to the deterministic target policy in importance resampling and learning adjustable bandwidth and Mahalanobis distance metrics.
Kernelised Normalising Flows
Eshant English (Hasso Plattner Institute for Digital Engineering), Christoph Lippert (Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai)
GenerationData SynthesisComputational EfficiencyFlow-based ModelTabular
🎯 What it does: Kernelised Normalising Flows (Ferumal flows) are proposed, using kernel functions to replace the scale and translation functions in the affine coupling layer of traditional neural networks, maintaining reversibility and achieving more efficient probability distribution modeling.
Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning
Binwu Wang (University of Science and Technology of China), Yang Wang (University of Science and Technology of China)
ClassificationConvolutional Neural NetworkImage
🎯 What it does: A dynamic optional data augmentation (DODA) mechanism is proposed to adaptively address the imbalance between data and augmentation in long-tail learning.
KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval
Marah I Abdin (Microsoft Research), Besmira Nushi (Microsoft Research)
RetrievalTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: The KITAB dataset is proposed to evaluate the constraint satisfaction ability of large language models in information retrieval tasks.
Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models
Shangbin Feng (University of Washington), Yulia Tsvetkov (University of Washington)
TransformerLarge Language ModelSupervised Fine-TuningTextRetrieval-Augmented Generation
🎯 What it does: The KNOWLEDGE CARD framework is proposed, which utilizes small language models (knowledge cards) trained from different fields and sources, along with a three-layer knowledge selector, to dynamically inject external knowledge during inference, enhancing the factuality and timeliness of large models.
Knowledge Distillation Based on Transformed Teacher Matching
Kaixiang Zheng (University of Waterloo), EN-HUI YANG
ClassificationKnowledge DistillationConvolutional Neural NetworkImage
🎯 What it does: This paper proposes a temperature scaling method applied only to the teacher side for knowledge distillation—Transform Teacher Matching (TTM) and its sample adaptive version (WTTM), and proves its equivalence to traditional KD plus R'‑enyi entropy regularization, thereby enhancing the generalization ability of the student model.
Knowledge Fusion of Large Language Models
Fanqi Wan (Sun Yat-sen University), Shuming Shi (Tencent AI Lab)
Knowledge DistillationTransformerLarge Language ModelText
🎯 What it does: A knowledge fusion framework called FUSELLM is proposed, which utilizes the probability distributions of various structurally different LLMs for lightweight continuous training of the target LLM, thereby transferring the capabilities of each source model into a single model.
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Jifan Yu (Tsinghua University), Juanzi Li (Tsinghua University)
TransformerLarge Language ModelTextBenchmark
🎯 What it does: A multi-level evaluation benchmark KoLA aimed at global knowledge has been constructed to systematically assess the four cognitive abilities of LLM: knowledge memory, understanding, application, and creation.
Koopman-based generalization bound: New aspect for full-rank weights
Yuka Hashimoto (NTT), Taiji Suzuki (A*STAR CFAR)
Image
🎯 What it does: This paper constructs a new upper bound for Rademacher complexity using the Koopman operator to analyze the generalization performance of neural networks with full-rank weight matrices.
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
Xichen Pan (Microsoft Research), Furu Wei (Microsoft Research)
GenerationTransformerLarge Language ModelDiffusion modelScore-based ModelImageTextMultimodality
🎯 What it does: Proposes the KOSMOS-G model, achieving subject-driven image generation with zero-shot, multi-image, and text interaction;
KW-Design: Pushing the Limit of Protein Design via Knowledge Refinement
Zhangyang Gao (Zhejiang University), Stan Z. Li (Westlake University)
Protein Structure PredictionGraph Neural NetworkMultimodality
🎯 What it does: A protein sequence design method named KW-Design is proposed, which significantly improves sequence recovery rates by iteratively refining low-confidence residues using structural and sequence knowledge provided by a pre-trained model.
L2MAC: Large Language Model Automatic Computer for Extensive Code Generation
Samuel Holt (University of Cambridge), Mihaela van der Schaar (University of Cambridge)
GenerationAI Code AssistantTransformerLarge Language ModelTextChain-of-Thought
🎯 What it does: Designed and implemented L2MAC, a storage program computing framework based on large language models, for generating long texts and code, instantiated as Code-L2MAC, capable of gradually building a complete large-scale codebase without being constrained by traditional context window limits.
L2P-MIP: Learning to Presolve for Mixed Integer Programming
Chang Liu (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
OptimizationGraph Neural NetworkReinforcement LearningGraph
🎯 What it does: This paper proposes a learning-driven preprocessing method (Learning to Presolve, L2P) that predicts optimal preprocessing parameters (priority, maximum iterations, temporal mask) for each mixed-integer programming instance and integrates them with the solver to improve solving efficiency.
Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models
Shaofei Shen (University of Queensland), Miao Xu (University of Queensland)
Safty and PrivacyData-Centric LearningAuto EncoderContrastive LearningImage
🎯 What it does: A framework LAF is proposed to achieve unlearning of deep models without label supervision, which can maintain the knowledge of remaining data while deleting specified data and resisting privacy attacks.
Label-Focused Inductive Bias over Latent Object Features in Visual Classification
Ilmin Kang (Gwangju Institute of Science and Technology), Kangil Kim (Gwangju Institute of Science and Technology)
ClassificationTransformerImage
🎯 What it does: Proposes the Label-focused Latent-object Biasing (LLB) method, which quantizes the visual features from the intermediate layers of ViT into latent objects and disconnects visual dependencies to learn non-visual features based solely on labels, then fuses them with the original visual features to enhance the generalization ability of image classification models.
Label-free Node Classification on Graphs with Large Language Models (LLMs)
Zhikai Chen (Michigan State University), Jiliang Tang (Michigan State University)
ClassificationGraph Neural NetworkLarge Language ModelTextGraph
🎯 What it does: A complete process for unsupervised node classification is proposed, utilizing a large language model (LLM) to annotate a small number of nodes, and then using these annotations to train a graph neural network (GNN) to complete the node classification for the entire graph.
Label-Noise Robust Diffusion Models
Byeonghu Na (Korea Advanced Institute of Science and Technology), Il-chul Moon
GenerationData SynthesisDiffusion modelImage
🎯 What it does: This paper proposes a target function for training conditional diffusion models with noisy labels—Transition-aware Weighted Denoising Score Matching (TDSM), which eliminates the conditional bias caused by noisy labels through a weighted score network.
LabelDP-Pro: Learning with Label Differential Privacy via Projections
Badih Ghazi (Google Research), Chiyuan Zhang (Google Research)
ClassificationSafty and PrivacySupervised Fine-TuningImageTabular
🎯 What it does: This paper proposes LabelDP Pro, a training framework that significantly improves model performance under high privacy (low ε) in the context of label differential privacy (LabelDP) through interleaving projection denoising and DP SGD.
Lagrangian Flow Networks for Conservation Laws
Fabricio Arend Torres (University of Basel), Volker Roth (University of Basel)
Flow-based ModelTime SeriesPhysics Related
🎯 What it does: A Lagrangian flow network (LFlows) based on conditional normalizing flows is proposed, which models density and velocity fields in continuous time-space through a differentiable invertible mapping, naturally satisfying the continuity equation.
LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Driving
Tianyu Li (Fudan University), Hongyang Li (OpenDriveLab)
SegmentationAutonomous DrivingTransformerImage
🎯 What it does: An end-to-end LaneSegNet network is proposed, which for the first time uses lane segments to uniformly represent map features and achieve complete road structure perception in autonomous driving scenarios.
Langevin Monte Carlo for strongly log-concave distributions: Randomized midpoint revisited
Lu Yu (Ensae Paris), Arnak S. Dalalyan
OptimizationTabularStochastic Differential Equation
🎯 What it does: This paper studies random midpoint discretization (RLMC) under strongly log-convex target distributions and its variant in the dynamical Langevin process (RKLMC), providing non-asymptotic error upper bounds in Wasserstein-2 distance, and further deriving improved error upper bounds for KLMC (Euler discretization);
Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks
Edwin Zhang (Harvard University), Amy Zhang (University of Texas at Austin)
OptimizationComputational EfficiencyRobotic IntelligenceTransformerLarge Language ModelDiffusion modelSequential
🎯 What it does: A language-controlled diffusion (LCD) framework is proposed, using natural language as a condition for hierarchical planning, leveraging diffusion models for efficient long-term, spatial, and task-dimensional planning, addressing the limitations of traditional skill libraries.
Language Model Beats Diffusion - Tokenizer is key to visual generation
Lijun Yu (Google), Lu Jiang (Carnegie Mellon University)
RecognitionGenerationCompressionConvolutional Neural NetworkLarge Language ModelImageVideo
🎯 What it does: This paper presents the MAGVIT-v2 video tokenizer, which can discretely encode both images and videos using a unified vocabulary and enhances vocabulary capacity through lookup-free quantization.
Language Model Cascades: Token-Level Uncertainty And Beyond
Neha Gupta (Google Research), Sanjiv Kumar (Google Research)
ClassificationTransformerLarge Language ModelText
🎯 What it does: This paper systematically studies the inference abandonment rules in language model cascading (small-model + large-model), proposing a token-level uncertainty quantification based on quantile scoring and a post-hoc learning abandonment decision method, and explores the use of intermediate embeddings from large models to further enhance performance.
Language Model Decoding as Direct Metrics Optimization
Haozhe Ji (Tsinghua University), Minlie Huang (Tsinghua University)
GenerationOptimizationTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: A framework that treats language model decoding as direct metric optimization (DAEMON) is designed. By minimizing reverse KL and aligning the generated text with human text expectations across multiple evaluation metrics, an analytical solution for the energy-based model (EBM) is obtained, allowing for feasible sampling from this distribution.
Language Model Detectors Are Easily Optimized Against
Charlotte Nicks (Stanford University), Stefano Ermon (Stanford University)
OptimizationAdversarial AttackReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
🎯 What it does: Fine-tuning large language models using Direct Preference Optimization (DPO) in reinforcement learning to generate text that can confuse existing text detectors.
Language Model Inversion
John Xavier Morris, Alexander M Rush
GenerationAdversarial AttackTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes and implements an attack method based on the inverse of the downstream probability distribution of language models, specifically recovering hidden prompt words from the model's next-word probability vector. It further investigates how to obtain the complete probability distribution through logit bias and binary search in different API access scenarios.
Language Model Self-improvement by Reinforcement Learning Contemplation
Jing-Cheng Pang (Nanjing University), Yang Yu (Nanjing University)
TransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: An unsupervised language model self-improvement method called RLC is proposed, utilizing the model's own evaluation capability as a reward for reinforcement learning updates.
Language Modeling Is Compression
Gregoire Deletang, Joel Veness (Google DeepMind)
CompressionTransformerLarge Language ModelImageTextAudio
🎯 What it does: This paper views language models as lossless compressors and systematically evaluates the compression performance of large pre-trained models across three different modalities: text, images, and audio. It also explores the impact of the compression perspective on model scaling laws, tokenization strategies, and context learning.
Language Models Represent Space and Time
Wes Gurnee (Massachusetts Institute of Technology), Max Tegmark (Massachusetts Institute of Technology)
TransformerLarge Language ModelText
🎯 What it does: The study investigates whether large language models (Llama-2 series) encode spatial and temporal coordinates in their internal linear encoding space, and extracts this information from model activations using linear probes.
Language-Informed Visual Concept Learning
Sharon Lee (Stanford University), Jiajun Wu (Stanford University)
Data SynthesisKnowledge DistillationRepresentation LearningTransformerVision Language ModelDiffusion modelImage
🎯 What it does: A framework is proposed to extract and reorganize image visual concepts along language-specified conceptual axes by distilling from pre-trained vision-language models (text-image generation models and visual question answering models).
Language-Interfaced Tabular Oversampling via Progressive Imputation and Self-Authentication
June Yong Yang (Korea Advanced Institute of Science and Technology), Eunho Yang (Korea Advanced Institute of Science and Technology)
GenerationData SynthesisTransformerLarge Language ModelPrompt EngineeringTabular
🎯 What it does: This paper proposes a table data oversampling framework called LITO based on a pre-trained generative language model, which automatically generates minority class samples using techniques such as boundary sampling, importance-aware filling, and self-discrimination.
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Bin Zhu (Peking University), Li Yuan (Peking University)
ClassificationRetrievalTransformerContrastive LearningVideoTextMultimodalityAudio
🎯 What it does: The LanguageBind framework is proposed, which binds language as different modalities to achieve multimodal semantic alignment, and constructs a multimodal dataset VIDAL-10M.
Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI
Weibang Jiang, Bao-liang Lu
Anomaly DetectionRepresentation LearningTransformerTime SeriesBiomedical Data
🎯 What it does: A large brain model called LaBraM is proposed, which utilizes 2500 hours of multi-task EEG for unsupervised pre-training and fine-tuning on multi-task downstream tasks.
Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior
Ashmit Khandelwal (Adobe), Balaji Krishnamurthy
Recommendation SystemOptimizationTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelVideoTextMultimodality
🎯 What it does: A large-scale multimodal content-behavior model (LCBM) has been constructed to predict and optimize recipient behavior on multimedia content.
Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning
Murong Yue (George Mason University), Ziyu Yao (George Mason University)
Computational EfficiencyTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: A reasoning pipeline based on LLM cascading has been designed and implemented, utilizing the answer consistency of weaker LLMs to determine whether to invoke more powerful LLMs, thereby reducing costs.
Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
Yuchen Hu (Nanyang Technological University), EngSiong Chng
RecognitionGenerationKnowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningAudio
🎯 What it does: This paper proposes RobustGER, which utilizes LLM for generative error correction of ASR N-best and constructs a noise-robust version of the RobustHP dataset.
Large Language Models Are Not Robust Multiple Choice Selectors
Chujie Zheng (Tsinghua University), Minlie Huang (Tsinghua University)
TransformerLarge Language ModelText
🎯 What it does: This paper studies the option position bias of large language models in multiple-choice question (MCQ) assessments, finding that models tend to favor specific option IDs (such as A), and proposes a label-free, inference-usable method called PriDe to eliminate this bias.
Large Language Models as Analogical Reasoners
Michihiro Yasunaga (Stanford University), Denny Zhou (Google DeepMind)
TransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: A new prompting method called 'analogy prompting' is proposed, allowing large language models to automatically generate examples and knowledge related to the current question, thereby guiding their reasoning process.
Large Language Models as Automated Aligners for benchmarking Vision-Language Models
Yuanfeng Ji (University of Hong Kong), Ping Luo (University of Hong Kong)
Large Language ModelSupervised Fine-TuningPrompt EngineeringVision Language ModelImageTextMultimodalityBenchmark
🎯 What it does: Auto-Bench has been constructed, an automated evaluation framework for visual language models (VLM) based on large language models (LLM), capable of automatically generating and judging triples that include visual symbolization, questions, answers, and reasoning chains, covering four major capabilities: perception, reasoning, planning, and value alignment.
Large Language Models as Generalizable Policies for Embodied Tasks
Andrew Szot (Apple), Alexander T Toshev
Robotic IntelligenceTransformerLarge Language ModelReinforcement LearningVision-Language-Action ModelTextMultimodalityBenchmark
🎯 What it does: By freezing a pre-trained large language model and adding visual and action adapters, LLaRP, trained using reinforcement learning, can perform actions in multimodal language-driven robotic rearrangement tasks.
Large Language Models as Optimizers
Chengrun Yang (Google DeepMind), Xinyun Chen (Google DeepMind)
OptimizationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: An optimization framework OPRO is proposed, implemented through large language models (LLM), which utilizes natural language to describe problems and iteratively generate better solutions, applied to linear regression, the traveling salesman problem, and prompt optimization.
Large Language Models as Tool Makers
Tianle Cai (Google Deepmind), Denny Zhou (Google Deepmind)
Computational EfficiencyAI Code AssistantTransformerLarge Language ModelTextChain-of-Thought
🎯 What it does: A closed-loop framework called LATM (Large Language Models As Tool Makers) is proposed, allowing large language models to first generate reusable Python tools through powerful models (such as GPT-4), and then enabling lightweight models (such as GPT-3.5 Turbo) to utilize these tools to complete various tasks; a scheduler is also included to implement tool caching and generation in dynamic task flows.
Large Language Models Cannot Self-Correct Reasoning Yet
Jie Huang (University of Illinois at Urbana-Champaign), Denny Zhou (Google DeepMind)
TransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper evaluates the self-correction ability of large language models without external feedback, systematically examining methods such as self-correction, multi-agent debate, and prompt design, and emphasizes the necessity of using external feedback, appropriate baselines, and equivalent prompt designs.
Large Language Models to Enhance Bayesian Optimization
Tennison Liu (University of Cambridge), Mihaela van der Schaar (University of Cambridge)
OptimizationHyperparameter SearchTransformerLarge Language ModelPrompt EngineeringTabular
🎯 What it does: This paper proposes LLAMBO, which integrates large language models (LLM) into the Bayesian optimization process through natural language prompts, achieving functionalities such as zero-shot warm start, ICL-based surrogate models, and candidate point sampling.
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Jinyi Hu (Tsinghua University), Maosong Sun (Tsinghua University)
GenerationData SynthesisTransformerLarge Language ModelDiffusion modelContrastive LearningImageTextMultimodality
🎯 What it does: Proposed and implemented the MPM training paradigm, utilizing a multilingual large language model as a bridge between visual and target languages to achieve the training of multimodal models in non-English languages (e.g., Chinese VISCPM);
Large-scale Training of Foundation Models for Wearable Biosignals
Salar Abbaspourazad (Apple), Ian Shapiro (Apple)
ClassificationRepresentation LearningContrastive LearningTime SeriesBiomedical DataElectrocardiogram
🎯 What it does: Using PPG and ECG data collected from large-scale wearable devices, a foundational model is trained through self-supervised learning, followed by linear probing on various health and demographic targets.
Large-Vocabulary 3D Diffusion Model with Transformer
Ziang Cao (Nanyang Technological University), Ziwei Liu (Nanyang Technological University)
GenerationData SynthesisTransformerDiffusion modelPoint CloudMesh
🎯 What it does: A single generator based on a diffusion model is proposed, capable of generating high-quality 3D objects across multiple categories (hundreds of types);
Latent 3D Graph Diffusion
Yuning You (Texas A&M University), Yang Shen (Texas A&M University)
GenerationData SynthesisDrug DiscoveryGraph Neural NetworkDiffusion modelAuto EncoderContrastive LearningGraph
🎯 What it does: A complete pipeline for diffusion generation of 3D graphs (molecules) in a low-dimensional latent space is proposed—first, a decomposed 2D and 3D graph autoencoder is used to learn a low-dimensional latent representation with low reconstruction error and symmetry preservation, then a diffusion model is trained in that latent space, and finally decoded back to 3D graphs; it also supports conditional generation of SE(3) invariant properties or variable objects, and regularizes the latent space using graph self-supervised learning.
Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video
Xiangming Zhu (Shanghai Jiao Tong University), Xiaokang Yang (Shanghai Jiao Tong University)
GenerationData SynthesisOptimizationRecurrent Neural NetworkNeural Radiance FieldVideoPhysics Related
🎯 What it does: Utilize a single 3D video to learn the hidden physical properties of fluids and achieve new scene simulations of unknown geometries, boundary conditions, and dynamics through a latent intuitive physics framework.
Latent Representation and Simulation of Markov Processes via Time-Lagged Information Bottleneck
Marco Federici (University of Amsterdam), Bastiaan S. Veeling (Microsoft Research AI4Science)
Representation LearningFlow-based ModelContrastive LearningTime SeriesSequential
🎯 What it does: A latent simulation framework based on the information bottleneck has been constructed, which can efficiently and accurately simulate Markov processes over larger time steps.
Latent Trajectory Learning for Limited Timestamps under Distribution Shift over Time
QIUHAO Zeng, Boyu Wang (University of Western Ontario)
Domain AdaptationTime SeriesStochastic Differential Equation
🎯 What it does: This paper proposes an evolution domain generalization method based on continuous-time stochastic differential equations (SDE) called SDE-EDG. It first constructs infinite fine grid evolution trajectories (IFGET) through sample correspondence and linear interpolation, and then uses SDE to learn the continuous evolution of the latent space and aligns it with maximum likelihood path regularization to enhance the model's generalization ability on future unseen domains.
Layer-wise linear mode connectivity
Linara Adilova (Ruhr University Bochum), Martin Jaggi (École Polytechnique Fédérale de Lausanne)
Federated LearningConvolutional Neural NetworkTransformerImageText
🎯 What it does: This paper introduces the concept of Layered Linear Model Connectivity (LLMC), systematically analyzes and verifies the loss barriers produced by different layers and combinations of layers on model averaging (especially in the context of federated learning); experiments demonstrate that single-layer averaging has almost no barriers, while combinations of intermediate layers produce significant barriers; the LLMC phenomenon is explained from a robustness perspective, and its implications for personalization in federated learning are explored.
LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models
Zecheng Tang (Soochow University), Nan Duan (Microsoft Research Asia)
GenerationTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Reformulate the graphic layout generation task as a code generation task, and use large language models to complete the generation and rendering of layouts.
LCOT: Linear Circular Optimal Transport
Rocio P Diaz Martin, Soheil Kolouri (Vanderbilt University)
OptimizationComputational Efficiency
🎯 What it does: A linear circular optimal transport (LCOT) distance for unit circle probability measures is proposed, which can efficiently compute and embed measure spaces.
LDReg: Local Dimensionality Regularized Self-Supervised Learning
Hanxun Huang (University of Melbourne), James Bailey (University of Melbourne)
Object DetectionRepresentation LearningContrastive LearningImage
🎯 What it does: This paper proposes a self-supervised learning method based on Local Intrinsic Dimension Regularization (LDReg), aimed at preventing dimensional collapse.
LEAP: Liberate Sparse-View 3D Modeling from Camera Poses
Hanwen Jiang (University of Texas at Austin), Qixing Huang (University of Texas at Austin)
GenerationComputational EfficiencyRepresentation LearningTransformerNeural Radiance FieldPoint Cloud
🎯 What it does: This paper presents LEAP, a sparse view 3D modeling method that does not require camera pose.
Learning 3D Particle-based Simulators from RGB-D Videos
William F Whitney, Kelsey R Allen
GenerationData SynthesisConvolutional Neural NetworkGraph Neural NetworkNeural Radiance FieldVideo
🎯 What it does: An end-to-end learning 3D particle benchmark simulator (VPD) using multi-view RGB-D video, which can predict future frames and re-render and edit scenes from any viewpoint.
Learning Adaptive Multiresolution Transforms via Meta-Framelet-based Graph Convolutional Network
Tianze Luo (Nanyang Technological University), Sinno Jialin Pan (The Chinese University of Hong Kong)
Representation LearningMeta LearningGraph Neural NetworkGraph
🎯 What it does: This paper proposes a multi-resolution transformable graph convolutional network (MM-FGCN) that learns a multi-resolution basis for each graph instance through a meta-framework, achieving a unified representation of fine-grained and global features.
Learning Conditional Invariances through Non-Commutativity
Abhra Chaudhuri (University of Exeter), Anjan Dutta (University of Surrey)
Domain AdaptationMultimodality
🎯 What it does: This paper proposes achieving conditional invariant feature learning through Non-commutative Invariance (NCI), which retains only specific information from the target domain during inference while discarding irrelevant disturbances from the source domain.