ICML 2024 Papers — Page 24
International Conference on Machine Learning · 2610 papers
Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC
Wu Lin (Vector Institute), Alireza Makhzani (University of Toronto)
OptimizationConvolutional Neural NetworkGraph Neural NetworkTransformerImage
🎯 What it does: A memory-efficient natural gradient descent algorithm called SINGD is proposed, which is free of matrix inversion and combines the advantages of both KFAC and INGD methods.
StrWAEs to Invariant Representations
Hyunjong Lee (Seoul National University), Joong-Ho Won (Seoul National University)
ClassificationGenerationRepresentation LearningAuto EncoderImageTabular
🎯 What it does: This study explores how to utilize Wasserstein autoencoders to construct structured encoders that learn representations insensitive to interference information, and validates their effectiveness in tasks such as semi-supervised learning, conditional generation, and attribute transformation.
Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens
Ross M Clarke, José Miguel Hernández-Lobato (University of Cambridge)
OptimizationConvolutional Neural NetworkImageTabular
🎯 What it does: The study applies the damping of K-FAC and learning rate selection techniques to Adam, proposing a new AdamQLR optimizer;
StyDeSty: Min-Max Stylization and Destylization for Single Domain Generalization
Songhua Liu (National University of Singapore), Xinchao Wang (National University of Singapore)
ClassificationDomain AdaptationNeural Architecture SearchGenerative Adversarial NetworkImage
🎯 What it does: The StyDeSty framework is proposed, utilizing a min-max adversarial mechanism of stylization and destylization to achieve single-domain generalization.
Sub-token ViT Embedding via Stochastic Resonance Transformers
Dong Lao (University of California Los Angeles), Stefano Soatto (University of California Los Angeles)
Object DetectionSegmentationRetrievalKnowledge DistillationTransformerImageVideo
🎯 What it does: A method is proposed to enhance the sub-token embedding resolution of Vision Transformer by applying translational perturbations to the input images and aggregating at the feature level, without training or modifying the model.
Subequivariant Reinforcement Learning in 3D Multi-Entity Physical Environments
Runfa Chen (Tsinghua University), Wenbing Huang (Renmin University of China)
Robotic IntelligenceGraph Neural NetworkTransformerReinforcement LearningGraphBenchmark
🎯 What it does: In a 3D multi-entity physical environment, a hierarchical neural network (SHNN) that combines task allocation with local sub-equivariant message passing is proposed to learn multi-entity reinforcement learning strategies.
Subgoal-based Demonstration Learning for Formal Theorem Proving
Xueliang Zhao (University of Hong Kong), Lingpeng Kong
Graph Neural NetworkLarge Language ModelDiffusion modelText
🎯 What it does: A demonstration learning framework based on sub-goals is proposed to enhance the reasoning efficiency of LLMs in formal theorem proving.
Subgraphormer: Unifying Subgraph GNNs and Graph Transformers via Graph Products
Guy Bar-Shalom (Technion Israel Institute of Technology), Haggai Maron (NVIDIA Research)
Drug DiscoveryGraph Neural NetworkTransformerGraph
🎯 What it does: A new graph neural network architecture called Subgraphormer is proposed, which unifies subgraph GNNs with graph Transformers.
Subhomogeneous Deep Equilibrium Models
Pietro Sittoni (Gran Sasso Science Institute), Francesco Tudisco (University of Edinburgh)
ClassificationOptimizationGraph Neural NetworkImageGraphOrdinary Differential Equation
🎯 What it does: This paper presents an analysis of the existence and uniqueness of fixed points in deep equilibrium networks (DEQ) by combining sub-homogeneous operator theory with nonlinear Perron-Frobenius theory, and based on this, designs a sub-homogeneous deep equilibrium model (SubDEQ).
Submodular framework for structured-sparse optimal transport
Piyushi Manupriya (Indian Institute of Technology Hyderabad), Bamdev Mishra (Microsoft)
OptimizationMixture of ExpertsTabular
🎯 What it does: This paper proposes a structured sparse unbalanced optimal transport (UOT) framework that can learn interpretable sparse transport plans while satisfying upper bound sparsity constraints.
Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation
Ossi Räisä (University of Helsinki), Antti Honkela (University of Helsinki)
OptimizationSafty and Privacy
🎯 What it does: This paper studies the impact of batch size on effective noise variance in differential privacy stochastic gradient descent (DP-SGD) and proves that as the number of iterations approaches infinity, the noise standard deviation is linearly related to the sampling rate, leading to a significant reduction in total gradient variance with large batches.
Successor Features for Efficient Multi-Subject Controlled Text Generation
Meng Cao (McGill University), Samira Shabanian
GenerationLarge Language ModelReinforcement LearningText
🎯 What it does: The SF-GEN method is proposed, which separates the dynamics of the language model from task rewards using successor features, enabling multi-objective controllable text generation.
SuDA: Support-based Domain Adaptation for Sim2Real Hinge Joint Tracking with Flexible Sensors
Fang Jiawei, Yipeng Qin (Cardiff University)
Pose EstimationDomain AdaptationRecurrent Neural NetworkVideo
🎯 What it does: A Sim2Real approach is proposed, utilizing low-dimensional flexible sensors for unsupervised domain adaptation of human joint angles, constructing a Support-based Domain Adaptation (SuDA) method that enables joint angle prediction without any real labeled data.
Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction
Diwen Wan (Peking University), Gang Zeng (Peking University)
Gaussian SplattingPoint Cloud
🎯 What it does: The Superpoint Gaussian Splatting (SP-GS) framework is proposed, which clusters 3D Gaussians with similar transformations into superpoints to achieve real-time high-quality dynamic scene rendering.
Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
Thomas Merth (Apple), Mahyar Najibi (Apple)
GenerationRetrievalComputational EfficiencyTransformerPrompt EngineeringTextRetrieval-Augmented Generation
🎯 What it does: A new prompting method called Superposition Prompting is proposed for Retrieval-Augmented Generation (RAG), which significantly improves inference speed and accuracy by parallelizing documents into paths and dynamically pruning irrelevant paths.
Supervised Matrix Factorization: Local Landscape Analysis and Applications
Joowon Lee (University of Wisconsin), Weixin Yao (University of California)
ClassificationOptimizationText
🎯 What it does: Conduct a local landscape analysis of supervised matrix factorization (SMF), design a block coordinate descent (BCD) algorithm with adaptive step size, and provide convergence and iteration complexity; introduce minimum L2 regularization to ensure local strong convexity and prove statistical consistency; implement a GPU-friendly neural network version of BCD and validate theory and practice on various datasets.
Surface-VQMAE: Vector-quantized Masked Auto-encoders on Molecular Surfaces
Fang Wu (Westlake University), Stan Z. Li (Westlake University)
Drug DiscoveryProtein Structure PredictionTransformerAuto EncoderPoint Cloud
🎯 What it does: This paper proposes Surface-VQMAE, an unsupervised pre-training framework based on protein surface point clouds, which uses chunking, Morton sequence sorting, SurfFormer Transformer, and vector quantization MAE to learn surface features.
SurfPro: Functional Protein Design Based on Continuous Surface
Zhenqiao Song (Carnegie Mellon University), Wengong Jin (Broad Institute of MIT and Harvard)
Drug DiscoveryProtein Structure PredictionGraph Neural NetworkTransformerLarge Language ModelPoint Cloud
🎯 What it does: This paper presents SurfPro, a functional protein design method based on protein continuous surfaces and their chemical properties, which can directly generate amino acid sequences that meet specific functions from a given surface;
Surprisingly Strong Performance Prediction with Neural Graph Features
Gabriela Kadlecová (Charles University), Frank Hutter (University of Freiburg)
OptimizationNeural Architecture SearchGraph Neural NetworkTabularBenchmark
🎯 What it does: A performance prediction method based on Graph Neural Features (GRAF) is proposed, which can accurately predict the performance of architectures in the NAS search space without training the network.
Swallowing the Bitter Pill: Simplified Scalable Conformer Generation
Yuyang Wang (Apple), Miguel Ángel Bautista (Apple)
GenerationDrug DiscoveryGraph Neural NetworkTransformerDiffusion modelScore-based ModelGraph
🎯 What it does: This paper proposes a molecular conformation generation method based on diffusion models—Molecular Conformer Fields (MCF), treating conformations as functions on molecular graphs and directly learning distributions in three-dimensional coordinate space.
Switchable Decision: Dynamic Neural Generation Networks
Shujian Zhang (University of Texas at Austin), Mingyuan Zhou (University of Texas at Austin)
GenerationComputational EfficiencyTransformerReinforcement LearningText
🎯 What it does: For autoregressive generative models, a Switchable Decision mechanism is proposed to dynamically determine whether to skip the attention layer, feed-forward layer, and input tokens, significantly reducing inference computation while maintaining output quality.
Switched Flow Matching: Eliminating Singularities via Switching ODEs
Qunxi Zhu (Fudan University), Wei Lin (Fudan University)
GenerationData SynthesisOptimizationFlow-based ModelImageMultimodalityOrdinary Differential Equation
🎯 What it does: A Switched Flow Matching (SFM) framework is proposed, which eliminates singularities caused by the heterogeneity of source/target distributions in Flow Matching (FM) by switching between multiple ODEs during the generation process, thereby achieving smoother and reversible probability flows.
Switching the Loss Reduces the Cost in Batch Reinforcement Learning
Alex Ayoub (University of Alberta), Csaba Szepesvari (University of Alberta)
Reinforcement LearningTabularSequential
🎯 What it does: This paper proposes the use of log-loss in batch reinforcement learning to train fitted Q-iteration (FQI-LOG) and provides a small-cost bound for sample complexity in the case of small optimal costs.
SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment
Ziping Ma (Ant Group), Qingpei Guo (Ant Group)
GenerationRetrievalTransformerVision Language ModelContrastive LearningImageTextMultimodality
🎯 What it does: The SyCoCa framework is proposed in visual language pre-training, integrating bidirectional local interaction text-guided masked image modeling with traditional contrastive learning and image description tasks.
Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion
Yujia Huang (California Institute of Technology), Yisong Yue (California Institute of Technology)
GenerationTransformerDiffusion modelAudio
🎯 What it does: This paper proposes a symbolic music generation framework based on diffusion models and designs a Stochastic Control Guidance (SCG) algorithm that can be directly used on pre-trained models to achieve automatic guidance for non-differentiable rules.
Symmetric Matrix Completion with ReLU Sampling
Huikang Liu (Shanghai Jiao Tong University), Laura Balzano (University of Michigan)
Optimization
🎯 What it does: This paper studies the problem of completing symmetric positive semi-definite low-rank matrices and proposes a non-convex matrix factorization method under ReLU (only observing non-negative entries) sampling; it proves that under certain assumptions, the global optimal solution can accurately recover the original matrix, and the objective function exhibits geographical strong convexity on the manifold near the global optimum; based on this, a specialized initialization scheme is designed to ensure that gradient descent converges to the global optimum; the theory is also extended to scenarios with noise and threshold sampling; experiments are conducted on synthetic data.
Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization
Hyeonah Kim (Korea Advanced Institute of Science and Technology), Jinkyoo Park
OptimizationReinforcement Learning
🎯 What it does: This paper proposes a Symmetric Replay Training (SRT) method, which enhances the sample efficiency of deep reinforcement learning in combinatorial optimization by performing symmetric transformations on high-reward trajectories obtained and replaying them without additional reward evaluations.
Symmetry Induces Structure and Constraint of Learning
Liu Ziyin (Massachusetts Institute of Technology)
Convolutional Neural NetworkImage
🎯 What it does: A unified theory of mirror symmetry is proposed, demonstrating that symmetry induces structured constraints in the learning process, and using this theory to explain phenomena such as sparsity, low-rank, and homogenization.
Synergistic Integration of Coordinate Network and Tensorial Feature for Improving Neural Radiance Fields from Sparse Inputs
Mingyu Kim (KAIST), Jin-Hwa Kim (NAVER AI Lab)
GenerationData SynthesisNeural Radiance FieldImage
🎯 What it does: A new sparse input NeRF training method is proposed by combining multi-plane representation with coordinate networks.
TabLog: Test-Time Adaptation for Tabular Data Using Logic Rules
Weijieying Ren (Pennsylvania State University), Vasant G Honavar
Domain AdaptationContrastive LearningTabular
🎯 What it does: A rule set model based on logical rules is proposed, and during testing, self-supervised contrastive loss adaptive learning is utilized with unlabelled target domain data to address the distribution shift problem in tabular data.
Tabular Insights, Visual Impacts: Transferring Expertise from Tables to Images
Jun-Peng Jiang (Nanjing University), De-Chuan Zhan (Nanjing University)
ClassificationExplainability and InterpretabilityImageTabular
🎯 What it does: This paper proposes a new method called CHARMS, aimed at transferring expert knowledge from tabular data to image models to improve image classification performance, especially when tabular data is lacking during the inference phase.
Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation
Wanpeng Zhang (Peking University), Zongqing Lu (Peking University)
Graph Neural NetworkReinforcement LearningAuto EncoderSequential
🎯 What it does: A dual GAT structure is proposed to address the non-stationarity problem in reinforcement learning by learning 'Causal Origin Representation (COREP)'.
Tackling Prevalent Conditions in Unsupervised Combinatorial Optimization: Cardinality, Minimum, Covering, and More
Fanchen Bu (Korea Advanced Institute of Science and Technology), Kijung Shin (Korea Advanced Institute of Science and Technology)
OptimizationGraph Neural NetworkGraph
🎯 What it does: The UCOM2 framework is proposed for unsupervised solving of combinatorial optimization problems, within which probability objectives are constructed for common constraints (cardinality constraints, minimum/maximum subset distance, coverage, clique/independent set, non-binary decisions, and uncertainty) and an incremental greedy de-randomization algorithm is designed.
Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains
Junhong Shen (Carnegie Mellon University), Nicolo Fusi (Microsoft Research)
Drug DiscoveryTransformerLarge Language ModelSupervised Fine-TuningTextBiomedical Data
🎯 What it does: Utilizing a general large language model, domain labels and functional labels that can be learned are introduced for task solving in specialized fields (such as multilingual translation, protein sequences, and chemical SMILES).
Tandem Transformers for Inference Efficient LLMs
Aishwarya P S (Google DeepMind), Praneeth Netrapalli (Google DeepMind)
Computational EfficiencyKnowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper proposes a Tandem Transformers architecture that combines small autoregressive models with large block models to enhance inference efficiency; it also embeds this architecture into Speculative Decoding (SPEED) and introduces adaptive block lengths.
Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation
Fengdi Che (University of Alberta), Dale Schuurmans (University of Alberta)
Reinforcement Learning
🎯 What it does: This paper proves that combining target networks with over-parameterized linear function approximation can guarantee the convergence of TD algorithms in offline or off-policy learning, addressing the long-standing 'deadly triad' problem.
Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models
Yongxian Wei (Tsinghua University), Dacheng Tao (Nanyang Technological University)
Meta LearningConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: To address the task conflict problem caused by the heterogeneity of pre-trained models in data-free meta-learning (DFML), the authors propose a Task Groupings Regularization method. This involves embedding tasks into the model and grouping them through spectral clustering, followed by using implicit gradient regularization within each group to align conflicting tasks, ultimately training a meta-model with better generalization capabilities.
Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning
Yusong Hu (Xidian University), Xinbo Gao (Chongqing University of Post and Telecommunications)
Image
🎯 What it does: Proposes a Task-aware Orthogonal Sparse Network (OSN) method, which utilizes a three-part network structure (old task sub-network, shared parameter sub-network, free parameter sub-network) and sharpness-aware orthogonal projection to maintain the performance of old tasks while enhancing the adaptability to new tasks in continual learning.
Taylor Videos for Action Recognition
Lei Wang (Australian National University), Liang Zheng (Australian National University)
RecognitionConvolutional Neural NetworkTransformerOptical FlowVideo
🎯 What it does: Proposes Taylor Video, a method that extracts motion information of displacement, velocity, and acceleration in three channels from the differences in grayscale frames using Taylor expansion;
Tell, Don't Show: Language Guidance Eases Transfer Across Domains in Images and Videos
Tarun Kalluri (University of California San Diego), Manmohan Chandraker (Allen Institute for Artificial Intelligence)
Domain AdaptationTransformerLarge Language ModelSupervised Fine-TuningImageVideoText
🎯 What it does: This paper proposes LaGTran, a framework for cross-domain image and video transfer learning using text supervision.
Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning
Zijian Guo (Boston University), Wenchao Li (Boston University)
Safty and PrivacyTransformerReinforcement LearningTabularBenchmark
🎯 What it does: A new model called Temporal Logic Specification-Conditioned Decision Transformer (SDT) is proposed in offline safe reinforcement learning, which learns a policy that maximizes rewards while satisfying given STL constraints from a fixed dataset.
Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning
Mingqing Xiao (Peking University), Zhouchen Lin (Peking University)
Graph Neural NetworkSpiking Neural NetworkGraph
🎯 What it does: A time-based spiking neural network (GRSNN) based on synaptic delay is proposed for reasoning tasks in knowledge graphs and general graphs.
TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision
Zhuo Chen (Massachusetts Institute of Technology), Di Luo (Harvard University)
OptimizationTime SeriesPhysics RelatedOrdinary Differential Equation
🎯 What it does: The researchers proposed a time evolution method based on natural gradient, TENG, for solving initial value PDEs on deep neural networks, significantly improving accuracy to machine precision.
TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors
Yichuan Mo (Peking University), Yisen Wang (Peking University)
GenerationData SynthesisSafty and PrivacyDiffusion modelImageStochastic Differential Equation
🎯 What it does: A unified framework named TERD is proposed for identifying and eliminating backdoor attacks in diffusion models, including the reverse recovery of triggers and detection of inputs/models.
Test-Time Degradation Adaptation for Open-Set Image Restoration
Yuanbiao Gou (Sichuan University), Xi Peng (Sichuan University)
RestorationDiffusion modelImage
🎯 What it does: A test-time adaptive open-set image restoration framework TAO is proposed, which combines a pre-trained diffusion model, a test-time degradation adapter, and a dynamic guidance strategy to achieve image restoration for single-sample unknown degradations.
Test-Time Model Adaptation with Only Forward Passes
Shuaicheng Niu (Nanyang Technological University), Peilin Zhao (Tencent AI Lab)
Domain AdaptationTransformerPrompt EngineeringImage
🎯 What it does: This paper proposes a testing-time adaptation method called FOA, which utilizes only forward propagation, employing input prompt learning and activation shifting to achieve online adaptation to distribution drift.
Test-Time Regret Minimization in Meta Reinforcement Learning
Mirco Mutti (Technion Israel Institute of Technology), Aviv Tamar (Technion Israel Institute of Technology)
Meta LearningReinforcement Learning
🎯 What it does: A theoretical analysis of regret minimization at test-time for meta-reinforcement learning (meta-RL) after perfect training is provided, presenting optimal lower bounds and corresponding upper bounds and algorithms under various structural assumptions;
Testing the Feasibility of Linear Programs with Bandit Feedback
Aditya Gangrade (Boston University), Clayton Scott (University of Michigan)
OptimizationReinforcement Learning
🎯 What it does: This paper proposes a method for testing the feasibility of linear programming in environments with linear bandit feedback.
The Balanced-Pairwise-Affinities Feature Transform
Daniel Shalam (University of Haifa), Simon Korman (University of Haifa)
ClassificationRecognitionOptimizationTransformerImage
🎯 What it does: This paper proposes a parameter-free, differentiable, and equivariant Balanced-Pairwise-Affinities (BPA) feature transformation, which re-embeds set features using self-matching minimum cost maximum flow/Optimal Transport methods, thereby enhancing the performance of set input tasks (such as few-shot classification, unsupervised clustering, and person re-identification).
The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents
Yatin Dandi (Ecole Polytechnique Federale de Lausanne), Florent Krzakala (Ecole Polytechnique Federale de Lausanne)
OptimizationTabular
🎯 What it does: This paper studies the use of multi-pass gradient descent (GD) with repeated batches of the same data in a two-layer neural network, exploring its dynamic performance when learning multi-objective functions.
The Computational Complexity of Finding Second-Order Stationary Points
Andreas Kontogiannis (National Technical University of Athens), Ioannis Panageas (University of California)
Optimization
🎯 What it does: This paper studies the computational complexity of finding approximate second-order stationary points (SOSP) in unconstrained non-convex optimization; by proving that this problem belongs to the PLS class and is PLS-complete, it addresses a previously unresolved open question.
The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks
Songhua He (Rutgers University), Periklis A. Papakonstantinou (Rutgers University)
Recurrent Neural Network
🎯 What it does: This paper analyzes the impact of high-precision weights on the number of neurons in deep ReLU networks and provides corresponding theoretical limits and size metrics.
The Emergence of Reproducibility and Consistency in Diffusion Models
Huijie Zhang (University of Michigan), Qing Qu (University of Michigan)
GenerationData SynthesisDiffusion modelImage
🎯 What it does: This paper explores and verifies the phenomenon of 'repeatability' in diffusion models, which produce highly consistent outputs under the same noise input.
The Entropy Enigma: Success and Failure of Entropy Minimization
Ori Press (University of Tübingen), Matthias Bethge (University of Tübingen)
Domain AdaptationOptimizationImage
🎯 What it does: This study investigates the mechanism of Entropy Minimization (EM) in test-time adaptation and proposes a label-flipping-based method for estimating unlabeled accuracy (Weighted Flips, WF).
The Expressive Power of Path-Based Graph Neural Networks
Caterina Graziani (University of Siena), Thomas Gärtner (TU Wien)
Graph Neural NetworkGraph
🎯 What it does: This paper systematically studies the expressive power of path-based graph neural networks, proposing a new color refinement algorithm called PATH-WL, and designs a GNN structure named PAIN that is equivalent to PATH-WL based on this theory.
The Fundamental Limits of Least-Privilege Learning
Theresa Stadler (École Polytechnique Fédérale de Lausanne), Carmela Troncoso (École Polytechnique Fédérale de Lausanne)
Safty and PrivacyRepresentation LearningConvolutional Neural NetworkReinforcement LearningImageTabular
🎯 What it does: This paper proposes and formalizes the concept of 'Least-Privilege Learning' (LPL) and proves through information theory that in common scenarios with label uncertainty, any feature representation useful for the target task will inevitably leak other attribute information besides the task label, making it impossible to achieve both high performance and full compliance with the least privilege principle.
The Good, The Bad, and Why: Unveiling Emotions in Generative AI
CHENG LI, Xing Xie (Microsoft Research)
GenerationExplainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringTextMultimodality
🎯 What it does: Three emotional stimulation methods were proposed and experimented with—EmotionPrompt (enhancing performance), EmotionAttack (reducing performance), and EmotionDecode (explaining emotional mechanisms), and their effects were evaluated on various generative AI models;
The Illusion of State in State-Space Models
William Merrill (New York University), Ashish Sabharwal (Allen Institute for Artificial Intelligence)
Recurrent Neural NetworkTransformerTabularSequential
🎯 What it does: This paper analyzes the expressive power of common linear state space models (such as S4 and Mamba) in state tracking, proving that they are limited by TC₀ like Transformers and cannot solve permutation composition and other NC1-hard problems, and proposes minimal modifications to overcome this limitation.
The Linear Representation Hypothesis and the Geometry of Large Language Models
Kiho Park (University of Chicago), Victor Veitch (University of Chicago)
Representation LearningTransformerLarge Language ModelText
🎯 What it does: Formalize the linear representation hypothesis in large language models, define three types of representations: subspace, measurement, and intervention, and introduce a causal inner product framework to unify these representations; validate the linearity and controllability of conceptual directions on LLaMA-2.
The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm
Giseung Park (Korea Advanced Institute of Science and Technology), Youngchul Sung (Korea Advanced Institute of Science and Technology)
OptimizationReinforcement Learning
🎯 What it does: This paper proposes a max-min fairness framework in multi-objective reinforcement learning (MORL) and provides the corresponding linear programming theory and model-independent algorithms.
The Merit of River Network Topology for Neural Flood Forecasting
Nikolas Kirschstein (University of Oxford), Yixuan Sun (Technical University of Munich)
Graph Neural NetworkGraphTime Series
🎯 What it does: This paper studies the impact of river network topology information on river flow prediction based on graph neural networks (GNNs). It constructs a GNN model based on the LamaH-CE dataset and compares the effects of different adjacency definitions (no adjacency, binary adjacency, physical weights, learned weights) on prediction performance.
The Non-linear $F$-Design and Applications to Interactive Learning
Alekh Agarwal (Google), Tong Zhang (University of Illinois Urbana-Champaign)
OptimizationReinforcement Learning
🎯 What it does: This paper proposes an experimental design framework suitable for nonlinear function classes—F-design—and introduces a corresponding F-condition number metric to measure design quality.
The Perception-Robustness Tradeoff in Deterministic Image Restoration
Guy Ohayon (Technion), Michael Elad (Technion)
RestorationSuper ResolutionAdversarial AttackGenerative Adversarial NetworkImage
🎯 What it does: This paper studies the properties of deterministic image restoration methods and proves that in the case of irreversible degradation, to achieve high perceptual quality and consistency (i.e., consistency with observations and similarity to natural image distributions), the Lipschitz constant of the prediction function must be large, leading to susceptibility to adversarial attacks. The authors validate and demonstrate this perceptual-robustness trade-off through theoretical derivation, experiments on various super-resolution models (calculating Lipschitz lower bounds, joint perceptual metrics), and adversarial attacks and posterior exploration on GFPGAN.
The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks
Ziquan Liu (Queen Mary University of London), Antoni B. Chan (City University of Hong Kong)
Adversarial AttackImage
🎯 What it does: This paper discusses synthetic prediction under adversarial attacks and proposes targeted adversarial training to enhance the efficiency of the prediction ensemble.
The Pitfalls of Next-Token Prediction
Gregor Bachmann (ETH Zurich), Vaishnavh Nagarajan (Google Research)
TransformerSupervised Fine-TuningGraph
🎯 What it does: This paper reveals that in tasks requiring forward planning (such as pathfinding), teacher-forced training can lead to two types of failures, namely 'clever mouse cheating' and 'hard-to-decode tokens', which result in the inability to learn the correct planning strategy even on in-distribution samples, by comparing autoregressive inference and teacher-forced training in two next-word prediction models.
The Privacy Power of Correlated Noise in Decentralized Learning
Youssef Allouah (École Polytechnique Fédérale de Lausanne), Rachid Guerraoui (École Polytechnique Fédérale de Lausanne)
Federated LearningSafty and PrivacyImageTabular
🎯 What it does: The DECOR algorithm is proposed, which achieves differential privacy in decentralized learning by introducing mutually canceling correlated Gaussian noise.
The Relative Value of Prediction in Algorithmic Decision Making
Juan Carlos Perdomo (Harvard University)
🎯 What it does: This study investigates the relative value of improving predictive models and expanding intervention coverage in resource-constrained social planning for enhancing social welfare.
The Role of Learning Algorithms in Collective Action
Omri Ben-Dov (Max Planck Institute for Intelligent Systems), Amartya Sanyal (Max Planck Institute for Intelligent Systems)
Reinforcement LearningImage
🎯 What it does: This paper studies the impact of learning algorithms on the success rate of collective action (embedded signals), proposes the concept of 'effective collective size', and explores the role of DRO, iterative weighting algorithms, and the simplicity bias of SGD on collective success through theoretical and experimental approaches.
The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright BreachesWithout Adjusting Finetuning Pipeline
Haonan Wang (National University of Singapore), Kenji Kawaguchi (National University of Singapore)
GenerationData SynthesisAdversarial AttackTransformerLarge Language ModelDiffusion modelImageTextMultimodality
🎯 What it does: A backdoor attack method called SilentBadDiffusion is designed, which utilizes the addition of poisoned text-image pairs in the training data, allowing the target text-to-image diffusion model to generate copyrighted works under specific trigger prompts without altering the training process.
The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling
Jiajun Ma (Hong Kong University of Science and Technology), Kenji Kawaguchi (National University of Singapore)
GenerationData SynthesisConvolutional Neural NetworkDiffusion modelImageOrdinary Differential Equation
🎯 What it does: This paper proposes a training-free Skip-Tuning method that enhances the generation quality of diffusion models in few-step sampling by adjusting the strength of skip connections in UNet.
The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning
Nathaniel Li (Center for AI Safety), Dan Hendrycks (Center for AI Safety)
Adversarial AttackTransformerLarge Language ModelSupervised Fine-TuningTextBiomedical DataBenchmark
🎯 What it does: This paper proposes a public dangerous knowledge benchmark WMDP and develops a new machine unlearning method RMU based on this benchmark to remove harmful knowledge from large language models.
Theoretical Analysis of Learned Database Operations under Distribution Shift through Distribution Learnability
Sepanta Zeighami (University of California), Cyrus Shahabi (University of Southern California)
🎯 What it does: This study investigates the theoretical performance of using machine learning models for database operations (indexing, cardinality estimation, and sorting) under data distribution drift, and provides the expected complexity for model training, querying, and insertion.
Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians
Tom Huix (Ecole Polytechnique), Eric Moulines (Ecole Polytechnique)
OptimizationMixture of ExpertsMultimodality
🎯 What it does: This paper studies the theoretical guarantees of fixed variance, equal-weight Gaussian mixture models in variational inference, proposing an asymptotic descent lemma and an upper bound on the approximation error;
Theoretical insights for diffusion guidance: A case study for Gaussian mixture models
Yuchen Wu (University of Pennsylvania), Yuting Wei (University of Pennsylvania)
ClassificationGenerationDiffusion modelStochastic Differential EquationOrdinary Differential Equation
🎯 What it does: This paper provides a theoretical analysis of guidance (classifier/classifier-free) for diffusion models within the framework of Gaussian Mixture Models (GMM), proving that guidance can enhance classification confidence, reduce sample diversity, and revealing potential phase transitions that may occur under strong guidance.
Theory of Consistency Diffusion Models: Distribution Estimation Meets Fast Sampling
Zehao Dou (Yale University), Zhuoran Yang (Princeton University)
GenerationData SynthesisOptimizationDiffusion modelScore-based ModelImageOrdinary Differential Equation
🎯 What it does: This paper presents the statistical theory of Consistency Diffusion Models and reformulates its training objective as a Wasserstein distance minimization problem.
Thermometer: Towards Universal Calibration for Large Language Models
Maohao Shen (Massachusetts Institute of Technology), Soumya Ghosh (IBM Research)
OptimizationTransformerLarge Language ModelText
🎯 What it does: A post-hoc calibration method named Thermometer is designed and implemented to predict and calibrate the temperature parameter for large-scale language models (LLMs) without using labeled data, thereby enhancing the statistical calibration of their probability predictions.
Think Before You Act: Decision Transformers with Working Memory
Jikun Kang (McGill University), Jie Fu (Polytechnique Montreal)
TransformerReinforcement LearningSequential
🎯 What it does: Proposed the Decision Transformer with Working Memory (DT-Mem), which uses an explicit working memory matrix to store, update, and retrieve task-related information, thereby enhancing the generalization and training efficiency of offline reinforcement learning agents;
TIC-TAC: A Framework For Improved Covariance Estimation In Deep Heteroscedastic Regression
Megh Shukla (Ecole Polytechnique Federale de Lausanne), Alexandre Alahi (Ecole Polytechnique Federale de Lausanne)
Pose EstimationOptimizationTabular
🎯 What it does: This paper proposes two methods that utilize the gradient and curvature from Taylor expansion to approximate the covariance of predictive distributions (Taylor Induced Covariance, TIC), and introduces the Task Agnostic Correlations (TAC) metric to evaluate the quality of covariance, thereby achieving better covariance estimation and training convergence in deep heteroscedastic regression.
Tight Partial Identification of Causal Effects with Marginal Distribution of Unmeasured Confounders
Zhiheng Zhang (Tsinghua University)
TabularFinance Related
🎯 What it does: This paper studies the tightest partial identification intervals for causal effects (P(Y=1|doX=x) and ATE) given only the marginal distribution P(U) of the potential confounder U, without imposing any additional constraints on P(U).
Tilt and Average : Geometric Adjustment of the Last Layer for Recalibration
Gyusang Cho (Korea Advanced Institute of Science and Technology), Chan-Hyun Youn (Korea Advanced Institute of Science and Technology)
OptimizationConvolutional Neural NetworkTransformerImage
🎯 What it does: This paper proposes a new recalibration method called Tilt and Average (TNA), which adjusts the model's confidence and improves calibration performance by geometrically rotating and averaging the weights of the final layer.
Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers
Johann Schmidt (Otto-von-Guericke University), Sebastian Stober (Otto-von-Guericke University)
ClassificationRecognitionConvolutional Neural NetworkImage
🎯 What it does: This paper proposes an Inverse Transformation Search (ITS) framework that post-processes a trained image classifier during the inference phase, enabling it to achieve zero-shot pseudo-invariance on unseen spatial transformation inputs.
Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks
Stefano Sarao Mannelli (University College London), Luca Saglietti (Bocconi University)
ImageOrdinary Differential Equation
🎯 What it does: This paper explores the interaction between over-parameterization and curriculum learning in the XOR-style high-dimensional Gaussian mixture problem, and verifies its impact on real datasets.
Time Series Diffusion in the Frequency Domain
Jonathan Crabbé (University of Cambridge), Mihaela van der Schaar (University of Cambridge)
GenerationData SynthesisAnomaly DetectionTransformerDiffusion modelScore-based ModelTime SeriesBiomedical DataElectrocardiogramFinance RelatedStochastic Differential Equation
🎯 What it does: Implement and evaluate a frequency domain diffusion model on time series data, proposing a diffusion SDE that uses mirrored Brownian motion in the frequency domain, and training a score model through denoising score matching.
Time Weaver: A Conditional Time Series Generation Model
Sai Shankar Narasimhan (University of Texas), Sandeep P. Chinchali (University of Texas)
GenerationData SynthesisTransformerDiffusion modelContrastive LearningMultimodalityTime SeriesElectrocardiogram
🎯 What it does: A diffusion model named TIME WEAVER is proposed, which can generate high-quality time series data based on multimodal time series contextual metadata (categorical, continuous, time-varying).
Time-Series Forecasting for Out-of-Distribution Generalization Using Invariant Learning
Haoxin Liu (Georgia Institute of Technology), B. Aditya Prakash (Georgia Institute of Technology)
Domain AdaptationRecurrent Neural NetworkTime SeriesFinance Related
🎯 What it does: This paper proposes a time series forecasting method called FOIL based on invariant learning, aimed at addressing the out-of-distribution generalization problem in time series forecasting.
TimeMIL: Advancing Multivariate Time Series Classification via a Time-aware Multiple Instance Learning
Xiwen Chen (Clemson University), Abolfazl Razi (Clemson University)
ClassificationTransformerTime Series
🎯 What it does: We propose TimeMIL, a weakly supervised multi-instance learning framework for multivariate time series classification that can locate important moments in the time series.
Timer: Generative Pre-trained Transformers Are Large Time Series Models
Yong Liu (Tsinghua University), Mingsheng Long (Tsinghua University)
Anomaly DetectionTransformerLarge Language ModelSupervised Fine-TuningTime Series
🎯 What it does: A unified time series dataset UTSD containing 1 billion temporal points was constructed, proposing a single series sequence (S3) format to unify heterogeneous multivariate time series. A GPT-style autoregressive pre-trained Transformer (Timer) was used and fine-tuned and evaluated on multiple tasks such as prediction, missing imputation, and anomaly detection.
TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling
Jiaxiang Dong (Tsinghua University), Mingsheng Long (Tsinghua University)
ClassificationRepresentation LearningTransformerContrastive LearningTime SeriesSequential
🎯 What it does: This paper proposes TimeSiam, a self-supervised pre-training framework based on Siamese networks, which reconstructs the current subsequence using past subsequences to learn temporal correlations.
TimeX++: Learning Time-Series Explanations with Information Bottleneck
Zichuan Liu (Nanjing University), Dongsheng Luo (Florida International University)
Explainability and InterpretabilityTransformerReinforcement LearningTime SeriesElectrocardiogram
🎯 What it does: This paper proposes a new time series interpretability framework TIMEX++, which improves the information bottleneck (IB) principle to learn interpretable subsequences that maintain label consistency and do not deviate from the original distribution, generating highly interpretable masks on black-box deep learning models.
tinyBenchmarks: evaluating LLMs with fewer examples
Felipe Maia Polo (University of Michigan), Mikhail Yurochkin (IBM Research)
OptimizationTransformerLarge Language ModelTextBenchmark
🎯 What it does: By selecting a small number (≈100) of carefully curated examples in each evaluation scenario and using Item Response Theory (IRT)-based tools, we can quickly and accurately estimate the overall performance of large language models (LLMs), significantly reducing evaluation costs.
TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge
Young D. Kwon (University of Cambridge), Cecilia Mascolo (University of Cambridge)
ClassificationComputational EfficiencyMeta LearningConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: TinyTrain implements adaptive sparse training with a small amount of labeled data on resource-constrained edge devices, significantly reducing memory and computational overhead while maintaining high accuracy.
tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs)
Junhua Zeng (Guangdong University of Technology), Guoxu Zhou (Guangdong University of Technology)
CompressionOptimizationTransformerLarge Language ModelPrompt EngineeringImage
🎯 What it does: Utilizing large language models to automatically generate and iteratively improve the tensor network structure search (TN-SS) algorithm, experiments have shown that its performance surpasses that of existing manually designed algorithms.
To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
Zi-Hao Qiu (Nanjing University), Tianbao Yang (Texas A&M University)
RetrievalOptimizationTransformerLarge Language ModelContrastive LearningImageText
🎯 What it does: This paper proposes a temperature network framework (TempNet) based on Distributed Robust Optimization (DRO), which can automatically predict personalized temperatures for each sample during the training and inference of large models (LLM and CLIP), enhancing model performance.
To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models
George-Octavian Bărbulescu (University of Warwick), Peter Triantafillou (University of Warwick)
OptimizationSafty and PrivacyTransformerLarge Language ModelContrastive LearningText
🎯 What it does: The study investigates how to personalize forgetting of different text sequences in large language models to reduce privacy and copyright leakage.
To the Max: Reinventing Reward in Reinforcement Learning
Grigorii Veviurko (Delft University of Technology), Mathijs de Weerdt (Delft University of Technology)
Reinforcement Learning
🎯 What it does: A reinforcement learning framework aimed at maximizing rewards (max-reward RL) is proposed, along with its theoretical foundation, Bellman-type recursive formulas, and policy gradient theorem.
Token-level Direct Preference Optimization
Yongcheng Zeng (Institute of Automation, Chinese Academy of Sciences), Jun Wang (University College London)
OptimizationReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: A Token-level Direct Preference Optimization (TDPO) method for aligning LLMs is proposed, utilizing forward KL constraints to control the bias of each token and achieving direct policy optimization through tokenization of the Bradley-Terry model.
Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models
Mingjia Huo (University of California), Pengtao Xie (University of California)
OptimizationTransformerLarge Language ModelSupervised Fine-TuningContrastive LearningText
🎯 What it does: A Token-Specific watermarking method based on multi-objective optimization is proposed, which dynamically generates the segmentation ratio (γ) and watermark logit (δ) for each token using a lightweight network during the LLM inference phase, thereby enhancing watermark detectability without compromising the semantic coherence of the text.
Topological Neural Networks go Persistent, Equivariant, and Continuous
Yogesh Verma (Aalto University), Vikas Garg
ClassificationDrug DiscoveryGraph Neural NetworkGraphOrdinary Differential Equation
🎯 What it does: This paper proposes a unified framework called TopNets, which integrates Topological Neural Networks (TNN) with Persistent Homology (PH), and further extends to Euclidean symmetry (E(n)) and continuous time (ODE) forms;
Total Variation Distance Meets Probabilistic Inference
Arnab Bhattacharyya (National University of Singapore), N. V. Vinodchandran (University of Nebraska - Lincoln)
Graph
🎯 What it does: This paper establishes a new connection between total variation (TV) distance estimation and probabilistic inference, proposing an efficient structure-preserving simplification method from relatively approximate TV distance to probabilistic inference in directed graphical models.
Total Variation Floodgate for Variable Importance Inference in Classification
Wenshuo Wang (Harvard University), Aaditya Ramdas (Carnegie Mellon University)
ClassificationExplainability and InterpretabilityTabular
🎯 What it does: This paper studies a model-free variable importance measure for classification problems—Expected Total Variation (ETV)—and provides statistical inference methods.