ICLR 2023 Papers — Page 15
International Conference on Learning Representations · 1573 papers
The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognition
Jun CEN, Qifeng Chen (Hong Kong University of Science and Technology)
ClassificationRecognitionSupervised Fine-TuningImageVideoBenchmark
🎯 What it does: This paper systematically studies the Unified Open Set Recognition (UOSR) problem, analyzes its uncertainty distribution, and proposes a few-shot UOSR evaluation framework and the FS-KNNS method.
The hidden uniform cluster prior in self-supervised learning
Mido Assran, Nicolas Ballas (Meta AI)
Representation LearningTransformerContrastive LearningImage
🎯 What it does: This study investigates the impact of unified clustering priors on feature learning in self-supervised learning and proposes the PMSN method, which changes the prior to a long-tail distribution to improve the representation quality of imbalanced datasets.
The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
Mor Shpigel Nacson (Technion), Daniel Soudry (Technion)
OptimizationTabular
🎯 What it does: This study investigates the implicit bias of the stable optimal solutions that stochastic gradient descent converges to when training single hidden layer multivariate ReLU networks. It provides a smoothness upper bound for stable solutions and proves theoretical results regarding deep separation and the approximation of smooth functions.
The In-Sample Softmax for Offline Reinforcement Learning
Chenjun Xiao (Huawei Noah's Ark Lab), Martha White (University of Alberta)
Reinforcement Learning
🎯 What it does: An offline reinforcement learning In-sample Actor-Critic (InAC) algorithm is proposed, which utilizes actions that appear in the dataset for softmax bootstrapping, avoiding overestimation of out-of-distribution actions.
The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks
Blake Bordelon (Harvard University), Cengiz Pehlevan (Harvard University)
Representation LearningTabular
🎯 What it does: The theoretical analysis of the learning dynamics and feature evolution of infinitely wide deep networks under different learning rules.
The KFIoU Loss for Rotated Object Detection
Xue Yang (COWAROBOT Co. Ltd.), Qi Tian (Huawei Cloud)
Object DetectionAutonomous DrivingImagePoint Cloud
🎯 What it does: A new rotation target detection loss function KFIoU is proposed to approximate the difficult-to-derive SkewIoU, while maintaining consistency with conventional regression losses, thereby improving detection accuracy.
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
Zonglin Li (Google Research), Sanjiv Kumar (Google Research)
Computational EfficiencyTransformerImageText
🎯 What it does: This paper studies the sparsity phenomenon of intermediate activation layers in the Transformer model after training and proposes an improved architecture to explicitly control sparsity through Top-k thresholding of activations.
The Lie Derivative for Measuring Learned Equivariance
Nate Gruver (New York University), Andrew Gordon Wilson (New York University)
Convolutional Neural NetworkTransformerImageBiomedical Data
🎯 What it does: This paper systematically evaluates the equivariance of hundreds of pre-trained visual models (CNN, ViT, Mixer) using Lie derivatives and hierarchical decomposition.
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
Zihui Xue (University of Texas at Austin), Hang Zhao (Tsinghua University)
Knowledge DistillationMultimodality
🎯 What it does: This study investigates the cross-modal knowledge distillation mechanism, proposing the Modal Venn Diagram (MVD) and the Modal Focus Hypothesis (MFH) to explain and predict the effects of cross-modal KD.
The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
Alexander Atanasov (Harvard University), Cengiz Pehlevan (Harvard University)
Tabular
🎯 What it does: This paper experimentally studies the learning curve of ReLU networks on polynomial regression tasks, exploring the starting point of the transition from infinite width behavior to the variance-limited stage in finite width networks and the impact of feature learning.
The Power of Regularization in Solving Extensive-Form Games
Mingyang Liu (Tsinghua University), Kaiqing Zhang (University of Maryland)
Optimization
🎯 What it does: This paper proposes two types of algorithms: Regularized Dilation Optimistic Mirror Descent (Reg-DOMD) and Regularized Counterfactual Regret Minimization (Reg-CFR) to find Nash equilibria in Extensive Form Games (EFG), and provides theoretical proofs for the terminal iteration convergence rate and average iteration convergence rate.
The Provable Benefit of Unsupervised Data Sharing for Offline Reinforcement Learning
Hao Hu (Tsinghua University), Chongjie Zhang (Tsinghua University)
Reinforcement Learning
🎯 What it does: A provable data sharing (PDS) offline reinforcement learning algorithm is proposed, which utilizes reward-free data to maintain conservativeness by adding uncertainty penalties to the learned reward function, thereby achieving provable sharing and utilization of offline datasets.
The Role of Coverage in Online Reinforcement Learning
Tengyang Xie (University of Illinois at Urbana-Champaign), Sham M. Kakade (Harvard University)
Reinforcement Learning
🎯 What it does: The paper theoretically explores the relationship between coverage conditions in offline RL and exploration in online RL, proving that the existence of a coverable distribution is sufficient to achieve sample efficiency in online RL, and provides corresponding upper bounds; it also demonstrates that several weak coverage conditions are insufficient for online RL; a new complexity measure, the Sequential Extrapolation Coefficient, is proposed to unify coverability and the Bellman-Eluder dimension.
The Role of ImageNet Classes in Fréchet Inception Distance
Tuomas Kynkäänniemi (Aalto University), Jaakko Lehtinen (Aalto University)
GenerationData SynthesisConvolutional Neural NetworkGenerative Adversarial NetworkContrastive LearningImage
🎯 What it does: This paper conducts an in-depth analysis of the Fréchet Inception Distance (FID), revealing its high sensitivity to ImageNet categories, and demonstrates that significantly reducing FID through matching Top-N classification histograms or optimizing sampling weights can be achieved without improving the perceived quality of generated images; experiments with FastGAN and StyleGAN2 show that this 'perceptual space' is easily exploitable.
The Surprising Computational Power of Nondeterministic Stack RNNs
Brian DuSell (University of Notre Dame), David Chiang (University of Notre Dame)
RecognitionRecurrent Neural NetworkText
🎯 What it does: This paper proposes and experiments with a differentiable non-deterministic stack RNN (RNS-RNN) and its vector stack version (VRNS-RNN), demonstrating its ability to recognize all context-free languages, their intersections, and some non-context-free languages, and conducts natural language modeling experiments on the Penn Treebank.
The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry
Dian Wang (Northeastern University), Robert Platt (Northeastern University)
Robotic IntelligenceConvolutional Neural NetworkReinforcement LearningImageVideo
🎯 What it does: In scenarios where the underlying symmetry is unknown or broken, this study investigates and verifies that using extrinsic equivariance constraints that do not perfectly match the true symmetry can enhance the performance of supervised learning and reinforcement learning models.
The Symmetric Generalized Eigenvalue Problem as a Nash Equilibrium
Ian Gemp (DeepMind), Brian McWilliams (Google Research)
OptimizationComputational EfficiencyMultimodality
🎯 What it does: This paper proposes a method to express the Symmetric Generalized Eigenvalue Problem (SGEP) as a Nash equilibrium game, and based on this, designs a parallel and online γ-EigenGame algorithm that can solve large-scale SGEP with a time complexity of O(dk), suitable for streaming data.
The Tilted Variational Autoencoder: Improving Out-of-Distribution Detection
Griffin Floto (University of Toronto), Mihai Nica (University of Guelph)
Anomaly DetectionAuto EncoderImage
🎯 What it does: Proposed and implemented a tilted Gaussian prior as a substitute for the standard Gaussian, used in variational autoencoders to improve the distribution coverage of high-dimensional latent space and enhance anomaly detection performance.
The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning
Zhenmei Shi (University of Wisconsin Madison), Somesh Jha (University of Wisconsin Madison)
Representation LearningContrastive LearningImage
🎯 What it does: This study investigates the trade-off between the generalizability (applicability to multiple tasks) and label efficiency (rapid convergence with a small amount of labeled data) of contrastive learning pre-trained representations, and proposes a method of fine-tuning through contrastive regularization to alleviate this trade-off.
Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning
Peizhong Ju (Ohio State University), Ness Shroff
Meta LearningTabular
🎯 What it does: This paper studies the generalization performance of MAML (Meta-Learning) trained with more model parameters (over-parameterization) under linear regression models, focusing on interpolation solutions with zero training error, and provides a non-asymptotic high-probability error upper bound.
This Looks Like It Rather Than That: ProtoKNN For Similarity-Based Classifiers
Yuki Ukai (GLORY Ltd.), Hironobu Fujiyoshi (Chubu University)
ClassificationExplainability and InterpretabilityImage
🎯 What it does: This paper proposes ProtoKNN, a model that extends ProtoPNet to use KNN as a similarity-based classifier while maintaining the interpretability of case-based reasoning.
TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization
Xiang Li (ETH Zurich), Niao He (ETH Zurich)
OptimizationGenerative Adversarial NetworkImage
🎯 What it does: This paper proposes a single-loop adaptive gradient ascent algorithm, TiAda, for solving non-convex-strongly concave minimax optimization problems, which automatically adjusts the step size ratio between primal and dual variables to achieve time-scale adaptation.
Tier Balancing: Towards Dynamic Fairness over Underlying Causal Factors
Zeyu Tang (Carnegie Mellon University), Kun Zhang (Mohamed bin Zayed University of Artificial Intelligence)
TabularFinance Related
🎯 What it does: Proposes and analyzes the dynamic fairness concept of Tier Balancing based on causal graphs, studying the interaction between decision-making and data generation processes to achieve long-term fairness;
TILP: Differentiable Learning of Temporal Logical Rules on Knowledge Graphs
Siheng Xiong (Georgia Institute of Technology), James Clayton Kerce (Georgia Institute of Technology)
Recurrent Neural NetworkGraph Neural NetworkGraphTime Series
🎯 What it does: A differentiable temporal knowledge graph rule learning framework TILP is proposed, which learns temporal logic rules using constrained random walks and temporal feature modeling for link prediction.
Time to augment self-supervised visual representation learning
Arthur Aubret (Clermont Auvergne University), Jochen Triesch (Frankfurt Institute for Advanced Studies)
ClassificationRecognitionRepresentation LearningContrastive LearningImageVideo
🎯 What it does: The system studies the impact of time augmentation generated by natural interactions on self-supervised visual representation learning, and validates it in both 3D simulation environments and real video data.
Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection
Jinhyung Park (Carnegie Mellon University), Wei Zhan (University of California, Berkeley)
Object DetectionAutonomous DrivingConvolutional Neural NetworkGaussian SplattingVideoPoint CloudBenchmark
🎯 What it does: The SOLOFusion framework is proposed, which combines long-term low-resolution and short-term high-resolution multi-view fusion for 3D object detection using only cameras, along with theoretical analysis and baseline implementation.
TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis
Haixu Wu (Tsinghua University), Mingsheng Long (Tsinghua University)
Anomaly DetectionConvolutional Neural NetworkRecurrent Neural NetworkTransformerTime Series
🎯 What it does: A general time series analysis framework called TimesNet is proposed, which is based on multi-periodicity, converting one-dimensional time series into two-dimensional tensors and using TimesBlock for multi-scale 2D transformations.
Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
David Henry Mguni (Huawei Noah's Ark Lab), Jun Wang
Autonomous DrivingOptimizationReinforcement LearningTime SeriesSequentialFinance Related
🎯 What it does: A learnable impact control reinforcement learning framework, LICRA, is proposed, which can simultaneously learn when to take actions and what actions to take, in order to reduce the cost incurred by each action.
Toeplitz Neural Network for Sequence Modeling
Zhen Qin (SenseTime Research), Yiran Zhong (Shanghai AI Laboratory)
TransformerTextSequential
🎯 What it does: This paper proposes a sequence modeling network based on Toeplitz matrices (TNN), which achieves token mixing through relative position encoding, eliminates the quadratic complexity of the attention matrix, and enables efficient processing of long sequences.
Token Merging: Your ViT But Faster
Daniel Bolya (Georgia Tech), Judy Hoffman (Georgia Tech)
ClassificationComputational EfficiencyTransformerImageVideoMultimodalityAudio
🎯 What it does: The Token Merging (ToMe) method is introduced in Vision Transformer (ViT), which reduces computational load and improves throughput by merging similar tokens in each Transformer block.
Topology-aware Robust Optimization for Out-of-Distribution Generalization
Fengchun Qiao (University of Delaware), Xi Peng (University of Delaware)
Domain AdaptationOptimizationImageTime Series
🎯 What it does: This paper proposes Topology-Aware Robust Optimization (TRO), which combines distribution topology learning with robust optimization to address the out-of-distribution (OOD) generalization problem.
Toward Adversarial Training on Contextualized Language Representation
Hongqiu Wu (Shanghai Jiao Tong University), Min Zhang (Soochow University)
OptimizationAdversarial AttackTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper studies adversarial training (AT) on pre-trained language models (PLM) and finds that traditional AT mainly perturbs the decoder with limited impact on the encoder. It subsequently proposes a new adversarial training method—Contextualized representation-Adversarial Training (CreAT), which aims to maximize the differences in contextualized representations, thereby achieving global worst-case optimization for the entire model. CreAT is applied in both the model pre-training and fine-tuning stages.
Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism
Zhijian Zhuo (Peking University), Yisen Wang (Peking University)
Representation LearningConvolutional Neural NetworkContrastive LearningImage
🎯 What it does: The Rank Differential Mechanism (RDM) is proposed to provide a unified explanation of why different asymmetric designs in non-contrastive learning can avoid feature collapse, and based on this theory, new low-pass/high-pass filters are designed to achieve self-supervised representation learning.
Towards Addressing Label Skews in One-Shot Federated Learning
Yiqun Diao (National University of Singapore), Bingsheng He (National University of Singapore)
ClassificationFederated LearningKnowledge DistillationImage
🎯 What it does: This paper proposes FedOV, a single-round federated learning framework designed for label skew, which utilizes open set voting to address the misclassification of unseen classes by local models.
Towards Better Selective Classification
Leo Feng (Mila Université de Montréal and Borealis AI), Amir H. Abdi
ClassificationSupervised Fine-TuningImage
🎯 What it does: This paper studies the selective classification problem and proposes a Softmax Response selection mechanism based on the classifier itself and entropy regularization to improve model accuracy.
Towards convergence to Nash equilibria in two-team zero-sum games
Fivos Kalogiannis (University of California Irvine), Emmanouil-Vasileios Vlatakis-Gkaragkounis
GenerationOptimizationReinforcement LearningGenerative Adversarial NetworkImage
🎯 What it does: This paper studies the Nash equilibrium in two-player zero-sum games, proving its computational difficulty and demonstrating that commonly used gradient methods cannot converge in this type of game. It then proposes a KPV-GDA algorithm based on control theory, which can locally converge to Nash equilibrium under certain conditions, and experiments are conducted on multi-generative adversarial network (multi-GAN) tasks for validation.
Towards Effective and Interpretable Human-Agent Collaboration in MOBA Games: A Communication Perspective
Yiming Gao (Tencent AI Lab), Wei Liu (Tencent AI Lab)
Explainability and InterpretabilityRobotic IntelligenceConvolutional Neural NetworkReinforcement LearningAgentic AIVideo
🎯 What it does: The study implements human-agent collaboration in MOBA games and proposes a Meta-Command Communication-based Framework (MCC) based on interpretable macro commands, enabling collaboration between humans and agents through a unified macro command.
Towards Inferential Reproducibility of Machine Learning Research
Michael Hagmann (Heidelberg University), Stefan Riezler (Heidelberg University)
Hyperparameter SearchTransformerText
🎯 What it does: In response to the randomness and unmeasurable noise in machine learning experiments, a statistical analysis framework based on Linear Mixed Effects Model (LMEM) and Generalized Likelihood Ratio Test (GLRT) is proposed to evaluate the significance of model performance differences, variance components, and reliability.
Towards Interpretable Deep Reinforcement Learning with Human-Friendly Prototypes
Eoin M. Kenny (Massachusetts Institute of Technology), Julie Shah
Explainability and InterpretabilityReinforcement Learning
🎯 What it does: This paper proposes the Prototype-Wrapper Network (PW-Net), an 'explainable by design' method that allows any pre-trained deep reinforcement learning (RL) agent to make decisions using human-friendly prototypes.
Towards Lightweight, Model-Agnostic and Diversity-Aware Active Anomaly Detection
Xu Zhang (Microsoft Research), Dongmei Zhang (Microsoft Azure)
Anomaly DetectionTabular
🎯 What it does: This paper proposes a lightweight, model-agnostic active anomaly detection method called LMADA, aimed at more effectively utilizing expert feedback to enhance anomaly detection accuracy.
Towards Minimax Optimal Reward-free Reinforcement Learning in Linear MDPs
Pihe Hu (Tsinghua University), Longbo Huang (Tsinghua University)
Reinforcement Learning
🎯 What it does: A reward-free reinforcement learning algorithm named LSVI-RFE is proposed for linear MDPs (Markov Decision Processes), which does not use reward signals during the exploration phase and outputs an ε-approximately optimal policy during the planning phase.
Towards One-shot Neural Combinatorial Solvers: Theoretical and Empirical Notes on the Cardinality-Constrained Case
Runzhong Wang (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
OptimizationRecurrent Neural NetworkGraph Neural NetworkReinforcement LearningGraphTabularFinance Related
🎯 What it does: A differentiable cardinality-constrained neural network (CardNN) is designed to solve combinatorial optimization problems in one go through Sinkhorn or Gumbel-Sinkhorn layers, achieving end-to-end learning for prediction-optimization.
Towards Open Temporal Graph Neural Networks
Kaituo Feng (Beijing Institute of Technology), JUN ZHOU
Graph Neural NetworkGraphTime Series
🎯 What it does: This paper proposes OTGNet, a graph neural network framework for open temporal graphs, addressing the issues of heterogeneous propagation and catastrophic forgetting as node categories expand over time.
Towards Robust Object Detection Invariant to Real-World Domain Shifts
Qi Fan (Hong Kong University of Science and Technology), Dengxin Dai (Max Planck Institute for Informatics)
Object DetectionDomain AdaptationAutonomous DrivingConvolutional Neural NetworkImage
🎯 What it does: This study proposes a 'Normalization Perturbation (NP)' mechanism that synthesizes potential domain styles by randomly perturbing the channel statistics of shallow CNN features during single-source domain training, thereby enhancing the robustness of object detection under real-world domain shifts.
Towards Robustness Certification Against Universal Perturbations
Yi Zeng (Virginia Tech), Ruoxi Jia (Virginia Tech)
OptimizationAdversarial AttackConvolutional Neural NetworkAuto EncoderImage
🎯 What it does: This paper studies methods for the robustness certification of Universal Perturbation (UP) in neural networks, proposing a combined framework of linear relaxation and integer linear programming, providing an optimal lower bound for UP, and offering error estimates for the entire data distribution.
Towards Smooth Video Composition
Qihang Zhang (Chinese University of Hong Kong), Bolei Zhou (University of California Los Angeles)
GenerationData SynthesisAutonomous DrivingGenerative Adversarial NetworkVideo
🎯 What it does: Utilizing an improved GAN architecture to achieve smooth transitions between video frames and generate infinitely long video sequences.
Towards Stable Test-time Adaptation in Dynamic Wild World
Shuaicheng Niu (South China University of Technology), Mingkui Tan (South China University of Technology)
Domain AdaptationOptimizationImage
🎯 What it does: This study improves the stability of Test-Time Adaptation (TTA) in wild testing scenarios (mixed perturbations, small batch sizes, online label imbalance) by proposing a Sharpness-Aware and Reliable Entropy Minimization (SAR) method, which can filter out high-gradient noise samples and approach flat minima, thereby suppressing model collapse and enhancing robustness.
Towards the Generalization of Contrastive Self-Supervised Learning
Weiran Huang (Shanghai Jiao Tong University), Zihao Jiang (Shanghai Jiao Tong University)
ClassificationRepresentation LearningConvolutional Neural NetworkContrastive LearningImage
🎯 What it does: Proposed (σ,δ)-augmentation to quantify data augmentation, provided a generalization error upper bound for contrastive self-supervised learning, and proved that InfoNCE and cross-correlation loss satisfy alignment and center separation conditions; experimentally verified the positive correlation between augmentation concentration and downstream performance.
Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning
Yujun Shi (National University of Singapore), Song Bai (ByteDance Inc.)
Federated LearningConvolutional Neural NetworkImage
🎯 What it does: Analyzes the phenomenon of representation dimension collapse caused by data heterogeneity in federated learning, provides a theoretical explanation, and proposes the addition of feature decorrelation regularization (FEDDECORR) during the local training phase to alleviate this issue.
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Zeyuan Allen-Zhu (Meta), Yuanzhi Li (Mohamed bin Zayed University of Artificial Intelligence)
OptimizationKnowledge DistillationConvolutional Neural NetworkTabular
🎯 What it does: This paper theoretically proves that under multi-view data distribution, a single deep network may still perform poorly on the test set after perfectly fitting the training set, while averaging the outputs of multiple homogeneous networks (ensemble) or using knowledge distillation can significantly improve test accuracy; at the same time, self-distillation can also bring some improvement.
Towards Understanding GD with Hard and Conjugate Pseudo-labels for Test-Time Adaptation
Jun-Kun Wang (Yale University), Andre Wibisono (Yale University)
Domain AdaptationOptimizationTabular
🎯 What it does: The study investigates the adaptive theoretical properties of gradient descent (GD) in an unlabeled test domain, comparing the convergence behavior of hard labels and conjugate labels under different loss functions.
Towards Understanding Why Mask Reconstruction Pretraining Helps in Downstream Tasks
Jiachun Pan (Sea AI Lab), Shuicheng YAN
ClassificationObject DetectionConvolutional Neural NetworkAuto EncoderImage
🎯 What it does: This paper provides a theoretical analysis of Mask-Reconstruction Pretraining (MRP), clarifying how it learns semantic features during the pretraining phase and demonstrating its superiority over zero-shot supervised learning in downstream classification tasks.
Trading Information between Latents in Hierarchical Variational Autoencoders
Tim Z. Xiao (University of Tuebingen), Robert Bamler (University of Tuebingen)
GenerationOptimizationRepresentation LearningAuto EncoderImage
🎯 What it does: This paper proposes a framework for information exchange among latent variables at different layers in Hierarchical Variational Autoencoders (HVAE), allowing for separate control of the bit rate at each layer, thus enabling customized optimization for different tasks (reconstruction, generation, representation learning).
Trainability Preserving Neural Pruning
Huan Wang (Northeastern University), Yun Fu (AInnovation Labs, Inc.)
OptimizationConvolutional Neural NetworkImage
🎯 What it does: A structured pruning method based on preserving network trainability is proposed—Trainability Preserving Pruning (TPP). It maintains the trainability of the network during the pruning process through decorrelation regularization of the Gram matrices of the filters to be pruned and the filters to be retained, as well as regularization of the batch normalization parameters.
Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions
Tao Li (Shanghai Jiao Tong University), Xiaolin Huang (Shanghai Jiao Tong University)
OptimizationComputational EfficiencyConvolutional Neural NetworkImage
🎯 What it does: Proposes a Trainable Weight Averaging (TWA) method that optimizes historical model weights in a low-dimensional subspace during the early stages of training to achieve more efficient training and better generalization performance.
Training language models to summarize narratives improves brain alignment
Khai Loong Aw (Max Planck Institute for Software Systems), Mariya Toneva (Max Planck Institute for Software Systems)
TransformerSupervised Fine-TuningTextBiomedical DataMagnetic Resonance Imaging
🎯 What it does: This paper explores the correlation between the deep understanding ability of language models and human brain representations by fine-tuning four baseline Transformer models (BART, LED, BigBird, LongT5) on the long narrative summarization task (BookSum) and comparing the fine-tuned models with baseline models in terms of fMRI brain activity while reading Chapter 9 of Harry Potter.
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
Weixi Feng (University of California), William Yang Wang (Google)
GenerationData SynthesisTransformerDiffusion modelImageText
🎯 What it does: A training-free, structure-based cross-attention guidance method (StructureDiffusion) is proposed, which significantly enhances the quality of attribute binding and multi-object composition by encoding noun phrases from syntactic trees or scene graphs and dynamically replacing values in the cross-attention layer of Stable Diffusion.
Transfer Learning with Deep Tabular Models
Roman Levin (University of Washington), Micah Goldblum (New York University)
ClassificationDomain AdaptationHyperparameter SearchTransformerSupervised Fine-TuningContrastive LearningTabularBiomedical DataElectronic Health Records
🎯 What it does: This paper studies transfer learning using deep tabular models under limited downstream data conditions and proposes a pseudo-feature method to address the feature space mismatch problem.
Transfer NAS with Meta-learned Bayesian Surrogates
Gresa Shala (University of Freiburg), Josif Grabocka (University of Freiburg)
OptimizationMeta LearningNeural Architecture SearchGraph Neural NetworkTransformerImage
🎯 What it does: A transferable Bayesian optimization method is proposed for Neural Architecture Search (NAS), utilizing a meta-learning deep kernel Gaussian process as a surrogate to automatically learn a joint embedding of network architecture and dataset features, and quickly search for the optimal architecture through Bayesian optimization.
Transferable Unlearnable Examples
Jie Ren (Michigan State University), Jiliang Tang (Michigan State University)
Adversarial AttackContrastive LearningImage
🎯 What it does: A transferable no-learning sample method (TUE) is proposed, which invalidates personal data in unauthorized training by generating perturbations.
Transformer Meets Boundary Value Inverse Problems
Ruchi Guo (University of California), Long Chen (University of California)
TransformerImagePhysics Related
🎯 What it does: A deep direct sampling method based on Transformer and PDE feature mapping is proposed for real-time reconstruction of the electrical impedance tomography (EIT) inverse problem.
Transformer-based model for symbolic regression via joint supervised learning
Wenqiang Li (Chinese Academy of Sciences), Songsong Tian (Chinese Academy of Sciences)
TransformerContrastive LearningTabular
🎯 What it does: A Transformer-based symbolic regression model is proposed, using a residual MLP as a feature extractor, and enhancing the recovery rate of expression skeletons through joint supervised learning (cross-entropy + contrastive learning).
Transformer-based World Models Are Happy With 100k Interactions
Jan Robine (Technical University of Dortmund), Stefan Harmeling (Technical University of Dortmund)
TransformerReinforcement LearningWorld ModelSequentialBenchmark
🎯 What it does: Construct a Transformer-XL based autoregressive world model (TWM) that utilizes imagined generated trajectories for training strategies, achieving efficient learning within 100,000 interactions on the Atari 100k benchmark.
Transformer-Patcher: One Mistake Worth One Neuron
Zeyu Huang (Beihang University), Zhang Xiong (Beihang University)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: To address the immediate correction of errors in large pre-trained language models (PLMs) that have already been deployed, we propose the Sequential Model Editing (SME) task and introduce the Transformer-Patcher method, which adds and trains a small number of learnable neurons (patches) in the FFN of the last layer of the Transformer to fix errors.
Transformers are Sample-Efficient World Models
Vincent Micheli (University of Geneva), François Fleuret (University of Geneva)
TransformerReinforcement LearningAuto EncoderWorld ModelImage
🎯 What it does: Under the condition of very limited real interaction experience, the IRIS agent is proposed, which compresses images into tokens using a discrete autoencoder, and then performs autoregressive prediction with a GPT-style Transformer to train policies in its 'imagined' world model, achieving efficient reinforcement learning.
Transformers Learn Shortcuts to Automata
Bingbin Liu (Carnegie Mellon University), Cyril Zhang (Microsoft Research)
TransformerSequential
🎯 What it does: This study investigates how Transformers can simulate state transitions of finite state machines (semiautomata) with a small number of layers (far fewer than the sequence length).
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation
Rongjie Huang (Zhejiang University), Zhou Zhao (Zhejiang University)
RecognitionComputational EfficiencyKnowledge DistillationTransformerSupervised Fine-TuningMultimodalityAudio
🎯 What it does: The TranSpeech model is proposed, which combines bidirectional perturbation (BiP) to address the multimodal problem of speech and achieves non-autoregressive speech-to-speech translation.
Treeformer: Dense Gradient Trees for Efficient Attention Computation
Lovish Madaan (Google Research), Prateek Jain (Google Research)
OptimizationComputational EfficiencyTransformerSupervised Fine-TuningText
🎯 What it does: Proposes the TREEFORMER architecture, which utilizes decision trees to achieve efficient attention, divided into two mechanisms: dense TF-ATTENTION and coarse-grained TC-ATTENTION, and achieves end-to-end optimization through bootstrapping training;
TrojText: Test-time Invisible Textual Trojan Insertion
Qian Lou (University of Central Florida), Bo Feng (Meta Platforms, Inc.)
Adversarial AttackTransformerText
🎯 What it does: This paper proposes a test-time executable text backdoor attack method called TrojText, which can make trigger sentences recognized as the target category by performing a small number of bit flips on the model weights without training data.
Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders
Huangjie Zheng (University of Texas at Austin), Mingyuan Zhou (University of Texas at Austin)
GenerationData SynthesisDiffusion modelAuto EncoderGenerative Adversarial NetworkImage
🎯 What it does: This paper proposes a Truncated Diffusion Probability Model (TDPM), which accelerates sampling by truncating the forward diffusion chain and learning an implicit prior, while maintaining or even improving generation quality.
Truthful Self-Play
Shohei Ohsawa (Daisy AI)
OptimizationRecurrent Neural NetworkReinforcement LearningSequential
🎯 What it does: A framework called Truthful Self-Play (TSP) is proposed, utilizing mechanism design and peer-prediction methods to achieve unsupervised and unbiased state representation in partially observable multi-agent environments.
TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation
Hyesu Lim (Qualcomm AI Research), Sungha Choi (Qualcomm AI Research)
ClassificationSegmentationDomain AdaptationImage
🎯 What it does: A domain shift-aware batch normalization layer TTN is proposed for adaptation during testing, dynamically balancing the statistics of the source domain with the current batch statistics.
Tuning Frequency Bias in Neural Network Training with Nonuniform Data
Annan Yu (Cornell University), Alex Townsend (Cornell University)
Auto EncoderImage
🎯 What it does: For non-uniform training data, it is proven that neural network training still exhibits frequency bias, and this bias is adjusted by introducing data-related quadrature rules and Sobolev norm loss.
Turning the Curse of Heterogeneity in Federated Learning into a Blessing for Out-of-Distribution Detection
Shuyang Yu (Michigan State University), Jiayu Zhou (Michigan State University)
Anomaly DetectionFederated LearningImage
🎯 What it does: A new anomaly detection method in federated learning called FOSTER is proposed, which utilizes data heterogeneity to synthesize virtual external class anomaly samples, thereby improving anomaly detection performance.
TVSPrune - Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning
Chaitanya Murti (Indian Institute of Science), Chiranjib Bhattacharyya (Indian Institute of Science)
CompressionOptimizationConvolutional Neural NetworkImage
🎯 What it does: A data-free structured pruning method called TVSPrune is proposed, which identifies and removes non-discriminative filters by utilizing the total variation distance of the class conditional distribution of the convolutional layer outputs, without the need for retraining or accessing the original data.
TypeT5: Seq2seq Type Inference using Static Analysis
Jiayi Wei (University of Texas at Austin), Isil Dillig (University of Texas at Austin)
AI Code AssistantTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Using CodeT5 for seq2seq code completion to automatically predict missing type annotations in Python/JavaScript programs.
UL2: Unifying Language Learning Paradigms
Yi Tay (Google Research), Donald Metzler (Google Research)
GenerationRetrievalTransformerLarge Language ModelTextChain-of-Thought
🎯 What it does: This study proposes the UL2 (Unifying Language Learning Paradigms) framework, which combines various self-supervised denoising tasks (R-Denoiser, S-Denoiser, X-Denoiser) through a Mixture-of-Denoisers (MoD) mixed pre-training objective, and introduces a mode switching mechanism, allowing a single model to perform consistently across various NLP tasks (from text generation to reasoning, question answering, classification, etc.).
Unbiased Stochastic Proximal Solver for Graph Neural Networks with Equilibrium States
Mingjie Li (Peking University), Zhouchen Lin (Peking University)
OptimizationComputational EfficiencyGraph Neural NetworkGraph
🎯 What it does: Two unbiased stochastic proximal solvers (USP and USP-VR) are proposed for the graph balancing model, significantly reducing the computational complexity of full graph aggregation.
Unbiased Supervised Contrastive Learning
Carlo Alberto Barbano (University of Turin), Pietro Gori (Telecom Paris)
Representation LearningContrastive LearningImageBenchmark
🎯 What it does: The study investigates how to learn unbiased contrastive learning representations in the presence of data bias, proposing a new ε-SupInfoNCE contrastive loss and a FairKL debiasing regularization term.
Understanding and Adopting Rational Behavior by Bellman Score Estimation
Kuno Kim (Stanford University), Stefano Ermon (Stanford University)
Reinforcement LearningContrastive Learning
🎯 What it does: This paper proposes and implements the concept of Bellman Score, using it to estimate reward gradients, achieving model-independent Score learning in high-dimensional environments, and applying it to maximum entropy inverse reinforcement learning, behavior imitation, counterfactual prediction, and cross-environment policy transfer.
Understanding DDPM Latent Codes Through Optimal Transport
Valentin Khrulkov (Yandex), Ivan Oseledets (Skolkovo Institute of Science and Technology)
GenerationData SynthesisDiffusion modelImageOrdinary Differential Equation
🎯 What it does: This study investigates the encoder mapping of DDPM (especially DDIM) and proves that this mapping is completely consistent with the optimal transport mapping under a multivariate Gaussian distribution. Subsequently, numerical experiments validate its approximate consistency on synthetic distributions and image data.
Understanding Edge-of-Stability Training Dynamics with a Minimalist Example
Xingyu Zhu (Duke University), Rong Ge (Duke University)
Optimization
🎯 What it does: This paper studies the behavior of deep neural networks under the training dynamics of 'stability edge' (EoS) by constructing a simple objective function to analyze its training dynamics.
Understanding Embodied Reference with Touch-Line Transformer
Yang Li (Institute for AI Industry Research), Yixin Zhu (Institute for Artificial Intelligence)
Object DetectionTransformerVision Language ModelImageTextMultimodality
🎯 What it does: This study investigates how to accurately locate the referent in contexts involving gestures and verbal expressions (Embodied Reference Understanding) and proposes the Touch-Line Transformer model to accomplish this task.
Understanding Influence Functions and Datamodels via Harmonic Analysis
Nikunj Saunshi (Princeton University), Sanjeev Arora (Princeton University)
OptimizationImage
🎯 What it does: This paper constructs a unified theoretical framework through harmonic analysis to explain the properties of influence functions and linear data models in deep learning, and proposes a method to efficiently estimate linear fitting residuals without training the model.
Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles
Martin Bjerke (University of Cambridge), Benjamin Adric Dunn
OptimizationRepresentation LearningConvolutional Neural NetworkAuto EncoderTabularTime Series
🎯 What it does: This paper proposes a differentiable neural latent variable model (faeLVM) that learns low-dimensional latent variables of neuronal populations through feature sharing (shared tuning curves) and cluster detection (soft clustering), achieving separation and tuning curve fitting of different neural groups within the same framework.
Understanding new tasks through the lens of training data via exponential tilting
Subha Maity (University of Michigan), Yuekai Sun (University of Michigan)
Domain AdaptationImageBenchmark
🎯 What it does: Proposes a method to assign weights to training samples through an exponential tilting model to approximate the target domain distribution, thereby achieving model evaluation, fine-tuning, and model selection in the unlabeled target domain.
Understanding the Covariance Structure of Convolutional Filters
Asher Trockman (Carnegie Mellon University), J Zico Kolter
Convolutional Neural NetworkImage
🎯 What it does: This study investigates the covariance structure of large filters in depthwise separable convolutions and proposes a no-learning initialization method based on closed-form multivariate Gaussian distribution.
Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
Difan Zou (University of Hong Kong), Quanquan Gu (University of California)
OptimizationConvolutional Neural NetworkImage
🎯 What it does: The theoretical study of the generalization gap between Adam and Gradient Descent (GD) in non-convex learning tasks with weight decay is conducted, and an analysis of their behavior in different learning stages (pattern learning and regularization stage) is provided.
Understanding The Robustness of Self-supervised Learning Through Topic Modeling
Zeping Luo (Duke University), Rong Ge (Duke University)
Data SynthesisRepresentation LearningContrastive LearningText
🎯 What it does: By applying self-supervised learning on the generated data of topic models, it is demonstrated that reconstruction and contrastive learning objectives can recover posterior information of document topics, thereby constructing useful document representations.
Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning
Yuandong Tian (Meta AI)
OptimizationRepresentation LearningRecurrent Neural NetworkContrastive LearningSequential
🎯 What it does: Analyzes the role of nonlinear activation in the dynamics of contrastive learning training, revealing its mechanisms for producing local optima and promoting diverse feature learning, and validates the theory through experiments with a 2-layer network.
Understanding Train-Validation Split in Meta-Learning with Neural Networks
Xinzhe Zuo (University of California), Quanquan Gu (University of California)
ClassificationMeta LearningConvolutional Neural NetworkImage
🎯 What it does: Theoretical and experimental research on First-Order MAML (FOMAML) in binary classification tasks with high noise and limited samples, exploring the impact of training/validation data splitting on meta-learning performance in neural networks.
Understanding weight-magnitude hyperparameters in training binary networks
Joris Quist (Delft University of Technology), Jan van Gemert (Delft University of Technology)
OptimizationHyperparameter SearchConvolutional Neural NetworkReinforcement LearningImage
🎯 What it does: Reformulate the training process of binary networks from the perspective of gradient filtering, reinterpreting the roles of hyperparameters such as weight decay, learning rate, and its decay in BNNs;
Understanding Why Generalized Reweighting Does Not Improve Over ERM
Runtian Zhai (Carnegie Mellon University), Pradeep Kumar Ravikumar
Domain AdaptationOptimizationConvolutional Neural NetworkImage
🎯 What it does: This paper studies the implicit bias of the Generalized Reweighting (GRW) algorithm in the context of distribution shift for over-parameterized linear models and wide neural networks, and proves that it produces almost the same models as traditional Empirical Risk Minimization (ERM) after convergence, making it difficult to enhance distribution robustness.
Understanding Zero-shot Adversarial Robustness for Large-Scale Models
Chengzhi Mao (Columbia University), Carl Vondrick (Columbia University)
Representation LearningAdversarial AttackTransformerContrastive LearningImageMultimodality
🎯 What it does: The study investigates how to enhance the zero-shot adversarial robustness of CLIP without the need for task labels.
Uni-Mol: A Universal 3D Molecular Representation Learning Framework
Gengmo Zhou (Renmin University of China), Guolin Ke (DP Technology)
Representation LearningDrug DiscoveryTransformerGraph
🎯 What it does: This paper presents Uni-Mol, a unified 3D molecular representation learning framework that can directly take 3D coordinates as input and output.
Unicom: Universal and Compact Representation Learning for Image Retrieval
Xiang An (DeepGlint), Tongliang Liu (University of Sydney)
RetrievalRepresentation LearningTransformerContrastive LearningImage
🎯 What it does: Using the large-scale LAION 400M dataset, we first performed k-means clustering on the visual and textual features from CLIP to generate approximately one million pseudo-classes. We then pre-trained ViT using a softmax loss with random negative classes and random feature subsets, resulting in a general and compressed image representation.
UNICORN: A Unified Backdoor Trigger Inversion Framework
Zhenting Wang (Rutgers University), Shiqing Ma (Rutgers University)
OptimizationAdversarial AttackConvolutional Neural NetworkAuto EncoderImage
🎯 What it does: This paper proposes a unified backdoor trigger inversion framework called UNICORN, which can reconstruct implanted backdoor triggers across different input spaces (pixels, signals, features, numerical) and detect whether the model has been infected by backdoors through trigger inversion.
Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimization
Zonghan Yang (Tsinghua University), Xing Xie (Microsoft Research)
GenerationOptimizationComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: A unified de-biasing and decontamination framework UDDIA is proposed for adaptive optimization during the inference phase.
Unified Discrete Diffusion for Simultaneous Vision-Language Generation
Minghui Hu (Qatar University), Ponnuthurai N. Suganthan (Nanyang Technological University)
GenerationData SynthesisTransformerVision Language ModelDiffusion modelImageTextMultimodality
🎯 What it does: This paper proposes a unified discrete diffusion model called UniD3, which can simultaneously perform joint generation of vision and language as well as bidirectional modality translation.
UNIFIED-IO: A Unified Model for Vision, Language, and Multi-modal Tasks
Jiasen Lu (Allen Institute for AI), Aniruddha Kembhavi (Allen Institute for AI)
ClassificationObject DetectionSegmentationGenerationTransformerPrompt EngineeringVision Language ModelGenerative Adversarial NetworkImageTextMultimodality
🎯 What it does: A unified model called UNIFIED-IO has been developed, capable of simultaneously handling various visual, language, and multimodal tasks, including classic computer vision tasks, vision-and-language tasks, and NLP tasks;