ICML 2024 Papers — Page 17
International Conference on Machine Learning · 2610 papers
On the Calibration of Human Pose Estimation
Kerui Gu (National University of Singapore), Angela Yao (National University of Singapore)
Pose EstimationConvolutional Neural NetworkImage
🎯 What it does: This paper studies the confidence calibration problem in 2D human pose estimation, finding that traditional confidence does not match pose accuracy, and proposes a theoretical calibration formula; subsequently, a lightweight Calibrated ConfidenceNet (CCNet) is designed for post-calibration on pre-trained models.
On the Complexity of Finite-Sum Smooth Optimization under the Polyak–Łojasiewicz Condition
Yunyan Bai (Fudan University), Luo Luo (Fudan University)
OptimizationTabular
🎯 What it does: This paper studies the complexity of finite-sum smooth optimization under the Polyak-Łojasiewicz (PL) condition, providing lower bounds in both single-machine and distributed environments, and proposing upper bound methods that match these lower bounds.
On The Complexity of First-Order Methods in Stochastic Bilevel Optimization
Jeongyeol Kwon (Wisconsin Institute for Discovery), Hanbaek Lyu
Optimization
🎯 What it does: The paper studies the algorithmic complexity of finding ε-saddle points in stochastic bilevel optimization where the lower-level problem is unconstrained and strongly convex, using a y∗-aware or 'lower-level solution aware' first-order oracle that can only obtain local gradient information.
On the Consistency of Kernel Methods with Dependent Observations
Pierre-François Massiani (RWTH Aachen University), Friedrich Solowjow (RWTH Aachen University)
Sequential
🎯 What it does: This paper proposes a new data generation hypothesis called empirical weak convergence (EWC) and proves the consistency of kernel methods (support vector machines, kernel mean embeddings, and conditional kernel mean embeddings) under this hypothesis with respect to observed data.
On the Convergence of Projected Bures-Wasserstein Gradient Descent under Euclidean Strong Convexity
Junyi FAN, Zhengyuan Zhou
Optimization
🎯 What it does: The paper studies the global convergence theory of the Bures-Wasserstein gradient descent algorithm (including the projection version) for positive definite matrices under Euclidean strong convexity and smoothness conditions, and provides a closed-form projection formula for the BW ball constraint.
On the Diminishing Returns of Width for Continual Learning
Etash Kumar Guha, Vihan Lakshman (ThirdAI)
Convolutional Neural NetworkTransformerImage
🎯 What it does: This study investigates the impact of width on catastrophic forgetting in continual learning, proposes a theoretical framework, and demonstrates that increasing width leads to diminishing returns, validating its performance on large-width FFN and WideResNet.
On the Duality Between Sharpness-Aware Minimization and Adversarial Training
Yihao Zhang (Peking University), Zeming Wei (Peking University)
OptimizationAdversarial AttackConvolutional Neural NetworkTransformerImageText
🎯 What it does: This paper proves and experimentally verifies that Sharpness-Aware Minimization (SAM) can enhance the adversarial robustness of models without using adversarial samples, and that the natural accuracy does not decrease.
On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning
Jeongheon Oh (Yonsei University), Kibok Lee (Yonsei University)
Object DetectionRepresentation LearningContrastive LearningImage
🎯 What it does: A model that introduces supervised information in a non-contrastive learning framework—SUPSIAM and SUPBYOL—is proposed, utilizing labels to further enhance representation learning effectiveness.
On the Embedding Collapse when Scaling up Recommendation Models
Xingzhuo Guo (Tsinghua University), Mingsheng Long (Tsinghua University)
Recommendation SystemTabular
🎯 What it does: This study systematically reveals the phenomenon of embedding collapse that occurs in the scaling process of traditional recommendation models, and based on this, proposes a multi-embedding design to alleviate the collapse and enhance model scalability.
On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm
Zhanpeng Zhou (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
ClassificationConvolutional Neural NetworkTransformerSupervised Fine-TuningImageText
🎯 What it does: This paper conducts experiments on pre-trained-fine-tuning models and discovers the Cross-Task Linear phenomenon (CTL), which indicates that models with linear interpolation in the parameter space have internal features that are almost equal to the linear interpolation of the corresponding single model features.
On the Error-Propagation of Inexact Hotelling's Deflation for Principal Component Analysis
Fangshuo Liao (Rice University), Anastasios Kyrillidis (Rice University)
Image
🎯 What it does: This paper explores through mathematical analysis how the error accumulates with the number of iterations when approximately solving principal components in the Hotelling deflation method, and how it affects the accuracy of subsequent principal components.
On the Expressive Power of Spectral Invariant Graph Neural Networks
Bohang Zhang (Peking University), Haggai Maron (NVIDIA Research)
Graph Neural NetworkGraphBenchmark
🎯 What it does: This paper proposes and unifies the spectral invariant graph neural network framework, namely Eigenspace Projection GNN (EPNN), and conducts a theoretical analysis of its expressive power;
On The Fairness Impacts of Hardware Selection in Machine Learning
Sree Harsha Nelaturu (Cohere For AI), Ferdinando Fioretto (University of Virginia)
ClassificationRecognitionConvolutional Neural NetworkImage
🎯 What it does: This study investigates the impact of hardware selection on the fairness and performance of machine learning models, revealing that different GPUs can lead to greater fluctuations in accuracy for minority groups. It proposes a fairness penalty method based on decision boundary distance to mitigate this issue.
On the Feasibility of Single-Pass Full-Capacity Learning in Linear Threshold Neurons with Binary Input Vectors
Ruipeng Liu (Syracuse University), Garrett Ethan Katz (Syracuse University)
Spiking Neural NetworkTabular
🎯 What it does: This study investigates the feasibility of single-pass and full-capacity linear threshold neuron learning rules, focusing on binary input vectors.
On the Generalization of Equivariant Graph Neural Networks
Rafal Karczewski, Vikas Garg
Graph Neural NetworkGraph
🎯 What it does: The research theoretically proves the generalization error upper bound of E(n)-equivariant graph neural networks (EGNN), revealing the key relationship between spectral norm and lower layer weights, and demonstrates that ε-normalization can significantly reduce generalization error.
On the Hardness of Probabilistic Neurosymbolic Learning
Jaron Maene (KU Leuven), Luc De Raedt (Orebro University)
Tabular
🎯 What it does: This paper studies the complexity and feasibility of approximating the gradient of Weighted Model Counting (WMC) in probabilistic neural-symbolic learning.
On the Identifiability of Switching Dynamical Systems
Carles Balsells-Rodas (Imperial College London), Yingzhen Li (University of Michigan)
VideoTime Series
🎯 What it does: This study investigates the identifiability of Markov switching models and switching dynamical systems, providing sufficient conditions for identifiability.
On the Implicit Bias of Adam
Matias D. Cattaneo (Princeton University), Boris Shigida (Princeton University)
OptimizationConvolutional Neural NetworkTransformerImageOrdinary Differential Equation
🎯 What it does: This study derives the second-order continuous piecewise ODE approximations of Adam and RMSProp through backward error analysis, revealing their anti-regularization characteristics regarding the gradient L1 norm under different hyperparameters.
On the Independence Assumption in Neurosymbolic Learning
Emile van Krieken (University of Edinburgh), Antonio Vergari (University of Edinburgh)
OptimizationTabular
🎯 What it does: This paper conducts a theoretical analysis of the commonly used conditional independence assumption in neural symbolic learning, proving that this assumption leads the model to favor deterministic solutions, fails to express uncertainty, and results in a non-convex semantic loss function with discrete local minima in the independent distribution space. By utilizing logical tools (implicational clauses, minimal implicants) and computational homology, a geometric representation (cube set) of all feasible independent distributions is constructed, and conditions for convexity and connectivity are provided. Subsequently, visual experiments are conducted on two examples: traffic lights and MNIST addition, to validate the theoretical conclusions.
On the Last-Iterate Convergence of Shuffling Gradient Methods
Zijian Liu (New York University), Zhengyuan Zhou (Arena Technologies)
Optimization
🎯 What it does: This paper proposes the Randomized Gradient Methods (RR, SO, IG) under the conditions of no strong convexity and general regularization, addressing the convergence rate of the last iteration regarding the objective value gap, thus bridging the gap between theory and practice.
On the Maximal Local Disparity of Fairness-Aware Classifiers
Jinqiu Jin (University of Science and Technology of China), Fuli Feng (University of Science and Technology of China)
ClassificationOptimizationImageTabular
🎯 What it does: A new fairness metric MCDP(ε) is proposed to measure the local maximum distribution differences of classifiers across different groups, along with precise and approximate computation algorithms and a bi-level optimization learning method based on this metric.
On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions
Denys Pushkin (École Polytechnique Fédérale de Lausanne), Emmanuel Abbe (Apple)
Transformer
🎯 What it does: This study investigates the minimum order bias of random feature models (RF) and Transformers in the context of non-Boolean functions' GOTU (generalization to unseen domains), exploring two scenarios: small features and sparse targets. It proves that under the small feature paradigm, RF converges to the lowest order interpolator and discusses the peculiarities of square root unit embeddings.
On the Nonlinearity of Layer Normalization
Yunhao Ni (Beihang University), Lei Huang (Beihang University)
ClassificationConvolutional Neural NetworkTransformerImage
🎯 What it does: This paper analyzes the nonlinearity and expressive power of Layer Normalization (LN) from both theoretical and experimental perspectives, proving that LN is a nonlinear transformation and constructing a network consisting only of LN and linear layers that can achieve perfect classification for samples with any label distribution.
On the Origins of Linear Representations in Large Language Models
Yibo Jiang (University of Chicago), Victor Veitch (University of Chicago)
OptimizationRepresentation LearningTransformerLarge Language ModelText
🎯 What it does: This paper proposes a simple latent variable model that formalizes the next word prediction of large language models (LLMs) and demonstrates that concepts under this model naturally exhibit linear encoding in the representation space.
On the Recoverability of Causal Relations from Temporally Aggregated I.I.D. Data
Shunxing Fan (Mohamed bin Zayed University of Artificial Intelligence), Kun Zhang (Carnegie Mellon University)
TabularTime Series
🎯 What it does: This study investigates the recoverability of non-temporal causal discovery on time-aggregated I.I.D. data, proposing two concepts of consistency: functional consistency and conditional independence consistency.
On the Role of Edge Dependency in Graph Generative Models
Sudhanshu Chanpuriya (University of Illinois Urbana-Champaign), Charalampos Tsourakakis (Boston University)
GenerationData SynthesisGraph Neural NetworkGraph
🎯 What it does: This study investigates the impact of edge dependencies in graph generation models on triangle counting and model overlap, proposes a three-layer nested hierarchical structure, provides theoretical upper bounds, and designs a simple model based on maximum cliques for comparison with deep learning models.
On the sample complexity of conditional independence testing with Von Mises estimator with application to causal discovery
Fateme Jamshidi (Ecole Polytechnique Federale de Lausanne), Negar Kiyavash (Ecole Polytechnique Federale de Lausanne)
Computational EfficiencyTabular
🎯 What it does: This paper studies a conditional independence test based on the Von Mises estimator, proposing a new testing method called VM-CI, and analyzes its sample complexity.
On the Second-Order Convergence of Biased Policy Gradient Algorithms
Siqiao Mu (Northwestern University), Diego Klabjan (Northwestern University)
OptimizationReinforcement Learning
🎯 What it does: A second-order convergence analysis is provided for biased policy gradient algorithms (including both standard and actor-critic), proving that they can escape saddle points and converge to ε-second-order critical points.
On The Statistical Complexity of Offline Decision-Making
Thanh Nguyen-Tang (Johns Hopkins University), Raman Arora (Johns Hopkins University)
OptimizationReinforcement Learning
🎯 What it does: The statistical complexity of offline decision-making is studied, establishing the (near) minimax optimal rates for stochastic contextual multi-armed bandits and Markov decision processes.
On the Tractability of SHAP Explanations under Markovian Distributions
Reda Marzouk (Universite Nantes), Colin de La Higuera
Explainability and InterpretabilityComputational Efficiency
🎯 What it does: This paper studies the computational complexity of the SHAP explanation method under the assumption of Markovian distribution, proving that SHAP scores can be accurately computed in polynomial time for weighted automata, disjoint DNF, and decision tree models, breaking the limitations of the traditional feature independence assumption.
On the Trajectory Regularity of ODE-based Diffusion Sampling
Defang Chen (Zhejiang University), Siwei Lyu (University at Buffalo)
GenerationOptimizationDiffusion modelAuto EncoderImageOrdinary Differential Equation
🎯 What it does: This paper studies the geometric laws of ODE-based diffusion model sampling trajectories and proposes a time scheduling method based on these laws.
On the Unexpected Effectiveness of Reinforcement Learning for Sequential Recommendation
Álvaro Labarca Silva (Pontificia Universidad Catolica de Chile), Rodrigo Toro Icarte (Pontificia Universidad Catolica de Chile)
Recommendation SystemReinforcement LearningSequential
🎯 What it does: Investigate the reasons for the performance improvement of next-item prediction in sequential recommendation using reinforcement learning and verify that simpler auxiliary tasks can replace RL.
On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows
Felix Draxler (Heidelberg University), Ullrich Koethe
Flow-based ModelImage
🎯 What it does: This paper theoretically analyzes the expressive power of volume-preserving and coupling normalization flows, proving that volume-preserving flows are not universal under KL divergence and providing a correction method; at the same time, it constructs an achievable hierarchical training scheme, demonstrating that flows based on affine and stronger coupling functions are universal approximators in distribution.
On the Weight Dynamics of Deep Normalized Networks
Christian H.X. Ali Mehmeti-Göpel (Johannes Gutenberg University), Michael Wand (Johannes Gutenberg University)
OptimizationConvolutional Neural NetworkImageOrdinary Differential Equation
🎯 What it does: This paper establishes a dynamic model of weight gradient magnitude caused by normalization layers, derives the evolution of effective learning rate (ELR) during the training process, and proves that under a constant learning rate, the ELR ratio of all layers ultimately converges to 1.
On Universally Optimal Algorithms for A/B Testing
Po-An Wang (KTH Royal Institute of Technology), Alexandre Proutiere (KTH Royal Institute of Technology)
Optimization
🎯 What it does: This paper addresses the problem of optimal arm identification (A/B testing) under a fixed budget, proving that no adaptive algorithm can outperform uniform sampling in the two-arm Bernoulli setting. It also provides a precise analysis of the error rate of the Successive Rejects algorithm in the multi-arm case, showing that uniform sampling is superior in certain instances.
On Which Nodes Does GCN Fail? Enhancing GCN From the Node Perspective
Jincheng Huang (University of Electronic Science and Technology of China), Xiaofeng Zhu (University of Electronic Science and Technology of China)
ClassificationOptimizationGraph Neural NetworkAuto EncoderGraph
🎯 What it does: This paper examines the limitations of Graph Convolutional Networks (GCN) in node classification from the perspective of nodes, proposing the identification of 'Out-of-Control' (OOC) nodes where feature smoothing and label smoothing are inconsistent, and designing the DaGCN model to address two types of issues related to OOC nodes (neighbor scarcity and distance from labeled nodes).
One for All: A Universal Generator for Concept Unlearnability via Multi-Modal Alignment
Chaochao Chen (Zhejiang University), Zhongxuan Han (Zhejiang University)
GenerationAdversarial AttackConvolutional Neural NetworkContrastive LearningImageMultimodality
🎯 What it does: A universal perturbation generator called One-for-All is proposed, which utilizes multimodal alignment to achieve concept-independent, non-learnable samples, with both cross-dataset transferability and label independence.
One Meta-tuned Transformer is What You Need for Few-shot Learning
Xu Yang (City University of Hong Kong), Ying Wei (Nanyang Technological University)
ClassificationMeta LearningTransformerImage
🎯 What it does: MetaFormer is proposed, a meta-learning framework based on Vision Transformer, which achieves adaptive feature learning for few-shot classification tasks through Masked Sample Attention and Patch-grained Task Attention.
One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts
Ruochen Wang (University of California), Cho-Jui Hsieh (University of California)
Large Language ModelPrompt EngineeringMixture of ExpertsTextBenchmark
🎯 What it does: Divides the task space into several sub-regions, automatically constructs expert prompts containing instructions and examples for each region, and uses these expert mixed prompts to respond to new inputs.
One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning
Doyoung Kim (KAIST), Jae-Gil Lee (KAIST)
ClassificationRepresentation LearningTransformerPrompt EngineeringImage
🎯 What it does: An adaptive prompt tuning framework called AdaPromptCL is proposed to handle varying degrees of semantic drift in continual learning.
One-Shot Strategic Classification Under Unknown Costs
Elan Rosenfeld (Carnegie Mellon University), Nir Rosenfeld (Technion - Israel Institute of Technology)
ClassificationOptimizationTabular
🎯 What it does: This paper proposes a one-time strategic classification framework under unknown user cost functions, aiming to learn robust linear classifiers in the face of cost uncertainty.
Online Adaptive Anomaly Thresholding with Confidence Sequences
Sophia Huiwen Sun (University of California San Diego), Balakrishnan Murali Narayanaswamy (Amazon Web Services)
Anomaly DetectionTabularTime Series
🎯 What it does: This paper proposes an online adaptive anomaly threshold algorithm based on confidence sequences, which can dynamically set thresholds in unsupervised, non-stationary data streams and decide whether to make a decision or abstain based on the confidence interval.
Online Algorithms with Uncertainty-Quantified Predictions
Bo Sun (University of Waterloo), Raouf Boutaba (University of Waterloo)
OptimizationReinforcement LearningTabular
🎯 What it does: This paper proposes two paradigms for predicting design online algorithms using uncertainty quantification (UQ), and provides optimal algorithms under given UQ predictions for two classic problems: ski rental and one-time transactions (online search), as well as a framework for online learning that can learn to utilize UQ in a multi-instance environment.
Online bipartite matching with imperfect advice
Davin Choo (National University of Singapore), Arnab Bhattacharyya
OptimizationGraph
🎯 What it does: This paper studies the online unweighted bipartite matching problem under a random arrival model and proposes a learning-enhanced algorithm called TESTANDMATCH, which can improve matching quality by utilizing histogram predictions of online vertex types.
Online Cascade Learning for Efficient Inference over Streams
Lunyiu Nie (University of Texas at Austin), Swarat Chaudhuri (University of Texas at Austin)
OptimizationComputational EfficiencyKnowledge DistillationTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: Proposes an online cascading learning framework that processes streaming queries layer by layer using low-cost models, ultimately handing over more difficult instances to large language models, while online updating small models and implementing a retreat strategy for efficient inference.
Online conformal prediction with decaying step sizes
Anastasios Nikolas Angelopoulos, Stephen Bates (Massachusetts Institute of Technology)
OptimizationTime Series
🎯 What it does: An online conformal prediction method is proposed, which uses a time-decreasing learning rate to ensure long-term coverage under any sequence and achieves threshold convergence under independent and identically distributed (IID) sequences.
Online Isolation Forest
Filippo Leveni (Politecnico di Milano), Giacomo Boracchi (Politecnico di Milano)
Anomaly DetectionTabularTime Series
🎯 What it does: A novel online isolation forest algorithm, ONLINE-IFOREST, is proposed for anomaly detection in streaming data, capable of immediately updating the model after a single read and providing an anomaly score for each sample.
Online Learning and Information Exponents: The Importance of Batch size & Time/Complexity Tradeoffs
Luca Arnaboldi (Ecole Polytechnique Federale de Lausanne), Ludovic Stephan (Ecole Polytechnique Federale de Lausanne)
Ordinary Differential Equation
🎯 What it does: This paper studies the impact of batch size on the iteration time and sample complexity when training a two-layer neural network with one-shot SGD, and proposes a method called 'correlated loss SGD' to overcome the time bottleneck caused by the self-interaction of traditional SGD in the case of large batches.
Online Learning in Betting Markets: Profit versus Prediction
Haiqing Zhu (Australian National University), Lexing Xie (Australian National University)
Recommendation SystemAnomaly DetectionOptimizationTabularFinance Related
🎯 What it does: This paper constructs a binary betting market model, analyzes the conflict between profit maximization and prediction markets, and proposes two online learning algorithms for dynamically setting odds to achieve profit maximization.
Online Learning in CMDPs: Handling Stochastic and Adversarial Constraints
Francesco Emanuele Stradi (Politecnico di Milano), Nicola Gatti (Politecnico di Milano)
OptimizationReinforcement Learning
🎯 What it does: This paper proposes an online learning algorithm based on primal-dual gradient descent, PDGD-OPS, for addressing CMDP problems with long-term constraints.
Online Learning under Budget and ROI Constraints via Weak Adaptivity
Matteo Castiglioni (Politecnico di Milano), Christian Kroer (Columbia University)
Optimization
🎯 What it does: This paper studies the online learning problem under budget and return on investment (ROI) constraints, proposing a new algorithmic framework to maximize expected rewards.
Online Learning with Bounded Recall
Jon Schneider (Google), Kiran Vodrahalli (Google)
Reinforcement LearningTime Series
🎯 What it does: In the framework of full-information online learning, this paper studies learning algorithms with bounded recall and provides upper and lower bounds for such algorithms.
Online Linear Regression in Dynamic Environments via Discounting
Andrew Jacobsen (University of Alberta), Ashok Cutkosky (Boston University)
OptimizationTime Series
🎯 What it does: This paper develops an online linear regression algorithm that achieves optimal static and dynamic regret guarantees in the complete absence of prior knowledge.
Online Matching with Stochastic Rewards: Provable Better Bound via Adversarial Reinforcement Learning
Qiankun Zhang (Huazhong University of Science and Technology), Bingqian Du (Huazhong University of Science and Technology)
OptimizationGraph Neural NetworkReinforcement LearningGraph
🎯 What it does: This paper proposes an adversarial reinforcement learning framework that trains two agents to compete against each other, generating the hardest online matching instances (adv) and learning robust matching algorithms (alg) to study the hardness and algorithm performance of the Online Matching with Stochastic Rewards (OMSR) problem.
Online Matrix Completion: A Collaborative Approach with Hott Items
Dheeraj Baby (University of California Santa Barbara), Soumyabrata Pal (Adobe)
Recommendation System
🎯 What it does: This paper proposes two collaborative online matrix completion algorithms based on hot items (hott), targeting single-item recommendation (S=1) and multi-item recommendation (S=r) scenarios, providing computable recommendation strategies and analyzing their asymptotic performance in low-rank matrix completion tasks involving multiple users and items.
Online Resource Allocation with Non-Stationary Customers
Xiaoyue Zhang (National University of Singapore), Mabel Chou
Recommendation SystemOptimizationReinforcement LearningTabularTime Series
🎯 What it does: An algorithm named ULwE (Unified Learning‑while‑Earning) is proposed to address the online resource allocation problem in the context of non-stationary customer arrivals and unknown click-through rates.
Online Speculative Decoding
Xiaoxuan Liu (University of California Berkeley), Hao Zhang (University of California San Diego)
Computational EfficiencyKnowledge DistillationTransformerLarge Language ModelTextFinance Related
🎯 What it does: This paper proposes Online Speculative Decoding (OSD), which enhances token acceptance rates and reduces inference latency of large language models by real-time distillation and updating of draft models during the inference process.
Online Variational Sequential Monte Carlo
Alessandro Mastrototaro (KTH Royal Institute of Technology), Jimmy Olsson (KTH Royal Institute of Technology)
GenerationOptimizationVideoSequential
🎯 What it does: An online variational sequential Monte Carlo (OVSMC) algorithm is proposed for simultaneously learning the parameters of state space models and the particle proposal distribution in data streams.
OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shift
Lin Li (King's College London), Michael W. Spratling
Domain AdaptationAdversarial AttackTransformerImageBenchmark
🎯 What it does: This paper proposes and systematically evaluates the adversarial robustness benchmark OODRobustBench in the OOD domain, covering 23 types of dataset distribution shifts and 6 types of threat model shifts.
Open Ad Hoc Teamwork with Cooperative Game Theory
Jianhong Wang (University of Manchester), Samuel Kaski (Aalto University)
OptimizationGraph Neural NetworkReinforcement LearningGraph
🎯 What it does: This paper introduces the Open Stochastic Bayesian Cooperative Affinity Game (OSB-CAG) model, providing a theoretical framework for the Open Ad Hoc Teamwork (OAHT) problem, and based on this, proposes the concept of Dynamic Variational Strict Core (DVSC); it further explains the existing joint Q-value representation based on graph neural networks and designs a new algorithm called CIAO.
Open-Domain Text Evaluation via Contrastive Distribution Methods
Sidi Lu (University of California), Nanyun Peng (University of California)
GenerationData SynthesisTransformerLarge Language ModelContrastive LearningText
🎯 What it does: A contrastive evaluation framework based on two language models of different scales is proposed—Contrastive Distribution Methods (CDM), which includes generative and discriminative variants, to generate pseudo-negative samples or directly calculate contrastive momentum, thereby obtaining reference-free text quality scores.
Open-Vocabulary Calibration for Fine-tuned CLIP
Shuoyuan Wang (Southern University of Science and Technology), Hongxin Wei (Southern University of Science and Technology)
ClassificationRecognitionTransformerSupervised Fine-TuningPrompt EngineeringImage
🎯 What it does: A Distance-Aware Calibration (DAC) method is proposed to correct the confidence distortion of fine-tuned CLIP under open vocabulary.
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
Fuzhao Xue (National University of Singapore), Yang You (National University of Singapore)
TransformerLarge Language ModelMixture of ExpertsText
🎯 What it does: This work presents for the first time a series of Mixture-of-Experts (MoE) based decoder models called OpenMoE, ranging from 0.65B to 34B parameters, trained on up to 1T tokens of data. An in-depth analysis of its routing mechanism reveals phenomena such as 'Context-Independent Specialization', 'Early Routing Learning', and 'Drop-towards-the-End'. Improvement ideas are proposed to address the identified issues. Additionally, comparisons with dense models on multiple benchmarks demonstrate a better cost-performance trade-off.
Operator SVD with Neural Networks via Nested Low-Rank Approximation
Jongha Jon Ryu (Massachusetts Institute of Technology), Gregory W. Wornell (Massachusetts Institute of Technology)
RetrievalOptimizationImagePhysics Related
🎯 What it does: A framework utilizing Nested Low-Rank Approximation (Nested LoRA) is proposed, which uses neural networks to directly learn the first L singular values and corresponding singular functions of linear operators (such as integral kernels or differential operators);
Optimal Acceleration for Minimax and Fixed-Point Problems is Not Unique
TaeHo Yoon (Seoul National University), Ernest K. Ryu (University of California)
OptimizationOrdinary Differential Equation
🎯 What it does: This study investigates accelerated algorithms for fixed points and convex-concave minimax problems, proving that the optimal acceleration mechanism is not unique, and proposes a family of new algorithms based on H-duality, such as Dual-OHM and Dual-FEG.
Optimal Batched Linear Bandits
Xuanfei Ren (University of Science and Technology of China), Pan Xu (Duke University)
OptimizationReinforcement LearningTabular
🎯 What it does: This paper proposes a new batch linear Bandit algorithm E4, which adopts the Explore-Estimate-Eliminate-Exploit framework to address the optimal decision-making problem under limited time and asymptotic limits.
Optimal bounds for $\ell_p$ sensitivity sampling via $\ell_2$ augmentation
Alexander Munteanu (TU Dortmund University), Simon Omlor (TU Dortmund University)
OptimizationComputational Efficiency
🎯 What it does: This paper studies and proves that by incorporating ℓ2 sensitivity (i.e., 'ℓ2 augmentation') into ℓp sensitivity sampling, it is possible to achieve approximately optimal sampling complexity for ℓp subspace embeddings for all p ∈ [1, 2].
Optimal Coresets for Low-Dimensional Geometric Median
Peyman Afshani (Aarhus University), Chris Schwiegelshohn (Aarhus University)
OptimizationPoint Cloud
🎯 What it does: This study investigates the core set problem of geometric medians in low-dimensional geometry, aiming to compute a small weighted summary such that the cost of any median query is approximated within a factor of (1 ± ε).
Optimal Differentially Private Model Training with Public Data
Andrew Lowy (University of Wisconsin Madison), Meisam Razaviyayn (University of Southern California)
OptimizationSafty and PrivacyTabular
🎯 What it does: The paper studies how to perform differential privacy (DP) training on private datasets when public data (with no privacy risks) is available, and seeks to solve the optimal error upper bounds for average estimation, empirical risk minimization (ERM), and stochastic convex optimization (SCO); it also proposes a series of algorithms that achieve smaller errors with the assistance of public data.
Optimal Exact Recovery in Semi-Supervised Learning: A Study of Spectral Methods and Graph Convolutional Networks
Haixiao Wang (University of California San Diego), Zhichao Wang (University of California San Diego)
ClassificationOptimizationGraph Neural NetworkGraph
🎯 What it does: This paper studies the semi-supervised node classification problem under the Contextual Stochastic Block Model (CSBM), providing theoretical thresholds for exact recovery and proposing a series of estimation methods to achieve this threshold.
Optimal Eye Surgeon: Finding image priors through sparse generators at initialization
Avrajit Ghosh (Michigan State University), Rongrong Wang (Michigan State University)
RestorationImageMagnetic Resonance Imaging
🎯 What it does: This paper proposes a framework for sparse pruning during the random initialization of networks, resulting in a sparse subnetwork (Sparse-DIP) that can serve as an image prior for image recovery with little or no training.
Optimal Hessian/Jacobian-Free Nonconvex-PL Bilevel Optimization
Feihu Huang (Nanjing University of Aeronautics and Astronautics)
Optimization
🎯 What it does: This paper proposes a Hessian/Jacobian-free method HJFBiO for non-convex Polyak-Łojasiewicz (PL) bilevel optimization problems, and proves that under the PL condition, it can achieve an optimal convergence rate of O(1/T) and a gradient complexity of O(1/ε).
Optimal Kernel Choice for Score Function-based Causal Discovery
Wenjie Wang (University of Melbourne), Mingming Gong (University of Melbourne)
Score-based ModelTabular
🎯 What it does: This paper proposes a method for automatically selecting the best kernel function in a general scoring function based on RKHS to improve the accuracy of causal structure discovery.
Optimal Kernel Quantile Learning with Random Features
Caixing Wang (Shanghai University of Finance and Economics), Xingdong Feng (Shanghai University of Finance and Economics)
🎯 What it does: This paper proposes a kernel quantile regression method based on random features (KQR-RF), aimed at addressing the limitations of existing kernel ridge regression (KRR-RF) when dealing with heterogeneous data and heavy-tailed noise.
Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction
Christoph Jürgen Hemmer (Heidelberg University), Daniel Durstewitz (Heidelberg University)
GenerationOptimizationRecurrent Neural NetworkTime SeriesBiomedical DataElectrocardiogramPhysics Related
🎯 What it does: This paper studies how to generate sparse, structured recurrent neural networks through geometric-based pruning methods in dynamic system reconstruction tasks, achieving high-quality generative models of system dynamics.
Optimal Ridge Regularization for Out-of-Distribution Prediction
Pratik Patil (University of California), Ryan Tibshirani
Domain AdaptationOptimizationImage
🎯 What it does: This study investigates the optimal regularization level and risk behavior of ridge regression in out-of-distribution (OOD) prediction, providing general conditions for judging the ridge regularization symbol and proving that the optimal risk is monotonically related to the sample ratio.
Optimal Transport for Structure Learning Under Missing Data
Vy Vo (Monash University), Dinh Phung (VinAI Research)
OptimizationGraph Neural NetworkScore-based ModelGraphBiomedical Data
🎯 What it does: A structure learning framework based on optimal transport, OTM, is proposed to specifically address causal graph learning under missing data.
Optimally Improving Cooperative Learning in a Social Setting
Shahrzad Haddadan (Rutgers University), Jie Gao (Rutgers University)
OptimizationGraph Neural NetworkGraph
🎯 What it does: The study investigates how to select k agents with their own classifiers in a social network to improve their original predictions, thereby maximizing the overall prediction accuracy of the network.
Optimistic Multi-Agent Policy Gradient
Wenshuai Zhao (Aalto University), Joni Pajarinen (Aalto University)
Reinforcement LearningBenchmark
🎯 What it does: A framework is proposed that incorporates optimistic updates into multi-agent policy gradient methods, primarily achieved by clipping negative values in the advantage function to 0 (or using Leaky ReLU);
Optimization without Retraction on the Random Generalized Stiefel Manifold
Simon Vary (University College London), Pierre-Antoine Absil
Optimization
🎯 What it does: A new stochastic iterative method is proposed for optimization on random generalized Stiefel manifolds, addressing optimization problems with only random estimates.
Optimizing Watermarks for Large Language Models
Bram Wouters (University of Amsterdam)
OptimizationLarge Language ModelText
🎯 What it does: A watermark model based on green-red word list segmentation is proposed, and the trade-off between recognizability and text quality is systematically studied from the perspective of multi-objective optimization, introducing the OPT watermark and proving its Pareto optimality;
OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models
Ali AhmadiTeshnizi (Stanford University), Madeleine Udell (Stanford University)
OptimizationTransformerLarge Language ModelAgentic AIPrompt EngineeringText
🎯 What it does: A multi-agent system named OptiMUS based on large language models has been developed, capable of automatically decomposing optimization problems described in natural language into parameters, constraints, and objectives, generating executable solving code, and self-debugging, ultimately solving linear programming and mixed-integer programming problems.
Orthogonal Bootstrap: Efficient Simulation of Input Uncertainty
Kaizhao Liu (Peking University), Yiping Lu (Northwestern University)
Tabular
🎯 What it does: This paper proposes Orthogonal Bootstrap, which combines influence functions and control variables to reduce the number of Monte Carlo repetitions in bootstrap and lower simulation errors.
OSN: Infinite Representations of Dynamic 3D Scenes from Monocular Videos
Ziyang Song (Hong Kong Polytechnic University), Bo Yang (Hong Kong Polytechnic University)
SegmentationGenerationDepth EstimationNeural Radiance FieldVideo
🎯 What it does: This paper proposes an OSN framework that learns all feasible dynamic 3D scene representations from monocular RGB videos, capable of infinitely generating 3D scene configurations that satisfy the video.
OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization
Xiang Meng (Massachusetts Institute of Technology), Rahul Mazumder (Massachusetts Institute of Technology)
OptimizationComputational EfficiencyTransformerVision Language ModelImageText
🎯 What it does: A one-shot (post-training) structured pruning framework called OSSCAR is proposed, which can prune channels, heads, and other structures in neural networks without retraining, significantly reducing model size and inference time.
OT-CLIP: Understanding and Generalizing CLIP via Optimal Transport
Liangliang Shi (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
ClassificationOptimizationRepresentation LearningContrastive LearningImage
🎯 What it does: The contrastive learning and inference processes of CLIP are treated as inverse optimal transport (Inverse OT) bi-level optimization and graph matching problems, proposing the OT-CLIP loss family and OT prediction framework.
OTMatch: Improving Semi-Supervised Learning with Optimal Transport
Zhiquan Tan (Tsinghua University), Weiran Huang (Shanghai Jiao Tong University)
ClassificationImageText
🎯 What it does: The OTMatch algorithm is proposed, which introduces optimal transport loss in semi-supervised learning to embed the semantic relationships between categories into the model, alleviating the issue of overconfidence in pseudo-labels.
Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift
Benjamin Eyre (Columbia University), Richard Zemel (University of Toronto)
Domain AdaptationImageTabular
🎯 What it does: This paper proposes a post-processing method for spectral adaptation of regression models under covariate shift, called SpAR, which projects the weights of the last layer of a pre-trained regression model using unlabeled target data to reduce out-of-distribution (OOD) errors caused by spectral inflation.
Out-of-Distribution Detection via Deep Multi-Comprehension Ensemble
Chenhui Xu (George Mason University), Xiang Chen (George Mason University)
Anomaly DetectionConvolutional Neural NetworkContrastive LearningImage
🎯 What it does: A multi-comprehension ensemble method is proposed, which enhances the diversity of feature representations by using different training tasks (cross-entropy, SimCLR, SupCon) within the same network structure, thereby improving the model's ability to detect OOD samples.
Out-of-Domain Generalization in Dynamical Systems Reconstruction
Niclas Alexander Göring (Heidelberg University), Daniel Durstewitz (Heidelberg University)
Recurrent Neural NetworkTime SeriesPhysics RelatedOrdinary Differential Equation
🎯 What it does: This paper proposes a formal theoretical framework for out-of-domain generalization (OODG) in dynamic system reconstruction (DSR) and quantitatively analyzes the generalization ability of existing deep learning methods in multi-stability systems.
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
Lu Yin (Google Research), Shiwei Liu (University of Oxford)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: A sparse strategy called OWL based on hierarchical outlier ratio is proposed to achieve up to 70% one-shot sparse pruning while maintaining the performance of the language model.
Outlier-aware Slicing for Post-Training Quantization in Vision Transformer
Yuexiao Ma (Xiamen University), Rongrong Ji (Xiamen University)
OptimizationTransformerImage
🎯 What it does: This study investigates the outlier problem of Vision Transformer in Post-Training Quantization (PTQ) and introduces the concept of 'reconstruction granularity'. Based on this concept, a slicing optimization algorithm is designed to improve quantization accuracy.
Outlier-Efficient Hopfield Layers for Large Transformer-Based Models
Jerry Yao-Chieh Hu (Northwestern University), Han Liu (Northwestern University)
Anomaly DetectionOptimizationTransformerLarge Language ModelImageTextTime Series
🎯 What it does: A model based on the modern Hopfield network called 'Outlier-Efficient' is proposed, which improves the Transformer attention mechanism to reduce output anomalies caused by low-information words (such as punctuation and separators).
Outlier-robust Kalman Filtering through Generalised Bayes
Gerardo Duran-Martin (Queen Mary University), Kevin Patrick Murphy
Anomaly DetectionOptimizationTabularTime SeriesStochastic Differential EquationOrdinary Differential Equation
🎯 What it does: A robust Kalman filtering method based on generalized Bayesian inference (WoLF) is proposed, which achieves robust online state estimation against outliers and model errors by using weighted log-likelihood in the update step.
Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors
Chun-Yin Huang (University of British Columbia), Xiaoxiao Li (University of British Columbia)
Domain AdaptationFederated LearningKnowledge DistillationImage
🎯 What it does: In a serverless decentralized federated learning scenario, this paper proposes the use of synthetic anchor data through feature regularization and knowledge distillation methods to alleviate data and model heterogeneity, thereby improving the generalization performance of local models on cross-domain tasks.
Overcoming Saturation in Density Ratio Estimation by Iterated Regularization
Lukas Gruber (Johannes Kepler University Linz), Werner Zellinger (Austrian Academy of Sciences)
Domain AdaptationOptimizationImageText
🎯 What it does: Proposes and implements an iterative regularization method (iterative Tikhonov regularization) for density ratio estimation, overcoming the saturation phenomenon of traditional kernel methods and achieving faster error convergence across various algorithms.
Overcoming the Optimizer's Curse: Obtaining Realistic Prescriptions from Neural Networks
Asterios Tsiourvas (Massachusetts Institute of Technology), Georgia Perakis (Massachusetts Institute of Technology)
OptimizationConvolutional Neural NetworkImage
🎯 What it does: This paper studies how to avoid obtaining extreme 'Optimizer's Curse' results that are far from the data distribution while ensuring optimality when using neural networks for data-driven decision-making. An adaptive sampling algorithm is proposed to find realistic solutions within a feasible polytope by reconstructing the Local Outlier Factor (LOF) into solvable linear/quadratic constraints.
Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
Michal Nauman (University of Warsaw), Marek Cygan (University of Warsaw)
Reinforcement LearningBenchmark
🎯 What it does: A systematic evaluation and comparison of over 60 Soft Actor-Critic-based regularization strategies on a wide range of control tasks reveals the key role of network regularization (such as layer normalization and spectral normalization) and full parameter reset in enhancing off-policy RL performance, especially on challenging problems like Dog, where it can approach the performance of model-based methods.
OxyGenerator: Reconstructing Global Ocean Deoxygenation Over a Century with Deep Learning
Bin Lu (Shanghai Jiao Tong University), Jing Zhang (Shanghai Jiao Tong University)
Recurrent Neural NetworkGraph Neural NetworkTime Series
🎯 What it does: Using the deep learning model OXYGENERATOR to reconstruct the global ocean deoxygenation history from 1920 to 2023 based on sparse observations.
PAC-Bayesian Error Bound, via Rényi Divergence, for a Class of Linear Time-Invariant State-Space Models
Deividas Eringis (Aalborg University), Mihaly Petreczky (University of Lille)
Time Series
🎯 What it does: This paper derives PAC-Bayesian error bounds for a class of linear time-invariant state space models, primarily focusing on stochastic dynamic systems with inputs.