ICML 2023 Papers — Page 15
International Conference on Machine Learning · 1828 papers
Rethinking Backdoor Attacks
Alaa Khaddaj (Massachusetts Institute of Technology), Aleksander Madry (Massachusetts Institute of Technology)
Anomaly DetectionOptimizationAdversarial AttackConvolutional Neural NetworkImage
🎯 What it does: This paper proposes viewing backdoor triggers as the strongest features in data and designs a backdoor detection and removal algorithm based on this, which estimates feature strength using a data model, solves for the maximum submatrix, and performs a greedy local search.
Rethinking Explaining Graph Neural Networks via Non-parametric Subgraph Matching
Fang Wu (Westlake University), Stan Z. Li (Westlake University)
Explainability and InterpretabilityGraph Neural NetworkGraph
🎯 What it does: By performing non-parametric subgraph matching between the target graph and other graphs in the reference set, MatchExplainer proposes a method for extracting interpretable subgraphs for GNN predictions, and designs the MatchDrop data augmentation method based on this interpreter.
Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues
Jiaxuan Chen (Zhejiang University), Gang Pan (Zhejiang University)
RestorationSuper ResolutionConvolutional Neural NetworkAuto EncoderImageMagnetic Resonance Imaging
🎯 What it does: This paper proposes a content completion method based on visual experience called VQ-fMRI, which maps fMRI signals to a discrete code space and achieves high-quality image reconstruction through token-level self-supervised inpainting and hierarchical super-resolution.
Rethinking Warm-Starts with Predictions: Learning Predictions Close to Sets of Optimal Solutions for Faster $\text{L}$-/$\text{L}^\natural$-Convex Function Minimization
Shinsaku Sakaue (University of Tokyo), Taihei Oki (University of Tokyo)
Optimization
🎯 What it does: This paper proposes a novel predictive heating start framework for minimizing L-/L-convex functions and establishes the relationship between time complexity and the distance to the optimal solution set.
Rethinking Weak Supervision in Helping Contrastive Learning
Jingyi Cui (Peking University), Yisen Wang (Peking University)
ClassificationRepresentation LearningContrastive LearningImage
🎯 What it does: A unified theoretical framework for contrastive learning under weak supervision (semi-supervised labels and noisy labels) has been constructed, and the impact of the two types of weak supervision information on contrastive learning has been systematically explored from both theoretical and experimental perspectives.
Retrieval-Augmented Multimodal Language Modeling
Michihiro Yasunaga (Stanford University), Wen-tau Yih (Meta AI)
GenerationRetrievalTransformerVision Language ModelImageTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: A retrieval-augmented multimodal model RA-CM3 is proposed, which can retrieve external multimodal documents based on input text or images and use the retrieved content as context to generate text and images.
Retrosynthetic Planning with Dual Value Networks
Guoqing Liu (Microsoft Research), Tie-Yan Liu (Microsoft Research)
OptimizationDrug DiscoveryReinforcement LearningGraph
🎯 What it does: Online training of a single-step predictor using reinforcement learning, combined with a dual-value network to improve the success rate and route quality of multi-step retro-synthesis planning.
Returning The Favour: When Regression Benefits From Probabilistic Causal Knowledge
Shahine Bouabid, Dino Sejdinovic (University of Adelaide)
TabularTime Series
🎯 What it does: Proposes a collider regression framework that utilizes the independence of collider structures in causal graphs as a prior bias for regression;
Revisiting Bellman Errors for Offline Model Selection
Joshua P Zitovsky (University of North Carolina at Chapel Hill), Michael Rene Kosorok
OptimizationConvolutional Neural NetworkReinforcement LearningTabularSequential
🎯 What it does: This paper studies the model selection problem in offline reinforcement learning, proposing a new estimation method based on Mean Squared Bellman Error (MSBE) and implementing the Supervised Bellman Validation (SBV) algorithm to select the optimal policy from offline data.
Revisiting Data-Free Knowledge Distillation with Poisoned Teachers
Junyuan Hong (Michigan State University), Jiayu Zhou (Michigan State University)
Knowledge DistillationAdversarial AttackConvolutional Neural NetworkImage
🎯 What it does: This study investigates the backdoor propagation risks of data-free knowledge distillation (data-free KD) when using contaminated teacher models and proposes a pluggable defense framework called ABD.
Revisiting Discriminative vs. Generative Classifiers: Theory and Implications
Chenyu Zheng (Renmin University of China), Jun Zhu (Tsinghua University)
ClassificationRepresentation LearningTransformerContrastive LearningImage
🎯 What it does: The study compares the statistical efficiency of generative naive Bayes and discriminative logistic regression classifiers in the linear evaluation of deep pre-trained models, and provides a sample complexity analysis in the case of multiple categories.
Revisiting Domain Randomization via Relaxed State-Adversarial Policy Optimization
Yun-Hsuan Lien (National Yang Ming Chiao Tung University), Yu-Shuen Wang (National Yang Ming Chiao Tung University)
OptimizationReinforcement LearningSequential
🎯 What it does: This paper proposes a Relaxed Adversarial Policy Optimization (RAPPO) method, which achieves domain randomization by adding adversarial perturbations after each state transition, and simultaneously enhances both average and worst-case rewards through a bi-level optimization approach, addressing the overly conservative issues caused by traditional state adversarial methods.
Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
Anastasia Koloskova (École Polytechnique Fédérale de Lausanne), Sebastian U Stich
OptimizationTabular
🎯 What it does: This study investigates the convergence properties of gradient clipping in both deterministic and stochastic optimization, providing a precise dependence on the clipping threshold c and proving the tightness of the associated upper and lower bounds.
Revisiting Over-smoothing and Over-squashing Using Ollivier-Ricci Curvature
Khang Nguyen (FPT Software AI Center), Tan Minh Nguyen
Graph Neural NetworkGraph
🎯 What it does: This paper re-examines the issues of over-smoothing and over-compression in GNNs from a local geometric perspective using Ollivier-Ricci curvature, and based on this, proposes a novel graph reconnection algorithm called BORF.
Revisiting Pseudo-Label for Single-Positive Multi-Label Learning
Biao Liu (Southeast University), Xin Geng (Southeast University)
ClassificationRecognitionImage
🎯 What it does: For the problem of Single Positive Sample Multi-Label Learning (SPMLL), theoretical conditions for the effectiveness and learnability of pseudo-labels are first provided, followed by the proposal of a pseudo-label generation and enhancement method based on mutual information bottleneck, called MIME.
Revisiting Sampling for Combinatorial Optimization
Haoran Sun (Georgia Tech), Hanjun Dai (Google Deepmind)
OptimizationReinforcement LearningGraphBenchmark
🎯 What it does: An improved discrete MCMC sampling method is used to implement a training-free universal combinatorial optimization solver.
Revisiting Simple Regret: Fast Rates for Returning a Good Arm
Yao Zhao (University of Arizona), Kwang-Sung Jun (University of Arizona)
Tabular
🎯 What it does: For the pure exploration task in multi-armed bandits, the theoretical analysis of the simple regret metric has been studied and improved. Improved Sequential Halving (SH) and Bracketing SH (BSH) algorithms have been proposed, achieving low simple regret in both data-rich and data-scarce scenarios.
Revisiting Structured Variational Autoencoders
Yixiu Zhao (Stanford University), Scott Linderman
GenerationData SynthesisOptimizationComputational EfficiencyConvolutional Neural NetworkRecurrent Neural NetworkAuto EncoderVideoTime SeriesSequential
🎯 What it does: Reimplement and evaluate the Structured Variational Autoencoder (SVAE) using JAX, parallel Kalman filtering, and self-supervised training methods for missing data.
Revisiting the Linear-Programming Framework for Offline RL with General Function Approximation
Asuman E. Ozdaglar (Massachusetts Institute of Technology), Kaiqing Zhang (University of Maryland)
Reinforcement Learning
🎯 What it does: Redesign the linear programming (LP) framework in offline reinforcement learning and propose two algorithms that utilize function approximation, achieving optimal sample complexity under both the completeness assumption and the realizability assumption.
Revisiting Weighted Aggregation in Federated Learning with Neural Networks
Zexi Li (Zhejiang University), Chao Wu (Zhejiang University)
Federated LearningConvolutional Neural NetworkImage
🎯 What it does: Revisiting the weighted aggregation of neural networks in federated learning, it was found that global weight shrinkage is related to client consistency, and based on this, the FedLAW method was designed.
Reward-Mixing MDPs with Few Latent Contexts are Learnable
Jeongyeol Kwon (University of Wisconsin Madison), Shie Mannor (Nvidia Research)
Reinforcement Learning
🎯 What it does: This paper studies the learning of multiple potential reward models in Reward Mixed Markov Decision Processes (RMMDP) and proposes a two-stage algorithm EM2, which can learn an ε-approximate optimal policy with a sample size of O((SA)^{O(M)}/ε^2).
RGE: A Repulsive Graph Rectification for Node Classification via Influence
Jaeyun Song (Korea Advanced Institute of Science and Technology), Eunho Yang (AITRICS)
ClassificationGraph Neural NetworkGraph
🎯 What it does: This paper proposes a rejection-based edge group elimination method (RGE) that prioritizes the removal of remote edges and performs multiple rounds of retraining to reduce group effect errors, thereby improving node classification performance.
Rigid Body Flows for Sampling Molecular Crystal Structures
Jonas Köhler, Frank Noe (Freie Universit¨ at Berlin)
Flow-based ModelPhysics Related
🎯 What it does: A new regularized flow model has been designed to sample the displacements and orientations of multiple rigid bodies in molecular crystals in three-dimensional space.
RLang: A Declarative Language for Describing Partial World Knowledge to Reinforcement Learning Agents
Rafael Rodriguez-Sanchez (Brown University), George Konidaris (Brown University)
Reinforcement Learning
🎯 What it does: This paper presents RLang, a domain-specific language for reinforcement learning, designed to describe various components of MDP (state features, goals, options, policies, rewards, transitions, etc.) with partial world knowledge.
RLEG: Vision-Language Representation Learning with Diffusion-based Embedding Generation
Liming Zhao (Alibaba Group), Jingren Zhou (Alibaba Group)
Object DetectionSegmentationGenerationRetrievalRepresentation LearningDiffusion modelContrastive LearningImageTextMultimodality
🎯 What it does: This paper proposes a diffusion model-based embedding generation method called RLEG, which enhances contrastive learning in visual-language pre-training by online generating diverse embeddings.
RLSbench: Domain Adaptation Under Relaxed Label Shift
Saurabh Garg (Carnegie Mellon University), Zachary Chase Lipton
Domain AdaptationImageTextBenchmark
🎯 What it does: The RLSBench benchmark is proposed, evaluating 13 mainstream domain adaptation methods, exploring their performance in the 'relaxed label shift' scenario where label proportions are mismatched, and introducing a two-step meta-algorithm (pseudo-balanced resampling + post-hoc reweighting) that can be combined with most methods to significantly improve performance.
Robust and private stochastic linear bandits
Vasileios Charisopoulos (Cornell University), Vahab Mirrokni (Google)
OptimizationSafty and Privacy
🎯 What it does: A batch linear bandit learning algorithm is proposed under the requirements of differential privacy and robustness;
Robust and Scalable Bayesian Online Changepoint Detection
Matias Altamirano (University College London), Jeremias Knoblauch (University College London)
Anomaly DetectionDiffusion modelTime SeriesFinance Related
🎯 What it does: A generalized Bayesian online change point detection method based on diffusion score matching (Dₘ-BOCD) is proposed, achieving robustness and scalability.
Robust Budget Pacing with a Single Sample
Santiago R. Balseiro (Columbia Business School), Di Wang (Google Research)
OptimizationTime Series
🎯 What it does: A budget pacing algorithm called Dual FTRL is proposed, which can perform real-time budget allocation for time-varying advertising auctions with only one sample available at each time point.
Robust Camera Pose Refinement for Multi-Resolution Hash Encoding
Hwan Heo (Korea University), Jin-Hwa Kim (NAVER AI Lab)
Pose EstimationOptimizationNeural Radiance FieldImage
🎯 What it does: To address the camera pose calibration problem under multi-resolution hash encoding (Instant-NGP), a joint optimization framework is proposed that combines smooth gradients (straight-through estimator) and hierarchical learning rate scheduling, achieving robust pose correction and scene reconstruction starting from noisy or unknown poses.
Robust Collaborative Learning with Linear Gradient Overhead
Sadegh Farhadkhani (École Polytechnique Fédérale de Lausanne), John Stephan
OptimizationFederated LearningReinforcement LearningImage
🎯 What it does: A robust collaborative learning algorithm named MONNA is proposed, which can still converge to the critical point of the correct node's average loss in the presence of Byzantine nodes.
Robust Consensus in Ranking Data Analysis: Definitions, Properties and Computational Issues
Morgane Goibert (Criteo AI Lab), Stephan CLEMENCON
Tabular
🎯 What it does: This study investigates the robustness of ranking data, defines the breakdown point of rankings, and proposes bucket ranking and Downward Merge statistics to enhance robustness.
Robust Counterfactual Explanations for Neural Networks With Probabilistic Guarantees
Faisal Hamman (University of Maryland), Sanghamitra Dutta (JP Morgan Chase AI Research)
Anomaly DetectionOptimizationExplainability and InterpretabilityComputational EfficiencyTabularBiomedical DataFinance Related
🎯 What it does: This paper proposes a theoretical and practical assessment of the robustness of causal explanations under potential natural model changes in neural network models, providing a stability measure and its relaxed form, and designing two algorithms (T-Rex:I and T-Rex:NN) for generating robust causal explanations based on this measure.
Robust Explanation for Free or At the Cost of Faithfulness
Zeren Tan (Tsinghua University), Yang Tian (Tsinghua University)
Explainability and InterpretabilityConvolutional Neural NetworkGaussian SplattingImage
🎯 What it does: This study explores the relationship between model robustness and explanation robustness, proving that robust models can obtain robust explanations for free. It proposes that applying kernel smoothing to any model can achieve robust explanations while revealing the trade-off between robustness and credibility.
Robust Non-Linear Feedback Coding via Power-Constrained Deep Learning
Junghoon Kim (Purdue University), Christopher Brinton (Purdue University)
Recurrent Neural NetworkAuto EncoderSequential
🎯 What it does: This paper proposes a nonlinear feedback coding scheme based on RNN autoencoders, significantly enhancing robustness against forward and feedback noise by introducing a power control layer and block-level coding.
Robust One-Class Classification with Signed Distance Function using 1-Lipschitz Neural Networks
Louis Béthune (IRIT Université Paul Sabatier), Andres Troya-Galvis (Thales Alénia Space)
Anomaly DetectionImagePoint CloudTabular
🎯 What it does: A one-class classification method based on first-order Lipschitz neural networks learning signature distance functions (SDF) is proposed, using SDF as a normalized metric to achieve provable robustness.
Robust Perception through Equivariance
Chengzhi Mao (Columbia University), Carl Vondrick (Columbia University)
ClassificationSegmentationAdversarial AttackContrastive LearningImage
🎯 What it does: This paper proposes a method to enhance the model's robustness against adversarial attacks by restoring equivariance of visual representations during the inference phase.
Robust Satisficing MDPs
Haolin Ruan (City University of Hong Kong), Chin Pang Ho (City University of Hong Kong)
OptimizationReinforcement Learning
🎯 What it does: A Robust Satisfiability (RSMDP) framework is proposed, which achieves the target expected return and minimizes constraint violations through soft constraints under uncertain transition probabilities.
Robust Situational Reinforcement Learning in Face of Context Disturbances
Jinpeng Zhang (Tsinghua University), Jiang Bian (Microsoft Research Asia)
Reinforcement LearningTabularTime Series
🎯 What it does: A robust contextual reinforcement learning framework (RS-MDP) is proposed, and based on this, a soft minimization smooth robust Bellman operator is designed, extending SAC to RS-SAC to handle transition uncertainty caused by external contextual changes in the environment.
Robust Speech Recognition via Large-Scale Weak Supervision
Alec Radford (OpenAI), Ilya Sutskever (OpenAI)
RecognitionTransformerAudio
🎯 What it does: This paper presents a large-scale weakly supervised speech recognition model called Whisper, which utilizes approximately 680,000 hours of internet audio and subtitles for unsupervised training across multiple languages and tasks, achieving zero-shot deployment without any dataset-specific fine-tuning.
Robust Subtask Learning for Compositional Generalization
Kishor Jothimurugan (University of Pennsylvania), Rajeev Alur (University of Pennsylvania)
Robotic IntelligenceReinforcement Learning
🎯 What it does: Train robust subtask policies (options) to complete tasks under any subtask sequence, maximizing worst-case performance.
Robust Weak Supervision with Variational Auto-Encoders
Francesco Tonolini (Amazon), Gabriella Kazai (University of Sheffield)
ClassificationAuto EncoderVideoTextBenchmark
🎯 What it does: A weakly supervised framework WS-VAE based on variational autoencoders is proposed, which can perform classification tasks without human annotations.
Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?
Ruisi Cai (University of Texas at Austin), Zhangyang Wang (University of Texas at Austin)
Adversarial AttackConvolutional Neural NetworkImage
🎯 What it does: Robust Weight Signatures (RWS) are proposed, achieving lightweight, adjustable, and composable robustness patches by adding RWS to standard model weights.
Robustly Learning a Single Neuron via Sharpness
Puqian Wang (University of Wisconsin Madison), Jelena Diakonikolas (University of Wisconsin Madison)
OptimizationSupervised Fine-Tuning
🎯 What it does: The study investigates how to effectively learn a single neuron in the presence of adversarial label noise, proposing an algorithm that can approximate the optimal L2 loss within a constant factor under a wide range of activation functions, including ReLU.
Robustness in Multimodal Learning under Train-Test Modality Mismatch
Brandon McKinzie (Apple), Alexander T Toshev
Knowledge DistillationRepresentation LearningTransformerAuto EncoderContrastive LearningImageVideoTextMultimodalityAudio
🎯 What it does: This paper addresses the issue of inconsistency in multimodal combinations during training and deployment, proposing a systematic robustness evaluation framework and improving multimodal representation learning methods.
Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch
Xunyi Zhao (Inria Center at the University of Bordeaux), Olivier Beaumont (Inria Center at the University of Bordeaux)
OptimizationComputational EfficiencyTransformerLarge Language ModelText
🎯 What it does: An automated tool named Rockmate has been developed, which can generate equivalent models from PyTorch nn.Module code and implement activation rematerialization under a given memory budget, significantly reducing GPU memory usage during training.
Rotation and Translation Invariant Representation Learning with Implicit Neural Representations
Sehyun Kwon (Seoul National University), Ernest K. Ryu (Seoul National University)
Representation LearningContrastive LearningImage
🎯 What it does: This paper proposes a framework called IRL-INR that utilizes implicit neural representations (INR) and hyper-networks to learn low-dimensional semantic representations unaffected by image rotation and translation, and applies it to unsupervised clustering.
RSC: Accelerate Graph Neural Networks Training via Randomized Sparse Computations
Zirui Liu (Rice University), Xia Hu (Rice University)
Graph Neural NetworkGraph
🎯 What it does: By using top-k sampling for sparse matrix multiplication approximation during the backpropagation phase, RSC improves the training speed of GNNs.
Run-off Election: Improved Provable Defense against Data Poisoning Attacks
Keivan Rezaei (University of Maryland), Soheil Feizi (University of Maryland)
ClassificationAdversarial AttackData-Centric LearningConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: A ROE aggregation method based on two-round elections is proposed to enhance the verifiable robustness of data poisoning defenses.
SAAL: Sharpness-Aware Active Learning
Yoon-Yeong Kim (Agency for Defense Development), Il-chul Moon
ClassificationObject DetectionSegmentationImage
🎯 What it does: A Sharpness-Aware Active Learning (SAAL) method is proposed, which utilizes the sharpness of the loss surface to measure the potential value of unlabeled samples for active learning.
Safe Offline Reinforcement Learning with Real-Time Budget Constraints
Qian Lin (Sun Yat-Sen University), Dong Wang (Meituan)
OptimizationReinforcement LearningDiffusion modelTabularSequential
🎯 What it does: An algorithm called TREBI is proposed for offline safe reinforcement learning, which can directly infer the optimal trajectory distribution under real-time budget constraints, addressing the challenge of quickly adapting to different budgets during deployment.
SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
Atish Agarwala (Google DeepMind), Yann Dauphin (Google DeepMind)
OptimizationConvolutional Neural NetworkImage
🎯 What it does: This paper studies the adaptive suppression mechanism of the maximum eigenvalue of the loss Hessian or NTK along the entire training trajectory using Sharpness Aware Minimization (SAM). It reveals the phenomenon of boundary stability (EOS) achieved by SAM through nonlinear discrete dynamics and validates it on a quadratic regression model and a practical network (WideResNet28-10).
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models
Hong Liu (Stanford University), Tengyu Ma (Stanford University)
OptimizationRepresentation LearningTransformerLarge Language ModelText
🎯 What it does: This paper studies that in the self-supervised pre-training of language models, even with the same pre-training loss, the model's performance on downstream tasks may differ, further proposing that the implicit bias of the pre-training process plays a decisive role in transferability.
Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation
Ilan Naiman (BenGurion University of the Negev), Omri Azencot (BenGurion University of the Negev)
GenerationData SynthesisRepresentation LearningAuto EncoderContrastive LearningVideoMultimodalityTime SeriesAudio
🎯 What it does: A novel unsupervised sequence information decoupling framework is proposed, utilizing the empirical distribution generated by variational autoencoders for contrastive learning, without the need for modality-based data augmentation.
Sample Complexity Bounds for Learning High-dimensional Simplices in Noisy Regimes
seyed amir hossein saberi, Babak Khalaj
🎯 What it does: This paper studies the sample complexity bounds for learning high-dimensional simplices in noisy environments and proposes an algorithm that outputs a simplex with an ℓ2 distance from the true simplex not exceeding ε with high probability.
Sample Complexity of Probability Divergences under Group Symmetry
Ziyu Chen (University of Massachusetts Amherst), Wei Zhu (University of Massachusetts Amherst)
Data SynthesisOptimizationBiomedical DataMagnetic Resonance Imaging
🎯 What it does: The study improves the sample complexity of estimating probability divergence using variational representation under group symmetric distributions.
Sampling-Based Accuracy Testing of Posterior Estimators for General Inference
Pablo Lemos (Mila Quebec AI Institute), Laurence Perreault-Levasseur (Mila Quebec AI Institute)
Diffusion modelGenerative Adversarial NetworkTabular
🎯 What it does: A coverage verification method based on random points (TARP) is proposed, which uses only posterior sampling to assess the accuracy of generative posterior estimators.
Sampling-based Nyström Approximation and Kernel Quadrature
Satoshi Hayakawa (Mathematical Institute, University of Oxford), Terry Lyons (Mathematical Institute, University of Oxford)
Tabular
🎯 What it does: Theoretical analysis of the Nystrom approximation is conducted, and an improved low-rank approximation method is proposed to enhance the performance of kernel integrals (kernel quads).
Scalable Adaptive Computation for Iterative Generation
Allan Jabri (Google), Ting Chen (Google)
GenerationData SynthesisRecurrent Neural NetworkDiffusion modelImageVideo
🎯 What it does: Proposes the Recurrent Interface Network (RIN), an adaptive architecture that focuses computation on latent representations and routes between the interface and latent through cross-attention, for pixel-level diffusion generation.
Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation
Siddharth Nayak (Massachusetts Institute of Technology), Hamsa Balakrishnan (Massachusetts Institute of Technology)
Robotic IntelligenceGraph Neural NetworkReinforcement LearningAgentic AIGraph
🎯 What it does: The InforMARL framework is proposed for multi-agent navigation and collision avoidance problems, utilizing graph neural networks to aggregate local neighborhood information, thereby achieving distributed decision-making and effective collaboration under local observations.
Scalable Safe Policy Improvement via Monte Carlo Tree Search
Alberto Castellini (University of Verona), Matthijs T. J. Spaan (Delft University of Technology)
OptimizationSafty and PrivacyReinforcement LearningTabular
🎯 What it does: By incorporating SPIBB constraints into MCTS, the MCTS-SPIBB algorithm is proposed to achieve online safe policy improvement.
Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation
Jeffrey Willette (KAIST), Sung Ju Hwang (KAIST)
ClassificationOptimizationTransformerImageText
🎯 What it does: A Universal Mini-Batch Consistent (UMBC) set encoder is proposed, along with an unbiased full gradient approximation algorithm, enabling scalable training on large-scale sets.
Scaling Laws for Generative Mixed-Modal Language Models
Armen Aghajanyan (Meta), Luke Zettlemoyer (University of Washington)
GenerationData SynthesisTransformerLarge Language ModelImageTextMultimodalityAudio
🎯 What it does: This paper studies the scaling law of mixed-modal generative language models through 250+ experiments and derives an extended formula that includes modal synergy and competition.
Scaling Laws for Multilingual Neural Machine Translation
Patrick Fernandes (Carnegie Mellon University), Orhan Firat (Google Research)
TransformerText
🎯 What it does: This study provides a large-scale empirical investigation of the scaling properties of multilingual neural machine translation models, exploring how increasing model size affects performance and examining the impact of training mixed components on scaling behavior.
Scaling Laws for Reward Model Overoptimization
Leo Gao (OpenAI), Jacob Hilton (OpenAI)
OptimizationReinforcement Learning from Human FeedbackLarge Language ModelReinforcement LearningText
🎯 What it does: The phenomenon of over-optimization when using proxy reward models in RLHF is studied, and scaling laws for best-n sampling and PPO are provided.
Scaling of Class-wise Training Losses for Post-hoc Calibration
Seungjin Jung (Chung Ang University), Jongwon Choi (Chung Ang University)
ClassificationOptimizationConvolutional Neural NetworkImage
🎯 What it does: A post-hoc calibration method based on class-wise loss scaling is proposed, which reduces calibration error and maintains the original model accuracy by balancing the training loss across different classes.
Scaling Spherical CNNs
Carlos Esteves (Google Research), Ameesh Makadia (Google Research)
Computational EfficiencyDrug DiscoveryConvolutional Neural NetworkTransformerTabularTime SeriesBenchmark
🎯 What it does: This paper proposes a scalable Spherical CNN architecture that enables efficient training and inference for large-scale molecular property prediction and weather forecasting tasks.
Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory
Justin Cui (University of California), Cho-Jui Hsieh (University of California)
OptimizationKnowledge DistillationConvolutional Neural NetworkImage
🎯 What it does: This paper proposes a method for dataset distillation under constant memory usage, allowing the originally memory-intensive Matching Training Trajectories (MTT) to scale to ImageNet-1K, and further enhances performance through soft label allocation.
Scaling Vision Transformers to 22 Billion Parameters
Mostafa Dehghani (Google Research), Neil Houlsby (Google Research)
ClassificationSegmentationDepth EstimationTransformerImageVideo
🎯 What it does: Trained and evaluated a Vision Transformer (ViT-22B) with up to 22 billion parameters, demonstrating its excellent performance across multiple tasks such as image classification, zero-shot learning, semantic segmentation, depth estimation, and video classification.
Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data
Minshuo Chen (Princeton University), Mengdi Wang (Princeton University)
Diffusion modelStochastic Differential Equation
🎯 What it does: This paper studies the score approximation, estimation, and distribution recovery problems of diffusion models on low-dimensional linear subspace data, providing theoretical sample complexity and convergence rates.
SDDM: Score-Decomposed Diffusion Models on Manifolds for Unpaired Image-to-Image Translation
Shikun Sun (Tsinghua University), Qi Tian (Huawei Cloud)
Image TranslationOptimizationDiffusion modelScore-based ModelImage
🎯 What it does: A fractional diffusion model for decomposition on manifolds (SDDM) is proposed for unpaired image-to-image translation.
SE(3) diffusion model with application to protein backbone generation
Jason Yim (Massachusetts Institute of Technology), Tommi S. Jaakkola
GenerationProtein Structure PredictionTransformerDiffusion modelScore-based ModelBiomedical Data
🎯 What it does: A diffusion model based on the SE(3) Lie group (FrameDiff) is proposed for unsupervised generation of protein backbone, balancing theoretical derivation and practical implementation.
Searching Large Neighborhoods for Integer Linear Programs with Contrastive Learning
Taoan Huang (University of Southern California), Benoit Steiner (Anthropic)
OptimizationGraph Neural NetworkContrastive LearningGraph
🎯 What it does: A CL-LNS method based on contrastive learning is proposed to learn a more efficient destroy sampling strategy in the LNS algorithm of ILP.
Second-Order Optimization with Lazy Hessians
Nikita Doikov (École Polytechnique Fédérale de Lausanne), Martin Jaggi (École Polytechnique Fédérale de Lausanne)
Optimization
🎯 What it does: This paper studies an algorithm that uses delayed Hessian updates in second-order optimization, proposing to reuse previous Hessians over multiple steps and incorporate new gradients to reduce computational complexity, along with providing global and local convergence analysis.
Second-order regression models exhibit progressive sharpening to the edge of stability
Atish Agarwala (Google DeepMind), Jeffrey Pennington (Google DeepMind)
OptimizationImage
🎯 What it does: This paper studies the phenomenon of asymptotic sharpening and the edge of stability in gradient descent with large step sizes by analyzing a second-order regression model (a model with quadratic parameters) and extends this model to high-dimensional cases, further conducting experimental validation on real neural networks.
Secure Federated Correlation Test and Entropy Estimation
Qi Pang (Carnegie Mellon University), Dawn Song (UC Berkeley)
Federated LearningSafty and PrivacyTabularBiomedical Data
🎯 What it does: This paper proposes the first federated correlation testing framework compatible with secure aggregation (FEDχ²) and the corresponding entropy estimation protocol (FED-H). By transforming correlation testing and entropy estimation into frequency moment estimation and using stable random projections to generate linearly aggregable encodings on the client side, it achieves federated statistical analysis while maintaining privacy.
SeedGNN: Graph Neural Network for Supervised Seeded Graph Matching
Liren Yu (Purdue University), Xiaojun Lin (Purdue University)
Graph Neural NetworkGraph
🎯 What it does: A supervised graph neural network SeedGNN is designed and implemented to match unlabeled graphs under the condition of having only a few seed nodes, and it learns to extract transferable matching knowledge from the training data.
SEGA: Structural Entropy Guided Anchor View for Graph Contrastive Learning
Junran Wu (Beihang University), Ke Xu (Beihang University)
Representation LearningGraph Neural NetworkContrastive LearningGraph
🎯 What it does: Proposes the SEGA framework, which utilizes structural entropy minimization to generate anchor views that preserve the basic information of the input graph, and applies it to graph contrastive learning.
SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation
Huaishao Luo (JD AI Research), Tianrui Li (Southwest Jiaotong University)
SegmentationTransformerContrastive LearningImageText
🎯 What it does: A SegCLIP model based on CLIP is proposed, utilizing learnable center aggregation of visual patches to achieve open vocabulary semantic segmentation.
Self-Attention Amortized Distributional Projection Optimization for Sliced Wasserstein Point-Cloud Reconstruction
Khai Nguyen (University of Texas at Austin), Nhat Ho
GenerationOptimizationTransformerPoint Cloud
🎯 What it does: This paper proposes a distributed projection optimization framework based on self-attention for point cloud reconstruction tasks.
Self-Interpretable Time Series Prediction with Counterfactual Explanations
Jingquan Yan (Rutgers University), Hao Wang (Rutgers University)
Explainability and InterpretabilityAuto EncoderTime SeriesBiomedical Data
🎯 What it does: A self-explanatory time series forecasting model called CounTS is proposed, which can generate executable and causally constrained actionable counterfactual explanations while maintaining prediction accuracy.
Self-Repellent Random Walks on General Graphs - Achieving Minimal Sampling Variance via Nonlinear Markov Chains
Vishwaraj Doshi (IQVIA Inc.), Do Young Eun (North Carolina State University)
Graph Neural NetworkGraphStochastic Differential EquationOrdinary Differential Equation
🎯 What it does: Construct a Self-Repellent Random Walk (SRRW) on any undirected graph, utilizing nonlinear Markov chains and full history visitation counts to achieve unbiased sampling of a given target distribution; prove that its dominant empirical distribution almost surely converges, and provide a central limit theorem, obtaining that the covariance matrix decreases with the parameter α.
Self-supervised learning of Split Invariant Equivariant representations
Quentin Garrido (Meta AI), Yann LeCun (New York University)
Representation LearningContrastive LearningImageBenchmark
🎯 What it does: A self-supervised learning framework is proposed that can simultaneously learn invariant and equivariant representations of images.
Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations
Weiwei Lin (Hong Kong Polytechnic University), Youzhi Tu (Hong Kong Polytechnic University)
ClassificationRecognitionRepresentation LearningTransformerSupervised Fine-TuningAudio
🎯 What it does: A self-supervised neural factor analysis (NFA) framework is proposed, utilizing K-means aligned Transformer features and a factor analysis model to separate sentence-level representations from speaker, emotion, language, and other information, thereby enhancing the performance of sentence-level speech tasks under unsupervised or few-label conditions.
SeMAIL: Eliminating Distractors in Visual Imitation via Separated Models
Shenghua Wan (Nanjing University), De-Chuan Zhan (Nanjing University)
Reinforcement LearningGenerative Adversarial NetworkVideo
🎯 What it does: This paper proposes SeMAIL, which uses a decoupled model to eliminate interference in visual imitation, based on the separation of task-related and unrelated dynamics for model learning and policy training.
Semi Bandit dynamics in Congestion Games: Convergence to Nash Equilibrium and No-Regret Guarantees.
Ioannis Panageas (University of California Irvine), Volkan Cevher (Ecole Polytechnique Federale de Lausanne)
OptimizationReinforcement LearningGraph
🎯 What it does: This paper proposes an online gradient descent algorithm with no regret and convergence to Nash equilibrium in a congestion game with semi-bandit feedback (SBGD-CE).
Semi-Autoregressive Energy Flows: Exploring Likelihood-Free Training of Normalizing Flows
Phillip Si (Cornell University), Volodymyr Kuleshov (Cornell University)
GenerationData SynthesisFlow-based ModelImageTabular
🎯 What it does: This paper proposes a training objective based on energy scores without the need for a Jacobian determinant, and designs a semi-autoregressive energy flow model that can train reversible networks without calculating the Jacobian determinant.
Semi-Dual Unbalanced Quadratic Optimal Transport: fast statistical rates and convergent algorithm.
Adrien Vacher (Universite Gustave Eiffel), François-Xavier Vialard (Universite Gustave Eiffel)
Optimization
🎯 What it does: This paper studies the semi-dual form of unbalanced quadratic optimal transport and provides upper and lower bounds for global Bregman divergence and statistical convergence rates.
Semi-Offline Reinforcement Learning for Optimized Text Generation
Changyu Chen (Renmin University of China), Rui Yan (Renmin University of China)
GenerationOptimizationTransformerSupervised Fine-TuningReinforcement LearningText
🎯 What it does: A semi-offline reinforcement learning framework (semi-offline RL) is proposed, achieving a smooth transition between offline and online while balancing exploration capability and optimization cost.
Semi-Parametric Contextual Pricing Algorithm using Cox Proportional Hazards Model
Young-Geun Choi (Sungkyunkwan University), Min-hwan Oh (Seoul National University)
Recommendation SystemOptimizationTabular
🎯 What it does: A new semi-parametric contextual pricing algorithm, called the CoxCP algorithm, is proposed. This algorithm is based on the Cox proportional hazards model and aims to maximize revenue through current contextual information and historical sales records.
Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories
Qinqing Zheng (Meta AI Research), Aditya Grover (University of California Los Angeles)
Reinforcement LearningSequentialBenchmark
🎯 What it does: A semi-supervised offline reinforcement learning framework SS-ORL is proposed, which uses a small number of labeled trajectories to train an inverse dynamics model to generate proxy labels for unlabeled trajectories, and then trains an offline RL policy using the complete dataset.
Semiparametrically Efficient Off-Policy Evaluation in Linear Markov Decision Processes
Chuhan Xie (Peking University), Zhihua Zhang (Peking University)
Reinforcement LearningTabular
🎯 What it does: This paper studies offline policy evaluation (OPE) in online reinforcement learning and proposes a semiparametric efficient estimation method under a known linear Markov decision process (MDP).
SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification
Pranjal Aggarwal (Indian Institute of Technology), Karthik R Narasimhan
ClassificationTransformerContrastive LearningText
🎯 What it does: Proposes the SemSup-XC model, which utilizes semantic descriptions for zero/few-shot inference in extreme classification.
Sequence Modeling with Multiresolution Convolutional Memory
Jiaxin Shi (Stanford University), Emily Fox
ClassificationGenerationConvolutional Neural NetworkTime SeriesSequentialElectrocardiogramBenchmark
🎯 What it does: Designed and implemented MultiresLayer, a sequence modeling module based on multi-resolution convolution, capable of efficiently capturing long-range dependencies.
Sequential Changepoint Detection via Backward Confidence Sequences
Shubhanshu Shekhar (Carnegie Mellon University), Aaditya Ramdas (Carnegie Mellon University)
Anomaly DetectionTime SeriesSequential
🎯 What it does: This paper proposes a general sequential change point detection framework based on Forward and Backward Confidence Sequences (BCS), which can convert any confidence sequence into an efficient change point detector.
Sequential Counterfactual Risk Minimization
Houssam Zenati (Criteo AI Lab), Pierre Gaillard (University of Grenoble Alpes)
OptimizationReinforcement Learning from Human FeedbackTabularSequential
🎯 What it does: A new Sequence Counterfactual Risk Minimization (SCRM) method is proposed, aimed at improving the handling of log-based feedback problems through multiple deployments of learning strategies.
Sequential Kernelized Independence Testing
Aleksandr Podkopaev (Carnegie Mellon University), Aaditya Ramdas (Amazon Research)
TabularTime Series
🎯 What it does: This paper proposes a sequential kernel independence testing method (SKIT) based on betting principles, which can monitor data streams at any time and decide whether to reject the independence hypothesis while controlling the type I error rate.
Sequential Monte Carlo Learning for Time Series Structure Discovery
Feras Saad, Vikash Mansinghka
OptimizationComputational EfficiencyTime Series
🎯 What it does: This paper proposes a full Bayesian structure learning framework based on Sequential Monte Carlo (SMC) and reversible MCMC for the automatic discovery of higher-order structures in time series models.
Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series
Aniruddh Raghu (Massachusetts Institute of Technology), Collin Stultz
Anomaly DetectionRepresentation LearningContrastive LearningMultimodalityTime SeriesBiomedical DataElectronic Health RecordsElectrocardiogram
🎯 What it does: A self-supervised pre-training framework for multimodal clinical time series is proposed.
Sequential Predictive Conformal Inference for Time Series
Chen Xu (Georgia Institute of Technology), Yao Xie (Georgia Institute of Technology)
Time SeriesSequential
🎯 What it does: A distribution-free confidence interval construction method for time series, SPCI, is proposed, which adaptively estimates the prediction interval using the time dependence of residuals through conditional quantile regression.