NeurIPS 2024 Papers — Page 32
Conference on Neural Information Processing Systems · 4035 papers
Robust and Faster Zeroth-Order Minimax Optimization: Complexity and Applications
Weixin An (Xidian University), Hongying Liu (Tianjin University)
OptimizationImage
🎯 What it does: A unified zero-order gradient descent extrapolated gradient ascent algorithm (ZO-GDEGA) is proposed for black-box non-convex-concave minimax problems.
Robust Conformal Prediction Using Privileged Information
Shai Feldman (Technion), Yaniv Romano (Technion)
Anomaly DetectionImageTabular
🎯 What it does: A method is proposed for providing a coverage-guaranteed prediction set even when the training samples are damaged (noise, missing, selective missing) — Privileged Conformal Prediction (PCP);
Robust Contrastive Multi-view Clustering against Dual Noisy Correspondence
Ruiming Guo (Sichuan University), Peng Hu (Sichuan University)
Representation LearningContrastive LearningImage
🎯 What it does: A robust clustering method called CANDY is proposed to address the issue of dual noise correspondence (i.e., false positives and false negatives) in contrastive multi-view clustering.
Robust Fine-tuning of Zero-shot Models via Variance Reduction
Beier Zhu (Nanyang Technological University), Hanwang Zhang (Nanyang Technological University)
Domain AdaptationOptimizationSupervised Fine-TuningContrastive LearningImage
🎯 What it does: A variance reduction fine-tuning (VRF) method based on sample distance is proposed, which assigns weights to each test sample using the zero-shot model failure set, thereby achieving sample-level fusion between the zero-shot model and the fine-tuned model.
Robust Gaussian Processes via Relevance Pursuit
Sebastian Ament (Meta), Eytan Bakshy (Meta)
Anomaly DetectionOptimizationTabularTime SeriesFinance Related
🎯 What it does: A Gaussian process regression method is proposed to achieve high robustness by learning the specific noise variance of samples, and sparse outlier detection is performed using the Relevance Pursuit algorithm;
Robust Graph Neural Networks via Unbiased Aggregation
Zhichao Hou (North Carolina State University), Xiaorui Liu (North Carolina State University)
Graph Neural NetworkGraph
🎯 What it does: A robust and unbiased graph signal estimator is proposed, which is expanded into RUNG layers to enhance the robustness of GNNs under adaptive attacks.
Robust group and simultaneous inferences for high-dimensional single index model
Weichao Yang (Beijing Normal University), Changliang Zou (Nankai University)
🎯 What it does: A robust group testing and multiple testing method is proposed for high-dimensional single index models (SIM), which maintains good statistical performance even in the presence of outliers or heavy-tailed errors.
Robust Mixture Learning when Outliers Overwhelm Small Groups
Daniil Dmitriev (ETH Zurich), Fanny Yang (ETH Zurich)
Anomaly DetectionOptimizationTabular
🎯 What it does: The study focuses on list-decodable mixed learning (LD-ML) in the presence of a large number of adversarial outliers, proposing a two-stage meta-algorithm that can output a short list containing all true means while being overwhelmed by outliers, along with optimal error guarantees.
Robust Neural Contextual Bandit against Adversarial Corruptions
Yunzhe Qi (University of Illinois), Jingrui He (University of Illinois)
Recommendation SystemOptimizationAdversarial AttackReinforcement LearningTabular
🎯 What it does: The R-NeuralUCB algorithm is proposed to resist adversarial reward corruption in the context of neural network multi-armed bandits.
Robust Offline Active Learning on Graphs
Yuanchen Wu (Pennsylvania State University), Yubai Yuan (Pennsylvania State University)
ClassificationOptimizationGraph Neural NetworkGraph
🎯 What it does: An offline graph structure-based active learning framework is proposed, which can simultaneously utilize network topology and node covariates to select the most informative nodes for labeling and recover the entire graph signal.
Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks
Andy Zhou (University of Illinois Urbana-Champaign), Haohan Wang (University of Illinois Urbana-Champaign)
OptimizationAdversarial AttackTransformerLarge Language ModelPrompt EngineeringTextBenchmark
🎯 What it does: Proposes Robust Prompt Optimization (RPO) to defend against jailbreak attacks on LLMs, generating transferable suffix tokens;
Robust Reinforcement Learning from Corrupted Human Feedback
Alexander Bukharin (Georgia Tech), Tuo Zhao (Amazon)
OptimizationRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningText
🎯 What it does: A robust RLHF method R M 3 is proposed, which can effectively learn the reward function and guide policy optimization even in the presence of sparse noise (mislabeling) in preference labels.
Robust Reinforcement Learning with General Utility
Ziyi Chen (University of Maryland), Heng Huang (University of Maryland)
OptimizationReinforcement Learning
🎯 What it does: A robust reinforcement learning framework with general utility is proposed, aimed at training robust policies in the worst-case environment.
Robust Sleep Staging over Incomplete Multimodal Physiological Signals via Contrastive Imagination
Qi Shen (Northeastern University), Zhiqiong Wang (Northeastern University)
ClassificationConvolutional Neural NetworkRecurrent Neural NetworkTransformerAuto EncoderContrastive LearningMultimodalityBiomedical Data
🎯 What it does: A robust multimodal sleep staging framework CIMSleepNet is proposed, capable of performing sleep staging in the absence of certain modalities.
Robust Sparse Regression with Non-Isotropic Designs
Chih-Hung Liu (National Taiwan University), Gleb Novikov (Lucerne School of Computer Science and Information Technology)
Optimization
🎯 What it does: A technique is proposed for designing efficient computable sparse linear regression estimators in the presence of two types of adversaries (unconscious and adaptive).
Robustly overfitting latents for flexible neural image compression
Yura Perugachi-Diaz (Vrije Universiteit Amsterdam), Sandjai Bhulai (Vrije Universiteit Amsterdam)
CompressionAuto EncoderImage
🎯 What it does: An extended method SGA+ is proposed to refine latent variables on a pre-trained neural image compression model to improve compression quality.
ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making
Woosung Kim (Korea University), Byung-Jun Lee (Korea University)
OptimizationReinforcement LearningTabularFinance Related
🎯 What it does: A framework for offline ROI maximization called ROIDICE is proposed, which optimizes the ratio of returns to cumulative costs using a fixed dataset.
RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions
Easton Knight Huch, Walter H. Dempsey (University of Michigan)
OptimizationReinforcement LearningBiomedical DataElectronic Health Records
🎯 What it does: In mobile health interventions, a robust mixed-effects contextual bandit algorithm called RoME is proposed to simultaneously address user heterogeneity, non-stationarity, and nonlinear relationships.
RoPINN: Region Optimized Physics-Informed Neural Networks
Haixu Wu (Tsinghua University), Mingsheng Long (Tsinghua University)
OptimizationTabularPhysics Related
🎯 What it does: This paper proposes the Region Optimization (RoPINN) training paradigm, which extends the optimization of PINN from discrete points to continuous neighborhoods, thereby enhancing the model's generalization and satisfaction of higher-order constraints.
Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching
Fernando Moreno-Pino (University of Oxford), Alvaro Cartea
ClassificationComputational EfficiencyTransformerTime SeriesSequentialOrdinary Differential Equation
🎯 What it does: This paper proposes the Rough Transformer (RFormer), a variant of Transformer that converts discrete time series into continuous time representations through path signatures, enabling efficient modeling under long sequences and irregular sampling conditions.
RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
Shuhao Chen (Southern University of Science and Technology), Yu Zhang (Southern University of Science and Technology)
OptimizationComputational EfficiencyTransformerLarge Language ModelContrastive LearningTextMultimodality
🎯 What it does: A query router called RouterDC based on dual contrastive learning is proposed for efficient selection and combination among multiple large language models (LLMs).
RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
Ziyao Zeng (Yale University), Alex Wong (Yale University)
Depth EstimationVision Language ModelImage
🎯 What it does: This paper studies the use of language descriptions to solve the scale ambiguity problem in monocular depth estimation, converting relative depth into absolute scale.
RTify: Aligning Deep Neural Networks with Human Behavioral Decisions
Yu-Ang Cheng (Brown University), Thomas Serre (Brown University)
ClassificationRecognitionConvolutional Neural NetworkRecurrent Neural NetworkImageSequential
🎯 What it does: The RTify framework is proposed, which allows recurrent neural networks to make decisions based on learnable thresholds after dynamically accumulating evidence, thereby predicting human reaction times (RT) while considering accuracy; a training method for the ideal observer model is also provided through self-punishment; further, a differentiable Wong-Wang (WW) decision circuit is embedded into CNNs to achieve RT fitting for natural image multi-classification tasks.
Rule Based Rewards for Language Model Safety
Tong Mu (OpenAI), Lilian Weng (OpenAI)
Safty and PrivacyReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: A rule-based reward (RBR) method is proposed, using AI feedback to replace a large amount of manual labeling to train the safe behavior of language models;
Rule Extrapolation in Language Modeling: A Study of Compositional Generalization on OOD Prompts
Anna Mészáros (University of Cambridge), Ferenc Huszár (University of Cambridge)
Recurrent Neural NetworkTransformerLarge Language ModelText
🎯 What it does: This study investigates the generalization of language models in the context of rule extrapolation, defining out-of-distribution (OOD) evaluation tasks using formal languages (regular, context-free, context-sensitive).
S-MolSearch: 3D Semi-supervised Contrastive Learning for Bioactive Molecule Search
Gengmo Zhou (Renmin University of China), Zhifeng Gao (Renmin University of China)
Drug DiscoveryContrastive LearningBiomedical Data
🎯 What it does: Developed the S-MolSearch framework, which combines 3D molecular structures and affinity information, using semi-supervised contrastive learning for molecular search.
S-SOS: Stochastic Sum-Of-Squares for Parametric Polynomial Optimization
Richard Licheng Zhu, Yuehaw Khoo (University of Chicago)
OptimizationTabularStochastic Differential Equation
🎯 What it does: This paper proposes the Stochastic Sum of Squares (S-SOS) algorithm for solving global polynomial optimization problems in the presence of random parameters, and provides its convergence properties and empirical validation.
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training
Yuezhou Hu (Tsinghua University), Jianfei Chen (Tsinghua University)
OptimizationComputational EfficiencyTransformerImageText
🎯 What it does: A new Sparse Transformer Estimation (S-STE) method is proposed for training a 2:4 sparse Transformer model from scratch.
S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
Xinyu Yang (Carnegie Mellon University), Beidi Chen (Carnegie Mellon University)
TransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper studies parameter-efficient fine-tuning of large-scale language models and proposes a Structured Sparse Fine-Tuning (S2FT) method.
S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning
Weihao Lin (Fudan University), Tao Chen (Fudan University)
OptimizationKnowledge DistillationTransformerImage
🎯 What it does: This study addresses and solves the discretization gap problem in differentiable mask pruning and proposes the Soft-to-Hard Pruner (S2HPruner) framework.
SA3DIP: Segment Any 3D Instance with Potential 3D Priors
Xi Yang (Xidian University), Xinbo Gao (Chongqing University of Posts and Telecommunications)
Object DetectionSegmentationPoint Cloud
🎯 What it does: The SA3DIP method is proposed, which generates fine-grained superpoints using geometric and color priors, and combines SAM 2D masks with a 3D detector for segmentation and merging, achieving unsupervised segmentation of arbitrary 3D instances.
Safe and Efficient: A Primal-Dual Method for Offline Convex CMDPs under Partial Data Coverage
Haobo Zhang (ShanghaiTech University), Xin Liu (ShanghaiTech University)
OptimizationReinforcement Learning from Human FeedbackReinforcement LearningTabularSequential
🎯 What it does: A primal-dual algorithm based on linear programming (POCC) is proposed to solve the offline convex constrained Markov decision process (convex CMDP) problem under the condition of partial data coverage.
Safe and Sparse Newton Method for Entropic-Regularized Optimal Transport
Zihao Tang (Shanghai University of Finance and Economics), Yixuan Qiu (Shanghai University of Finance and Economics)
OptimizationImage
🎯 What it does: The paper proposes a Safe Sparse Newton Method (SSNS) for solving large-scale entropy-regularized optimal transport problems.
Safe Exploitative Play with Untrusted Type Beliefs
Tongxin Li (Chinese University of Hong Kong), Adam Wierman (California Institute of Technology)
Reinforcement LearningTabularTime Series
🎯 What it does: This paper studies the trade-off between trust and distrust in beliefs about untrustworthy types in Bayesian games, providing upper and lower bounds for opportunities and risks in both normal form and stochastic Bayesian games, and proposing corresponding safe-exploitation strategies.
Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models
Chia-Yi Hsu (National Yang Ming Chiao Tung University), Chun-Ying Huang (National Yang Ming Chiao Tung University)
Safty and PrivacyTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper proposes Safe LoRA, a post-projection technique for LoRA fine-tuning that requires no training or data, maintaining the safety alignment of LLMs without compromising downstream performance.
Safe Time-Varying Optimization based on Gaussian Processes with Spatio-Temporal Kernel
Jialin Li (ETH Zurich), John Lygeros (ETH Zurich)
OptimizationTime Series
🎯 What it does: The TVSAFEOPT algorithm is proposed, which combines Bayesian optimization with spatiotemporal kernels to address the sequential decision-making problem with unknown time-varying rewards and safety constraints.
SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models
Linglan Zhao (Tencent), Weiran Huang
ClassificationRecognitionTransformerSupervised Fine-TuningImage
🎯 What it does: A SAFE framework is proposed based on pre-trained models, where a Slow Learner inherits general knowledge and is frozen, while a Fast Learner continuously learns new categories and prevents catastrophic forgetting, ultimately using entropy weighting to aggregate the outputs of both during inference.
Safety through feedback in Constrained RL
Shashank Reddy Chirra (Singapore Management University), Praveen Paruchuri (IIIT Hyderabad)
Autonomous DrivingSafty and PrivacyReinforcement Learning
🎯 What it does: This paper proposes an RLSF framework that utilizes trajectory-level feedback to learn a safety cost function and train a safety policy, reducing the burden of manual evaluation.
SafeWorld: Geo-Diverse Safety Alignment
Da Yin (University of California Los Angeles), Nanyun Peng (University of California Los Angeles)
Safty and PrivacyTransformerLarge Language ModelReinforcement LearningTextBenchmark
🎯 What it does: Proposes the SAFEWORLD benchmark to evaluate the safety dialogue capabilities of LLMs in the context of cross-regional cultural and legal diversity;
Saliency-driven Experience Replay for Continual Learning
Giovanni Bellitto (University of Catania), Concetto Spampinato (University of Catania)
ClassificationRecognitionConvolutional Neural NetworkImage
🎯 What it does: This paper proposes Saliency-driven Experience Replay (SER), which uses visual saliency prediction to assist classification models in continual learning, achieving robust learning for non-i.i.d. task sequences.
SAM-Guided Masked Token Prediction for 3D Scene Understanding
Zhimin Chen (Clemson University), Bing Li (Clemson University)
Object DetectionSegmentationKnowledge DistillationTransformerPoint Cloud
🎯 What it does: A pre-training framework based on SAM segmentation for 3D point cloud tokenization and two-stage mask token prediction is proposed, which can distill region-level knowledge from 2D foundational models (such as SAM, DINOv2/CLIP) into a 3D transformer network.
Samba: Severity-aware Recurrent Modeling for Cross-domain Medical Image Grading
Qi Bi (University of Amsterdam), Yefeng Zheng (Westlake University)
ClassificationDomain AdaptationRecurrent Neural NetworkImageBiomedical Data
🎯 What it does: A Severity-aware Recurrent Modeling (Samba) framework is proposed for cross-domain medical image grading tasks, which achieves better grading performance in unseen target domains after training in the source domain.
SAMPa: Sharpness-aware Minimization Parallelized
Wanyun Xie (École Polytechnique Fédérale de Lausanne), Volkan Cevher (École Polytechnique Fédérale de Lausanne)
OptimizationTransformerReinforcement LearningImageText
🎯 What it does: A parallelizable Sharpness-Aware Minimization method, SAMPa, is designed by introducing an auxiliary sequence y_t within SAM and using its gradient to achieve gradient parallel computation, resulting in a twofold speedup and improved model generalization performance.
SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification
Yanxin Yang (East China Normal University), Mingsong Chen (East China Normal University)
ClassificationObject DetectionConvolutional Neural NetworkDiffusion modelImage
🎯 What it does: A backdoor defense method called SampDetox is proposed for black-box environments, which eliminates various backdoor triggers by adding noise to the samples and then recovering them using a diffusion model, while maintaining the model's inference performance.
Sample and Computationally Efficient Robust Learning of Gaussian Single-Index Models
Puqian Wang (University of Wisconsin), Jelena Diakonikolas (University of Wisconsin)
OptimizationComputational Efficiency
🎯 What it does: Under high-dimensional Gaussian distribution, a robust learning algorithm with both sample and computational efficiency is proposed for learning single index models (SIM) in adversarial label noise models.
Sample Complexity of Algorithm Selection Using Neural Networks and Its Applications to Branch-and-Cut
Hongyu Cheng (Johns Hopkins University), Amitabh Basu (Johns Hopkins University)
OptimizationGraph Neural NetworkReinforcement LearningTabular
🎯 What it does: This study investigates the sample complexity of algorithm parameter selection using neural networks on unknown instance distributions and applies this framework to branch-and-bound pruning in mixed-integer linear programming to learn the selection strategy for CG and GMI cutting planes at the root node.
Sample Complexity of Interventional Causal Representation Learning
Emre Acartürk (Rensselaer Polytechnic Institute), Ali Tajer (Rensselaer Polytechnic Institute)
Representation LearningGraph
🎯 What it does: This paper presents a sample complexity analysis for causal representation learning (CRL) under finite sample conditions, providing PAC recovery upper bounds for graphical structures and latent variables.
Sample Complexity of Posted Pricing for a Single Item
Billy Jin (Cornell University), Sahil Singla (Georgia Tech)
🎯 What it does: This paper studies the sample complexity of using posted pricing mechanisms in the sale of a single item. It proves that under independent distributions, the sample complexity for welfare maximization is independent of the number of buyers, while for revenue maximization, a linear number of samples is required. It provides upper and lower bounds on sample complexity related to variable price points under correlated distributions.
Sample Complexity Reduction via Policy Difference Estimation in Tabular Reinforcement Learning
Adhyyan Narang (University of Washington), Kevin Jamieson (University of Washington)
Reinforcement LearningTabular
🎯 What it does: This paper addresses the pure exploration problem in discrete Markov decision processes and proposes a new low-variance estimator for estimating policy differences. Based on this, the PERP algorithm is designed, achieving nearly optimal instance-dependent sample complexity; comparative results in context are also provided.
Sample Efficient Bayesian Learning of Causal Graphs from Interventions
Zihan Zhou (Purdue University), Murat Kocaoglu (Purdue University)
Graph Neural NetworkGraph
🎯 What it does: A Bayesian method is proposed to learn causal graphs under limited intervention samples, along with a sample complexity analysis.
Sample Selection via Contrastive Fragmentation for Noisy Label Regression
Chris Dongjoo Kim (Seoul National University), Gunhee Kim (LG AI Research)
Convolutional Neural NetworkMixture of ExpertsContrastive LearningTabular
🎯 What it does: A framework named ConFrag is proposed for regression tasks, which segments the label space and constructs contrastive fragment pairs to train a set of expert feature extractors. It cleans samples based on neighborhood consistency, ultimately improving the performance of regression models in the presence of noisy labels.
Sample-Efficient Agnostic Boosting
Udaya Ghai (Amazon), Karan Singh (Carnegie Mellon University)
OptimizationTabular
🎯 What it does: In an agnostic learning scenario, a new boosting algorithm is proposed that significantly reduces sample complexity to (log|B|)/(γ ε³) through sample reuse and second-order potential function estimation while maintaining computational (weak learner invocation) complexity.
Sample-efficient Bayesian Optimisation Using Known Invariances
Theodore Brown (University College London), Ilija Bogunovic (University College London)
OptimizationTabularPhysics Related
🎯 What it does: This study investigates how to leverage known group invariance in Bayesian optimization to improve sample efficiency; it proposes embedding invariant kernels into Gaussian processes to achieve complete or partial invariance modeling and optimization of the objective function.
Sample-Efficient Constrained Reinforcement Learning with General Parameterization
Washim Uddin Mondal (Indian Institute of Technology Kanpur), Vaneet Aggarwal (Purdue University)
OptimizationReinforcement Learning
🎯 What it does: This paper proposes an accelerated primal-dual natural policy gradient (PD-ANPG) algorithm for solving constrained Markov decision processes (CMDP) with general parameterization.
Sample-Efficient Geometry Reconstruction from Euclidean Distances using Non-Convex Optimization
Ipsita Ghosh (University of North Carolina at Charlotte), Christian Kümmerle (University of North Carolina at Charlotte)
OptimizationComputational EfficiencyProtein Structure PredictionPoint CloudMesh
🎯 What it does: This paper aims to address the point embedding or geometric configuration problem given only Euclidean distance information, proposing an algorithm based on Iteratively Reweighted Least Squares (IRLS) to reconstruct geometric structures with a minimal number of samples.
Sample-Efficient Private Learning of Mixtures of Gaussians
Hassan Ashtiani (McMaster University), Shyam Narayanan (Citadel Securities)
OptimizationSafty and PrivacyMixture of Experts
🎯 What it does: The study investigates the sample complexity of learning high-dimensional Gaussian Mixture Models (GMM) under approximate differential privacy constraints, providing new theoretically optimal or near-optimal upper bounds.
SAND: Smooth imputation of sparse and noisy functional data with Transformer networks
Ju-Sheng Hong (University of California), Jane-Ling Wang (University of California)
TransformerTime Series
🎯 What it does: A SAND (Self-Attention on Derivatives) layer based on Transformer is proposed for smoothing interpolation of sparse and noisy functional data.
SARAD: Spatial Association-Aware Anomaly Detection and Diagnosis for Multivariate Time Series
Zhihao Dai (University of Warwick), Matthew Leeke (University of Birmingham)
Anomaly DetectionTransformerAuto EncoderTime Series
🎯 What it does: The SARAD method is proposed, which utilizes Transformer to learn the spatial correlations of multivariate time series and implements anomaly detection and diagnosis through the phenomenon of correlation drop.
SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale SAR Object Detection
Yuxuan Li (University College London), Jian Yang (Nankai University)
Object DetectionConvolutional Neural NetworkSupervised Fine-TuningImageBenchmark
🎯 What it does: This paper constructs the first COCO-level SAR target detection large-scale multi-class dataset SARDet-100K and proposes a multi-stage filtering enhancement pre-training framework (MSFA), while also open-sourcing the data and code.
Satformer: Accurate and Robust Traffic Data Estimation for Satellite Networks
Liang Qin (Xidian University), Huaxi Gu (Xidian University)
Graph Neural NetworkTransformerTime Series
🎯 What it does: A deep learning model named Satformer is proposed to accurately estimate the global traffic matrix of satellite networks from partially sampled data.
SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain
Pierre Colombo (Equall), Michael Desa (Equall)
Domain AdaptationTransformerLarge Language ModelMixture of ExpertsText
🎯 What it does: Developed and released two large-scale hybrid expert models specifically for the legal field, SaulLM-54B and SaulLM-141B.
SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learning
Paul Mangold (École polytechnique), Eric Moulines (École polytechnique)
Federated LearningReinforcement LearningTabular
🎯 What it does: This paper analyzes the sample and communication complexity of Federated Linear Stochastic Approximation (FedLSA), reveals the impact of heterogeneity on FedLSA, and proposes SCAFFLSA, an improved algorithm that reduces communication volume while maintaining linear acceleration through the control variate method.
Scalable and Effective Arithmetic Tree Generation for Adder and Multiplier Designs
Yao Lai (University of Hong Kong), Ping Luo (University of Hong Kong)
OptimizationReinforcement Learning
🎯 What it does: A reinforcement learning-based tree generation framework, ArithTreeRL, is proposed to automate the generation of efficient and compact arithmetic tree structures for adders and multipliers, and a comparative evaluation with existing technologies has been implemented.
Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes
Yunyue Wei (Tsinghua University), Yanan Sui (Tsinghua University)
OptimizationTabular
🎯 What it does: FocalBO is proposed, a hierarchical Bayesian optimization framework that combines Focalized Sparse Gaussian Processes (Focalized GP) to achieve efficient search on large-scale offline data and high-dimensional problems.
Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning
Lijun Zhang (Shanxi University), Jiye Liang (Shanxi University)
OptimizationSafty and PrivacyReinforcement LearningTabular
🎯 What it does: A scalable safe multi-agent reinforcement learning method called Scalable MAPPO-L is proposed, which achieves a balance between safety constraints and rewards in distributed training by utilizing local κ-hop interactions.
Scalable DBSCAN with Random Projections
HaoChuan Xu (University of Auckland), Ninh Pham (University of Auckland)
Computational EfficiencyTabular
🎯 What it does: Two scalable density clustering algorithms, sDBSCAN and sOPTICS, are proposed, which can efficiently cluster and visualize in high-dimensional space using cosine distance and various other distance metrics.
Scalable DP-SGD: Shuffling vs. Poisson Subsampling
Lynn Chua (Google Research), Chiyuan Zhang (Google Research)
OptimizationSafty and PrivacyTabular
🎯 What it does: This work proposes a privacy lower bound for the Adaptive Batch Linear Queries (ABLQ) mechanism under multi-round (persistent and dynamic) shuffling, and uses it to evaluate the privacy-efficiency gap of DP-SGD; it also provides a truncated Poisson sampling scheme that can be implemented on large-scale datasets.
Scalable Kernel Inverse Optimization
Youyuan Long (Delft University of Technology), Peyman Mohajerin Esfahani (Delft University of Technology)
OptimizationReinforcement Learning from Human FeedbackTabular
🎯 What it does: Proposes the Kernel Inverse Optimization (KIO) model and the Sequential Selection Optimization (SSO) algorithm, utilizing kernel methods to learn the objective function of expert decisions and achieve a scalable implementation of inverse optimization.
Scalable Neural Network Verification with Branch-and-bound Inferred Cutting Planes
Duo Zhou (University of Illinois Urbana-Champaign), Huan Zhang (University of Illinois Urbana-Champaign)
OptimizationComputational EfficiencyConvolutional Neural NetworkReinforcement LearningImageBenchmark
🎯 What it does: This paper proposes a technique for real-time generation and reinforcement of pruning planes during the branch-and-bound process of neural network verification (BICCOS), significantly improving lower bound accuracy and accelerating verification;
Scalable Optimization in the Modular Norm
Tim Large (Columbia University), Jeremy Bernstein (Massachusetts Institute of Technology)
OptimizationConvolutional Neural NetworkTransformerImageText
🎯 What it does: A modular norm is proposed and used to normalize arbitrary network weight updates, allowing basic optimizers like Adam and SGD to maintain a nearly constant learning rate as the network width and depth increase.
Scale Equivariant Graph Metanetworks
Ioannis Kalogeropoulos (National and Kapodistrian University of Athens), Yannis Panagakis (National and Kapodistrian University of Athens)
ClassificationMeta LearningGraph Neural NetworkImageGraph
🎯 What it does: This paper proposes a novel Scale Equivariant Graph MetaNetwork for handling the parameters of feedforward neural networks with different activation functions.
Scale-invariant Optimal Sampling for Rare-events Data and Sparse Models
Jing Wang (University of Connecticut), Hao Zhang
Tabular
🎯 What it does: This study investigates scale-invariant optimal sampling methods for rare event data under sparse models, proposing a sampling function based on prediction error and validating its performance in parameter estimation, variable selection, and prediction.
ScaleKD: Strong Vision Transformers Could Be Excellent Teachers
Jiawei Fan (Intel Labs China), Anbang Yao (Intel Labs China)
ClassificationKnowledge DistillationConvolutional Neural NetworkTransformerImage
🎯 What it does: A cross-architecture knowledge distillation method called ScaleKD is proposed, which utilizes a pre-trained Vision Transformer teacher model to transfer knowledge to student models including CNN, MLP, and different ViT structures, achieving scalability.
Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits
Gennaro Gala (Eindhoven University of Technology), Erik Quaeghebeur (Eindhoven University of Technology)
GenerationData SynthesisOptimizationImage
🎯 What it does: Proposed and implemented a scalable continuous latent variable model called Probabilistic Integral Circuits (PICs) in a DAG format, providing a complete pipeline for construction, learning, and scalable training.
Scaling Law for Time Series Forecasting
Jingzhe Shi (Tsinghua University), Lei Li (University of Copenhagen)
Convolutional Neural NetworkTransformerTime Series
🎯 What it does: A scaling law theory for time series forecasting is proposed, explaining the impact of dataset size, model complexity, and the granularity of time series data (especially the lookback period).
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
Alexander Hägele (École Polytechnique Fédérale de Lausanne), Martin Jaggi (École Polytechnique Fédérale de Lausanne)
TransformerLarge Language ModelText
🎯 What it does: This paper proposes and validates that using a constant learning rate combined with short-term cooldown instead of the traditional cosine learning rate scheduling in large-scale language model training can achieve performance comparable to or even better than cosine, without the need to pre-determine the number of training steps. Additionally, it introduces Stochastic Weight Averaging (SWA) and a Scheduler-Free Optimizer (SFO) to further enhance model quality during the training process.
Scaling laws for learning with real and surrogate data
Ayush Jain (Granica Computing Inc), Eren Sasoglu (Granica Computing Inc)
ClassificationRecommendation SystemConvolutional Neural NetworkRecurrent Neural NetworkSupervised Fine-TuningImageTabularBiomedical Data
🎯 What it does: This study investigates how to improve the model's test error on the original distribution by incorporating 'proxy data' from different sources when the original data is limited, and provides an optimal choice of the weight parameter α and a scaling law.
Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
Rafael Rafailov (Stanford University), Scott Niekum (UMass Amherst)
OptimizationReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: This paper studies the phenomenon of reward over-optimization in Direct Alignment Algorithms (DAAs), particularly its impact on the training of Large Language Models (LLMs). Through extensive empirical experiments, the authors explore the performance of DAAs under different objectives, training schemes, and model sizes.
Scaling Laws in Linear Regression: Compute, Parameters, and Data
Licong Lin (University of California Berkeley), Jason D. Lee (Princeton University)
Tabular
🎯 What it does: This paper analyzes the risk scaling law of one-shot SGD training under high-dimensional feature projection models (sketch) within the framework of infinite-dimensional linear regression. It proves that the dominant error term decreases according to a power law with respect to the model size M and sample size N, and that the variance error is suppressed to a higher order by the implicit regularization of SGD.
Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
Chaofan Tao (University of Hong Kong), Ngai Wong (University of Hong Kong)
Large Language ModelText
🎯 What it does: This study investigates the impact of vocabulary size on the scaling behavior of large language models (LLMs) by training models with parameters ranging from 33 million to 3 billion, using different vocabulary configurations to analyze how vocabulary size affects model performance.
Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
Lirui Wang (Massachusetts Institute of Technology), Kaiming He (Meta)
Robotic IntelligenceTransformerSupervised Fine-TuningVideoMultimodality
🎯 What it does: This study investigates how to utilize Heterogeneous Pre-trained Transformers (HPT) for policy learning across various robot postures and tasks, constructing a shareable backbone network and fine-tuning it for different tasks.
Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
Rulin Shao (University of Washington), Pang Wei Koh (Allen Institute for AI)
RetrievalOptimizationTextRetrieval-Augmented Generation
🎯 What it does: A 1.4 trillion token multi-domain retrieval memory, MASSIVEDS, was constructed, and the performance improvement and computational optimality of retrieval-based language models were systematically studied under different storage scales.
Scaling Sign Language Translation
Biao Zhang (Google DeepMind), Orhan Firat (Google DeepMind)
TransformerSupervised Fine-TuningVideoText
🎯 What it does: This paper constructs a scalable cross-language sign language translation model through large-scale pre-training and multi-task learning;
Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%
Lei Zhu (Peking University), Dong Chen (Microsoft Research Asia)
GenerationData SynthesisGenerative Adversarial NetworkImage
🎯 What it does: A new image quantization model VQGAN-LC is proposed, capable of expanding the codebook size to 100,000 while maintaining over 99% utilization.
Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
Tung Nguyen (University of California Los Angeles), Aditya Grover (University of California Los Angeles)
TransformerTime Series
🎯 What it does: This paper proposes a simple Transformer structure called Stormer, which achieves performance comparable to or even better than current state-of-the-art methods in weather prediction tasks, while significantly reducing training data and computational costs.
Scaling White-Box Transformers for Vision
Jinrui Yang (University of California Santa Cruz), Cihang Xie (University of California Santa Cruz)
ClassificationObject DetectionSegmentationRepresentation LearningTransformerContrastive LearningImage
🎯 What it does: This paper studies the scalability of the white-box Transformer CRATE in visual tasks and proposes the CRATEα model. By making three improvements to the sparse coding blocks of CRATE (overcomplete dictionary, decoupled dictionary, and residual connections) and an improved training recipe, the authors achieve significant performance enhancements as the model scales from Base to Huge and the data scales from ImageNet-21K to DataComp1B, while maintaining or even improving the model's interpretability.
Scanning Trojaned Models Using Out-of-Distribution Samples
Hossein Mirzaei (Sharif University of Technology), Mohammad Hossein Rohban (Sharif University of Technology)
Anomaly DetectionAdversarial AttackConvolutional Neural NetworkImageBenchmark
🎯 What it does: A general method called TRODO is proposed, which induces the model to misclassify out-of-vocabulary (OOV) samples as in-distribution (ID) by applying slight adversarial perturbations, thereby scanning for the presence of backdoors.
SCaR: Refining Skill Chaining for Long-Horizon Robotic Manipulation via Dual Regularization
Zixuan Chen (Nanjing University), Yang Gao (Nanjing University)
Robotic IntelligenceReinforcement LearningSequential
🎯 What it does: The SCaR framework is proposed to achieve smooth and stable execution of long-term robotic manipulation tasks through double regularization.
Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation
Yunnan Wang (Shanghai Jiao Tong University), Xin Jin (Ningbo Institute of Digital Twin, Eastern Institute of Technology)
GenerationData SynthesisGraph Neural NetworkDiffusion modelAuto EncoderImage
🎯 What it does: This paper proposes a scene graph-based image generation framework called DisCo, which can disentangle spatial layout and interaction semantics from textual scene graphs and generate diverse and relationship-compliant complex scene images through a diffusion model.
Scene Graph Generation with Role-Playing Large Language Models
Guikun Chen (Zhejiang University), Wenguan Wang (National Key Laboratory of Human-Machine Hybrid Augmented Intelligence)
GenerationRetrievalTransformerLarge Language ModelPrompt EngineeringVision Language ModelContrastive LearningImageTextMultimodalityPhysics Related
🎯 What it does: This paper studies Open Vocabulary Scene Graph Generation (OVSGG) and proposes a solution to the scene blind spots and misleading issues of traditional fixed prompt schemes by generating Scene-Specific Descriptions (SSD) through large language models (LLM) and combining them with the CLIP visual-language model.
SceneCraft: Layout-Guided 3D Scene Generation
Xiuyu Yang (Shanghai Jiao Tong University), Yu-Xiong Wang (University of Illinois Urbana-Champaign)
GenerationData SynthesisDiffusion modelScore-based ModelNeural Radiance FieldImage
🎯 What it does: Generate high-quality indoor scenes that conform to text descriptions and user-provided 3D layouts.
SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout
Chiyu Max Jiang (Waymo LLC), Dragomir Anguelov (Waymo LLC)
Data SynthesisAutonomous DrivingComputational EfficiencyTransformerLarge Language ModelPrompt EngineeringDiffusion modelPoint Cloud
🎯 What it does: This paper presents SceneDiffuser, a unified scene-level diffusion model for scene initialization and closed-loop rolling generation in traffic simulation.
Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing
Haonan Lin (Xi'an Jiaotong University), QianYing Wang
Image TranslationGenerationDiffusion modelImage
🎯 What it does: A new noise scheduling method called Logistic Schedule is proposed to improve the DDIM reverse process in text-guided image editing, enhancing content retention and editing quality.
Schrodinger Bridge Flow for Unpaired Data Translation
Valentin De Bortoli (Google DeepMind), Arnaud Doucet (Google DeepMind)
Image TranslationOptimizationDiffusion modelImageStochastic Differential Equation
🎯 What it does: A new Schrödinger bridge (Entropic Optimal Transport) solving algorithm is proposed, called α-Diffusion Schrödinger Bridge Matching (α-DSBM), which achieves online fitting without the need to train multiple DDM models.
Schur Nets: exploiting local structure for equivariance in higher order graph neural networks
QINGQI ZHANG, Risi Kondor (University of Chicago)
Drug DiscoveryGraph Neural NetworkGraph
🎯 What it does: This paper proposes a Schur network based on spectral graph theory, utilizing graph Laplacian spectral decomposition to construct automata for isomorphic equivariant linear mappings of subgraphs, thereby making fuller use of subgraph structures in higher-order graph neural networks.
Score Distillation via Reparametrized DDIM
Artem Lukoianov (Massachusetts Institute of Technology), Justin Solomon (Massachusetts Institute of Technology)
GenerationData SynthesisKnowledge DistillationDiffusion modelScore-based ModelImageText
🎯 What it does: Improved Score Distillation Sampling (SDS) in 3D shape generation, proposing the Score Distillation via Inversion (SDI) method.
Score-based 3D molecule generation with neural fields
Matthieu Kirchmeyer (Genentech), Saeed Saremi (Genentech)
GenerationDrug DiscoveryScore-based ModelNeural Radiance FieldPoint Cloud
🎯 What it does: This paper proposes a 3D molecular generation method called FuncMol based on continuous atomic occupancy fields, utilizing neural fields to encode molecular structures and perform score-based walk-jump sampling in the latent space.
Score-based generative models are provably robust: an uncertainty quantification perspective
Nikiforos Mimikos-Stamatopoulos (Universite Cote d'Azur), Markos Katsoulakis
GenerationScore-based Model
🎯 What it does: This paper provides a theoretical proof of the robustness of Score-based Generative Models (SGM) from the perspective of uncertainty quantification (UQ) and proposes the Wasserstein Uncertainty Propagation (WUP) theorem.
Score-Optimal Diffusion Schedules
Christopher Williams (University of Oxford), Saifuddin Syed (University of Oxford)
GenerationOptimizationDiffusion modelScore-based ModelImageOrdinary Differential Equation
🎯 What it does: An adaptive discretization scheduling algorithm based on Stein divergence is proposed to optimize the sampling process of diffusion models.
SCOREQ: Speech Quality Assessment with Contrastive Regression
Alessandro Ragano (University College Dublin), Andrew Hines (University College Dublin)
Contrastive LearningAudio
🎯 What it does: A contrastive regression loss function SCOREQ is proposed for speech quality prediction, which uses triplet loss to rank continuous MOS targets during training and constructs a generalizable low-dimensional representation of speech quality.