ICLR 2023 Papers — Page 13
International Conference on Learning Representations · 1573 papers
Revisiting Robustness in Graph Machine Learning
Lukas Gosch (Technical University of Munich), Stephan Günnemann (Technical University of Munich)
ClassificationAdversarial AttackGraph Neural NetworkGraph
🎯 What it does: This paper studies the robustness of graph neural networks in node classification tasks, proposing a definition of semantic-aware adversarial robustness and discovering the phenomenon of 'over-robustness'.
Revisiting the Assumption of Latent Separability for Backdoor Defenses
Xiangyu Qi (Princeton University), Prateek Mittal (Tsinghua University)
Anomaly DetectionAdversarial AttackConvolutional Neural NetworkImage
🎯 What it does: This paper proposes an adaptive backdoor attack suitable for only training data poisoning (poison-only), which can suppress the model's ability to distinguish between backdoor samples and normal samples in the latent space.
Revisiting the Entropy Semiring for Neural Speech Recognition
Oscar Chang (Google), Olivier Siohan (Google)
RecognitionKnowledge DistillationRecurrent Neural NetworkSupervised Fine-TuningAudio
🎯 What it does: In unaligned streaming speech recognition, the alignment distribution of CTC and RNN-T models is regularized and distilled using entropy semi-rings to improve recognition accuracy and pronunciation latency.
Revocable Deep Reinforcement Learning with Affinity Regularization for Outlier-Robust Graph Matching
Chang Liu (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
Anomaly DetectionOptimizationGraph Neural NetworkReinforcement LearningGraph
🎯 What it does: This paper proposes a reinforcement learning-based graph matching framework RGM, which employs serialized node matching and a revocable action mechanism to address the Lawler QAP problem with outliers.
Reward Design with Language Models
Minae Kwon (Stanford University), Dorsa Sadigh (Stanford University)
TransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: Using large language models (LLM) as proxy reward functions, users provide goals to the LLM through natural language prompts (containing a few examples or descriptions). The LLM gives reward signals based on the behavior in the RL episodes, thereby training the RL agent to achieve the user-specified behavior.
RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection
Shancong Mou (Georgia Institute of Technology), Jianjun Shi (Georgia Institute of Technology)
RestorationAnomaly DetectionGenerative Adversarial NetworkImage
🎯 What it does: Robust GAN-Inversion (RGI) and its improved version R-RGI are proposed to simultaneously restore images and identify damaged areas under unknown rough damage, achieving unmasked semantic repair and unsupervised pixel-level anomaly detection in a unified manner.
Rhino: Deep Causal Temporal Relationship Learning with History-dependent Noise
Wenbo Gong (Microsoft Research), Nick Pawlowski (Microsoft Research)
Time SeriesSequential
🎯 What it does: A temporal causal relationship learning framework named Rhino is proposed, which can simultaneously handle nonlinear relationships, instantaneous effects, and historical dependency noise.
Riemannian Metric Learning via Optimal Transport
Christopher Scarvelis (Massachusetts Institute of Technology), Justin Solomon (Massachusetts Institute of Technology)
OptimizationNeural Architecture SearchTime SeriesBiomedical Data
🎯 What it does: This study investigates a model based on optimal transport to learn a variable Riemannian metric tensor through cross-temporal sample probability distributions, and uses the learned metric to improve trajectory inference.
Risk-Aware Reinforcement Learning with Coherent Risk Measures and Non-linear Function Approximation
Thanh Lam (National University of Singapore), Patrick Jaillet (Massachusetts Institute of Technology)
Reinforcement LearningTime SeriesFinance Related
🎯 What it does: This paper proposes a risk-aware reinforcement learning framework for unknown MDPs, aiming to minimize the risk of low rewards, using consistent risk measures combined with nonlinear function approximation.
RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch
Yiqin Tan (Tsinghua University), Longbo Huang (Tsinghua University)
Reinforcement LearningSequential
🎯 What it does: A framework called RLx2 has been developed to train sparse deep reinforcement learning models from scratch, achieving almost no performance loss at high sparsity rates.
Robust Active Distillation
Cenk Baykal (Google Research), Erik Vee (Google Research)
ClassificationKnowledge DistillationConvolutional Neural NetworkImage
🎯 What it does: A robust active distillation method (RAD) is proposed, which formulates the balance between sample information and teacher accuracy in querying soft labels through game theory, addressing the dual challenges of query efficiency and confirmation bias in large-scale knowledge distillation.
Robust Algorithms on Adaptive Inputs from Bounded Adversaries
Yeshwanth Cherapanamjeri (University of California Berkeley), Samson Zhou (Rice University)
OptimizationSafty and Privacy
🎯 What it does: For adversarial inputs with limited sparsity or interaction capabilities, algorithms have been proposed to achieve robustness and efficiency in tasks such as dynamic least squares regression, adaptive distance estimation, and kernel density estimation. By dynamically maintaining sparse label updates and adaptive querying in a centralized environment, algorithms have been constructed to maintain high accuracy in adversarial settings.
Robust and Controllable Object-Centric Learning through Energy-based Models
Ruixiang ZHANG, Liam Paull (Nvidia Research)
Object DetectionSegmentationTransformerImage
🎯 What it does: An unsupervised object-centric learning framework EGO based on the energy-based model (EBM) is proposed, which constructs a permutation-invariant energy function using the Transformer attention module and infers object latent variables through gradient Markov Chain Monte Carlo (Langevin MCMC), automatically achieving object segmentation and reconstruction.
Robust Explanation Constraints for Neural Networks
Matthew Robert Wicker (Alan Turing Institute), Adrian Weller (Alan Turing Institute)
Explainability and InterpretabilityAdversarial AttackTabularBiomedical Data
🎯 What it does: A framework based on interval propagation is proposed, which can provide upper bounds for the gradient explanations of neural networks under input or parameter perturbations, thereby achieving formal certification of explanation robustness and incorporating this constraint into training to obtain verifiably robust models.
Robust Fair Clustering: A Novel Fairness Attack and Defense Framework
Anshuman Chhabra (University of California), Hongfu Liu (Brandeis University)
OptimizationAdversarial AttackContrastive LearningTabular
🎯 What it does: A black-box fairness attack and robust defense framework for fair clustering is proposed.
Robust Graph Dictionary Learning
Weijie Liu (Zhejiang University), Hui Qian (Zhejiang University)
OptimizationGraph Neural NetworkGraph
🎯 What it does: A robust graph dictionary learning framework (RGDL) is proposed, with the core being the definition and efficient computation of the robust Gromov-Wasserstein distance (RGWD) to measure the differences between graphs with structural noise, and to learn a graph dictionary that can represent noisy graphs using this distance.
Robust Multivariate Time-Series Forecasting: Adversarial Attacks and Defense Mechanisms
Linbo Liu (Amazon Web Services), Luke Huan (Amazon Web Services)
OptimizationAdversarial AttackRecurrent Neural NetworkTime Series
🎯 What it does: This paper studies the robustness of multivariate probabilistic prediction models under adversarial attacks and proposes a sparse indirect attack method along with two defense mechanisms.
Robust Scheduling with GFlowNets
David W Zhang, Roberto Bondesan (Qualcomm AI Research)
OptimizationGraph Neural NetworkReinforcement LearningGraph
🎯 What it does: To address the computation graph scheduling problem, the authors propose a generative method based on GFlowNet, which first trains on a proxy to generate diverse candidate scheduling schemes, and then evaluates a small number of the best schemes on the target hardware, thereby enhancing robustness against proxy errors.
Robustness to corruption in pre-trained Bayesian neural networks
Xi Wang (University of Massachusetts Amherst), Laurence Aitchison (University of Bristol)
ClassificationOptimizationConvolutional Neural NetworkContrastive LearningImage
🎯 What it does: This paper proposes ShiftMatch, a training data-related likelihood aimed at enhancing the robustness of Bayesian neural networks against input corruption.
ROCO: A General Framework for Evaluating Robustness of Combinatorial Optimization Solvers on Graphs
Han Lu (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
OptimizationAdversarial AttackGraph Neural NetworkReinforcement LearningGraph
🎯 What it does: This paper proposes the ROCO framework to evaluate the robustness of combinatorial optimization solvers on graphs when faced with perturbations. It modifies problem instances using the 'no bad optimal cost' constraint, which has no optimal solution, and generates hard instances using black-box attackers such as reinforcement learning. Systematic experiments are conducted on four classic CO tasks (DAG scheduling, ATSP, maximum coverage, MCSCC) with various solvers (traditional heuristics, learning-based, MILP).
RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Data
Sangwoo Mo (KAIST), Sean Bell (Meta AI)
Representation LearningContrastive LearningImage
🎯 What it does: This paper studies a robust semi-supervised learning method called RoPAWS, which can train on unorganized unlabeled data and avoids excessive confidence of PAWS on OOD samples.
ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
Olga Golovneva (Meta AI Research), Asli Celikyilmaz (Meta AI Research)
Explainability and InterpretabilityTransformerLarge Language ModelText
🎯 What it does: Developed the ROSCOE evaluation suite for unsupervised and interpretable assessment of step-by-step reasoning steps generated by language models.
Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction
Shitong Luo (Helixon Research), Jianzhu Ma (Institute for AI Industry Research)
Protein Structure PredictionTransformerFlow-based ModelBiomedical Data
🎯 What it does: An unsupervised rotamer density estimator (RDE) was constructed, and the entropy estimation of this model was used to predict the impact of mutations on binding free energy in protein-protein interactions.
RPM: Generalizable Multi-Agent Policies for Multi-Agent Reinforcement Learning
Wei Qiu (Nanyang Technological University), Zhongwen Xu (Sea AI Lab)
Reinforcement LearningBenchmark
🎯 What it does: The Ranked Policy Memory (RPM) method is proposed to enhance the generalization ability in unseen agent behavior scenarios in multi-agent reinforcement learning by saving and randomly sampling historical policies of different performance levels.
S-NeRF: Neural Radiance Fields for Street Views
Ziyang Xie (Fudan University), Li Zhang (Fudan University)
GenerationAutonomous DrivingNeural Radiance FieldVideoPoint Cloud
🎯 What it does: A novel NeRF model suitable for street scenes (S-NeRF) is proposed, capable of simultaneously reconstructing high-quality new views of large-scale backgrounds and moving foreground vehicles.
Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-Free RL
Ruiquan Huang (Pennsylvania State University), Yingbin Liang (Ohio State University)
Safty and PrivacyReinforcement Learning
🎯 What it does: A safe reward-free reinforcement learning framework (Safe RF-RL) is proposed, which must satisfy safety constraints during both the reward-agnostic exploration phase and the planning phase, along with a unified SWEET algorithm.
Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation
Yannick Hogewind (Radboud University), Nils Jansen (Radboud University)
Reinforcement LearningImage
🎯 What it does: A safe reinforcement learning framework called Safe SLAC is proposed in partially observable environments with high-dimensional pixel observations, utilizing a stochastic latent variable model and a safety critic to achieve constrained learning.
SAM as an Optimal Relaxation of Bayes
Thomas Möllenhoff (RIKEN Center for Advanced Intelligence Project), Mohammad Emtiyaz Khan (RIKEN Center for Advanced Intelligence Project)
OptimizationConvolutional Neural NetworkImage
🎯 What it does: This paper establishes a theoretical connection between Sharpness-Aware Minimization (SAM) and Bayesian objectives through Fenchel dual conjugate, proposing a Relaxed-Bayes objective based on this connection, and designs an Adam-style algorithm bSAM that can automatically learn uncertainty estimates.
Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networks
Xiang Ji (Princeton University), Tuo Zhao (Georgia Institute of Technology)
Convolutional Neural NetworkReinforcement LearningTabular
🎯 What it does: This paper studies the problem of non-parametric offline policy evaluation using deep convolutional neural networks, analyzing the ability of the deep fitted Q evaluation method to estimate the expected cumulative reward of the target policy based on data generated by an unknown behavior policy.
Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier
Pierluca D'Oro (Mila), Aaron Courville (Mila)
Reinforcement LearningSequential
🎯 What it does: By periodically resetting (fully or partially) the parameters of the reinforcement learning model, the replay ratio's scalability is enhanced, significantly improving sample efficiency.
Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions
Sitan Chen (University of California), Anru Zhang
GenerationData SynthesisDiffusion modelScore-based Model
🎯 What it does: Under the minimal assumption, theoretical convergence guarantees for score-based generative models (SGM) are provided under L₂ accurate score estimation, proving that it can efficiently sample any non-logarithmically compressible target distribution;
Sampling with Mollified Interaction Energy Descent
Lingxiao Li (Massachusetts Institute of Technology), Justin Solomon (Massachusetts Institute of Technology)
OptimizationTabular
🎯 What it does: An optimized sampling method called Fuzzy Interaction Energy Descent (MIED) is proposed, which approximates the target distribution by minimizing the Fuzzy Interaction Energy (MIE), applicable to both unconstrained and constrained sampling problems.
Sampling-based inference for large linear models, with application to linearised Laplace
Javier Antoran, José Miguel Hernández-Lobato (University of Cambridge)
ClassificationRestorationOptimizationConvolutional Neural NetworkSupervised Fine-TuningImageComputed Tomography
🎯 What it does: A Bayesian inference and hyperparameter selection method based on sampled EM is proposed for large-scale linear models, particularly for quantifying uncertainty in neural networks within linearized Laplace approximations.
Sampling-free Inference for Ab-Initio Potential Energy Surface Networks
Nicholas Gao (Technical University of Munich), Stephan Günnemann (Technical University of Munich)
Graph Neural NetworkReinforcement LearningPhysics Related
🎯 What it does: Proposes the PlaNet framework, which utilizes intermediate noise energy to train a surrogate during the VMC training process of PESNet++, achieving unsampled potential energy surface inference and improving the neural wave function architecture of PESNet++.
Scaffolding a Student to Instill Knowledge
Anil Kag (Boston University), Venkatesh Saligrama (Boston University)
ClassificationKnowledge DistillationConvolutional Neural NetworkImage
🎯 What it does: A new knowledge distillation method called DiSK is proposed, which filters difficult and easy samples during the student training process using a guiding function generated by the teacher, achieving more effective knowledge transfer in scenarios with significant capacity differences between the student and teacher.
Scalable and Equivariant Spherical CNNs by Discrete-Continuous (DISCO) Convolutions
Jeremy Ocampo (Kagenova Limited), Jason McEwen
SegmentationDepth EstimationConvolutional Neural NetworkImage
🎯 What it does: This paper proposes the DISCO spherical convolutional network, which combines rotational equivariance with high efficiency and scalability, achieving dense prediction tasks such as semantic segmentation and depth estimation on high-resolution panoramic images.
Scalable Batch-Mode Deep Bayesian Active Learning via Equivalence Class Annealing
Renyu Zhang (University of Chicago), Yuxin Chen (University of Chicago)
ClassificationOptimizationConvolutional Neural NetworkReinforcement LearningImage
🎯 What it does: This paper proposes a scalable batch deep Bayesian active learning framework called Batch‑BALANCE, which utilizes equivalence class annealing and decision theory to obtain acquisition functions. It can select high-information samples through greedy selection in small batches and efficiently construct query batches through clustering and random sampling in large batches.
Scalable Subset Sampling with Neural Conditional Poisson Networks
Adeel Pervez (University of Amsterdam), Efstratios Gavves (University of Amsterdam)
ClassificationOptimizationComputational EfficiencyImageText
🎯 What it does: A differentiable subset sampling method based on conditional Poisson sampling (NCPSS) is proposed, which significantly reduces variance while maintaining the expected subset size through iterative Poisson sampling, making it suitable for large-scale subset sampling tasks.
Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernel
SungYub Kim, Eunho Yang (Korea Advanced Institute of Science and Technology)
Tabular
🎯 What it does: This paper proposes a PAC-Bayes generalization bound that is invariant to scale transformations, and based on this, introduces the Connectivity Tangent Kernel (CTK) and Connectivity Sharpness (CS), while also proposing the Connectivity Laplace (CL) Bayesian network.
SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency
Junfeng Guo (University of Texas at Dallas), Cong Liu (Lehigh University)
Anomaly DetectionConvolutional Neural NetworkImage
🎯 What it does: A black-box input-level backdoor detection method called SCALE-UP is proposed, which uses scaled prediction consistency to determine whether a sample has been implanted with a backdoor.
Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting
Mohammad Amin Shabani (Simon Fraser University), Tristan Sylvain (Borealis AI)
TransformerTime Series
🎯 What it does: This paper proposes Scaleformer, a multi-scale iterative refinement framework built on top of the Transformer to enhance the accuracy of time series forecasting.
Scaling Forward Gradient With Local Losses
Mengye Ren (New York University), Geoffrey Hinton (Google)
ClassificationOptimizationImage
🎯 What it does: This study proposes a backpropagation-free learning algorithm based on forward gradients (activity perturbation).
Scaling Laws for a Multi-Agent Reinforcement Learning Model
Oren Neumann (Institute for Theoretical Physics Goethe University Frankfurt), Claudius Gros (Institute for Theoretical Physics Goethe University Frankfurt)
Reinforcement Learning
🎯 What it does: This study investigates the power-law scaling relationship between performance during the training process of AlphaZero and the number of model parameters and available computational resources, deriving a formula for the optimal network size under a given computational budget.
Scaling Laws For Deep Learning Based Image Reconstruction
Tobit Klug (Technical University of Munich), Reinhard Heckel (Technical University of Munich)
RestorationSuper ResolutionConvolutional Neural NetworkTransformerImageMagnetic Resonance Imaging
🎯 What it does: This study investigates the performance scaling laws of deep learning image reconstruction models under different training set sizes and provides a theoretical analysis to explain these laws.
Scaling Pareto-Efficient Decision Making via Offline Multi-Objective RL
Baiting Zhu (University of California), Aditya Grover (University of California)
OptimizationTransformerReinforcement LearningSequentialBenchmark
🎯 What it does: A research framework for offline multi-objective reinforcement learning is proposed, and the Pareto front is approximated through the design of a new dataset D4MORL and the algorithm PEDA.
Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation
Linfeng Zhao (Northeastern University), Lawson L.S. Wong (Northeastern University)
OptimizationRobotic IntelligenceConvolutional Neural NetworkReinforcement LearningSequential
🎯 What it does: This paper proposes a differentiable planning method based on implicit differentiation (IDP), which decouples forward evaluation and backward gradient computation by applying implicit differentiation to the Bellman fixed-point equation, enabling end-to-end learning for scalable path planning networks (such as VIN, SymVIN, ConvGPPN) and large-scale maps.
Scaling Up Probabilistic Circuits by Latent Variable Distillation
Anji Liu (University of California), Guy Van den Broeck (University of California)
OptimizationKnowledge DistillationTransformerLarge Language ModelImageText
🎯 What it does: A method called 'Latent Variable Distillation' (LVD) is proposed, which uses deep generative models to assign values to latent variables in probabilistic circuits (PC), thereby enabling scalable training of PCs.
Scenario-based Question Answering with Interacting Contextual Properties
Haitian Sun (Carnegie Mellon University), Ruslan Salakhutdinov (Carnegie Mellon University)
TransformerLarge Language ModelText
🎯 What it does: A three-module model named T-Reasoner is proposed for identifying answers and reasoning about missing conditions in scene-based question answering.
Schema Inference for Interpretable Image Classification
Haofei Zhang (Zhejiang University), Mingli Song (Shanghai Institute for Advanced Study, Zhejiang University)
ClassificationExplainability and InterpretabilityGraph Neural NetworkTransformerImage
🎯 What it does: This paper proposes the reasoning paradigm of schema inference, achieving interpretable reasoning for image classification through the construction of instance graphs (IR-Graph) and category graphs (IR-Atlas) for graph matching.
SCoMoE: Efficient Mixtures of Experts with Structured Communication
zhiyuan zeng, Deyi Xiong (Tianjin University)
OptimizationComputational EfficiencyMixture of ExpertsText
🎯 What it does: Proposes the SCoMoE structured MoE model, which reduces inter-node communication volume and improves training speed by slicing/projecting along the sequence or feature dimensions, dividing communication into three layers: within the accelerator, within the node, and between nodes;
Score-based Continuous-time Discrete Diffusion Models
Haoran Sun (Georgia Institute of Technology), Hanjun Dai (Google Research)
GenerationData SynthesisTransformerDiffusion modelScore-based ModelImageAudio
🎯 What it does: A continuous-time score-based diffusion model (SDDM) is proposed in the discrete category space, achieving reverse denoising through jump processes and continuous-time Markov chains;
SeaFormer: Squeeze-enhanced Axial Transformer for Mobile Semantic Segmentation
Qiang Wan (Fudan University), Li Zhang (Fudan University)
ClassificationSegmentationTransformerImage
🎯 What it does: A compressed enhanced axial transformer (SeaFormer) for mobile semantic segmentation is proposed, achieving end-to-end segmentation and classification tasks through lightweight design;
Searching Lottery Tickets in Graph Neural Networks: A Dual Perspective
Kun Wang (University of Science and Technology of China), Yang Wang (University of Science and Technology of China)
ClassificationExplainability and InterpretabilityComputational EfficiencyGraph Neural NetworkGraph
🎯 What it does: A Dual Graph Lottery Ticket (DGLT) framework is proposed, which utilizes hierarchical graph sparsification and gradually increased regularization to transform a randomly initialized complete graph neural network into a highly sparse, high-performance, and easily interpretable graph lottery ticket.
Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning
Xin-Qiang Cai (Nanjing University), Zhi-Hua Zhou (Nanjing University)
Robotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningGenerative Adversarial NetworkImageSequential
🎯 What it does: Proposes a Heterogeneous Observable Imitation Learning (HOIL) framework and designs an Importance Weighting and Rejection (IWRE) algorithm to address the situation where the observation spaces of experts and learners do not overlap and expert observations are limited.
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
Antonia Creswell (DeepMind), Irina Higgins (DeepMind)
Explainability and InterpretabilityTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: A Selection-Inference (SI) framework is proposed, which conducts multi-step logical reasoning in a hierarchical recursive manner using a pre-trained large language model, outputting causally interpretable reasoning trajectories through a selection-inference two-step process.
Selective Annotation Makes Language Models Better Few-Shot Learners
Hongjin SU, Tao Yu (University of Washington)
ClassificationRetrievalTransformerLarge Language ModelPrompt EngineeringTextRetrieval-Augmented Generation
🎯 What it does: A two-step framework is proposed: first, selectively label the unlabeled data, and then during testing, retrieve these labeled samples as contextual examples for the large language model, significantly enhancing few-shot learning performance.
Selective Frequency Network for Image Restoration
Yuning Cui (Sun Yat-sen University), Alois Knoll (Technical University of Munich)
RestorationConvolutional Neural NetworkImage
🎯 What it does: A frequency-selective image restoration network SFNet is designed, utilizing dynamic multi-branch filtering and channel attention for frequency decomposition and re-weighting of features;
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Xuezhi Wang (Google Research), Denny Zhou (Google Research)
OptimizationTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
🎯 What it does: A self-consistency decoding strategy is proposed, which replaces greedy decoding in chain-of-thought prompts by sampling multiple reasoning paths and aggregates the final answer through voting.
Self-Distillation for Further Pre-training of Transformers
Seanie Lee (Korea Advanced Institute of Science and Technology), Kenji Kawaguchi (National University of Singapore)
ClassificationKnowledge DistillationTransformerAuto EncoderImageText
🎯 What it does: Proposes self-distillation as a regularization method for further pre-training to enhance the downstream task performance of pre-trained Transformers in the target domain.
Self-Ensemble Protection: Training Checkpoints Are Good Data Protectors
Sizhe Chen (Shanghai Jiao Tong University), Xiaolin Huang (Shanghai Jiao Tong University)
Safty and PrivacyAdversarial AttackConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: The study investigates how to add imperceptible perturbations to the training set to render unauthorized model training ineffective, proposing a protection method based on self-ensemble from training checkpoints (SEP).
Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning
Jiahui Gao (University of Hong Kong), Lingpeng Kong (University of Hong Kong)
ClassificationGenerationData SynthesisRecurrent Neural NetworkTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: This paper proposes a self-supervised noise-free data generation framework called SUNGEN, which improves the quality and efficiency of data generated by large-scale pre-trained language models (PLM) in zero-shot learning.
Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
Alex Damian (Princeton University), Jason D. Lee (Princeton University)
OptimizationConvolutional Neural NetworkTransformerImageText
🎯 What it does: Analyze the dynamics of gradient descent in the Edge of Stability phase, propose a self-stabilization mechanism, and provide precise predictions for loss and sharpness;
Self-Supervised Category-Level Articulated Object Pose Estimation with Part-Level SE(3) Equivariance
Xueyi Liu (Tsinghua University), Li Yi (Fudan University)
Pose EstimationPoint Cloud
🎯 What it does: This paper proposes an unsupervised, category-level decoupled joint object pose estimation framework that can self-learn and infer the poses and joint states of various components from unlabeled point clouds.
Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild
Kaifeng Zhang (Tsinghua University), Xiaolong Wang (UC San Diego)
Pose EstimationVideo
🎯 What it does: A fully self-supervised category-level 6D pose estimation method is proposed, which learns 3D geometric correspondences directly from real videos.
Self-supervised learning with rotation-invariant kernels
Léon Zheng (Valeo), Rémi Gribonval (Univ Lyon)
ClassificationRepresentation LearningConvolutional Neural NetworkContrastive LearningImage
🎯 What it does: A regularization loss based on rotation-invariant kernel mean embedding is proposed for self-supervised learning of image representations, and the SFRIK method is introduced.
Self-Supervised Set Representation Learning for Unsupervised Meta-Learning
Dong Bok Lee (Korea Advanced Institute of Science and Technology), Sung Ju Hwang (Korea Advanced Institute of Science and Technology)
Representation LearningMeta LearningTransformerContrastive LearningImage
🎯 What it does: A self-supervised set representation learning framework called Set-SimCLR is proposed to address the problem of unsupervised meta-learning.
Self-supervision through Random Segments with Autoregressive Coding (RandSAC)
Tianyu Hua (University of British Columbia), Leonid Sigal (University of British Columbia)
Representation LearningTransformerImage
🎯 What it does: A self-supervised visual pre-training method called RandSAC is proposed, which learns visual features using segment-wise autoregressive prediction with random segmentation.
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
Lorenz Kuhn (University of Oxford), Sebastian Farquhar (University of Oxford)
GenerationTransformerLarge Language ModelText
🎯 What it does: Proposes an unsupervised 'semantic entropy' method for estimating the uncertainty of generated text in large language models.
Semi-Implicit Variational Inference via Score Matching
Longlin Yu (Peking University), Cheng Zhang (Peking University)
Score-based ModelTabular
🎯 What it does: A semi-implicit variational inference method based on score matching (SIVI-SM) is proposed, which addresses the non-differentiable density issue in the semi-implicit variational family through a min-max framework.
Semi-Parametric Inducing Point Networks and Neural Processes
Richa Rastogi (Cornell University), Volodymyr Kuleshov (Cornell University)
Computational EfficiencyMeta LearningTransformerTabular
🎯 What it does: A semi-parametric induced point network (SPIN) is proposed, which can query the training set during inference and supports linear time complexity.
Semi-supervised Community Detection via Structural Similarity Metrics
Yicong Jiang (Harvard University), Tracy Ke (Harvard University)
Graph Neural NetworkGraph
🎯 What it does: This paper proposes a semi-supervised community detection method based on structural similarity, which predicts the community of new nodes by utilizing the connection patterns between labeled nodes and new nodes.
Semi-supervised learning with a principled likelihood from a generative model of data curation
Stoil Krasimirov Ganev (University of Bristol), Laurence Aitchison (University of Bristol)
ClassificationData-Centric LearningGenerative Adversarial NetworkImage
🎯 What it does: View semi-supervised learning (SSL) as the log-likelihood of a data cleaning generative model, proving that common low-density separation methods (entropy minimization, pseudo-labeling, FixMatch, etc.) are lower bounds of this log-likelihood, and based on this, propose a Bayesian SSL extension; experiments conducted on toy data, CIFAR-10, and Galaxy Zoo 2.
SemPPL: Predicting Pseudo-Labels for Better Contrastive Representations
Matko Bošnjak (DeepMind), Jovana Mitrovic (DeepMind)
Representation LearningConvolutional Neural NetworkContrastive LearningImage
🎯 What it does: A semi-supervised contrastive learning framework SEMPPL is proposed, which improves representation learning by predicting pseudo-labels using a small number of labeled samples to select semantically positive samples.
Sequential Attention for Feature Selection
Taisuke Yasuda (Carnegie Mellon University), Vahab Mirrokni (Google Research)
Tabular
🎯 What it does: Proposes the Sequential Attention algorithm, which gradually selects the most valuable feature subset during the training process of neural networks, while considering the marginal contributions of residual features.
Sequential Gradient Coding For Straggler Mitigation
Nikhil Krishnan Muralee Krishnan (Indian Institute of Technology Palakkad), Ashish J Khisti (University of Toronto)
Convolutional Neural NetworkRecurrent Neural NetworkImage
🎯 What it does: Two sequence gradient coding schemes (SR‑SGC and M‑SGC) were developed to mitigate stragglers and reduce computational load in distributed learning.
Sequential Latent Variable Models for Few-Shot High-Dimensional Time-Series Forecasting
Xiajun Jiang (Rochester Institute of Technology), Linwei Wang (Rochester Institute of Technology)
Meta LearningRecurrent Neural NetworkAuto EncoderTime SeriesSequentialPhysics RelatedOrdinary Differential Equation
🎯 What it does: A sequence latent variable model framework based on Bayesian meta-learning is proposed for long-term prediction of high-dimensional time series with few samples.
Sequential Learning of Neural Networks for Prequential MDL
Jorg Bornschein (DeepMind), Marcus Hutter (DeepMind)
CompressionOptimizationConvolutional Neural NetworkImage
🎯 What it does: This study investigates how to compute the prequential MDL description length in neural networks and proposes an efficient implementation based on online learning and replay.
Serving Graph Compression for Graph Neural Networks
Si Si (Google Research), Sanjiv Kumar (Google Research)
CompressionGraph Neural NetworkGraph
🎯 What it does: A compression method using virtual nodes (VNG) to replace the original training graph and features during GNN inference is proposed, significantly reducing storage requirements while maintaining inference quality.
SGDA with shuffling: faster convergence for nonconvex-PŁ minimax optimization
Hanseul Cho (Kim Jaechul Graduate School of Artificial Intelligence KAIST), Chulhee Yun (Kim Jaechul Graduate School of Artificial Intelligence KAIST)
Optimization
🎯 What it does: This paper studies the stochastic gradient descent ascent (SGDA) algorithm with random reshuffling (RR) sampling under non-convex-non-concave objective functions, and provides the global convergence rate under the Polyak-Łojasiewicz (PŁ) condition; it also presents the optimal lower bound for strongly convex-strongly concave problems.
Share Your Representation Only: Guaranteed Improvement of the Privacy-Utility Tradeoff in Federated Learning
Zebang Shen (ETH Zurich), Reza Shokri (National University of Singapore)
Federated LearningSafty and PrivacyRepresentation LearningConvolutional Neural NetworkSupervised Fine-TuningImage
🎯 What it does: The CENTAUR algorithm is proposed, which applies differential privacy (DP) only to the shared representation layer in federated learning, while each client retains a personalized classification head, thus achieving collaborative learning under DP constraints.
Sharper Bounds for Uniformly Stable Algorithms with Stationary Mixing Process
Shi Fu (University of Science and Technology of China), Dacheng Tao (JD.com)
Time SeriesSequential
🎯 What it does: This paper studies the generalization performance of learning algorithms with uniform stability under ψ-mixing processes for non-independent and identically distributed (i.i.d.) data such as time series, providing a high-probability generalization error upper bound of ˜O(1/√n); subsequently, this general result is applied to algorithms such as kernel regularization, stochastic gradient descent (SGD), and local iterative regularization, yielding corresponding risk upper bounds.
Short-Term Memory Convolutions
Grzegorz Stefański (Samsung Research Institute Poland), Artur Szumaczuk (Samsung Research Institute Poland)
SegmentationComputational EfficiencyConvolutional Neural NetworkAudio
🎯 What it does: This paper proposes Short-Term Memory Convolution (STMC) and its deconvolution variant, aimed at online temporal signal processing, significantly reducing the inference time, latency, and memory usage of convolutional neural networks.
Sign and Basis Invariant Networks for Spectral Graph Representation Learning
Derek Lim (Massachusetts Institute of Technology), Stefanie Jegelka (Massachusetts Institute of Technology)
Representation LearningDrug DiscoveryGraph Neural NetworkTransformerMeshGraph
🎯 What it does: This paper proposes two neural network architectures, SignNet and BasisNet, which can eliminate sign flips and basis invariance while processing graph Laplacian spectra, thereby enhancing graph representation learning performance.
SimPer: Simple Self-Supervised Learning of Periodic Targets
Yuzhe Yang (Massachusetts Institute of Technology), Daniel McDuff (Google)
Representation LearningConvolutional Neural NetworkContrastive LearningVideoTime Series
🎯 What it does: The SimPer framework is proposed, which utilizes a periodic self-supervised learning method to extract representations of periodic signals from unlabeled data.
Simple and Scalable Nearest Neighbor Machine Translation
Yuhan Dai (University of Science and Technology of China), Tong Xu (University of Science and Technology of China)
RetrievalTransformerTextRetrieval-Augmented Generation
🎯 What it does: This paper proposes a lightweight k-NN translation framework SK-MT based on dynamic retrieval, utilizing BM25 to retrieve sentence-level similar corpora and constructing a minimal retrieval library, and then adaptively fusing the k-NN results with a pre-trained NMT model through a distance-aware adapter.
Simple Emergent Action Representations from Multi-Task Policy Training
Pu Hua (Tsinghua University), Huazhe Xu (Tsinghua University)
Robotic IntelligenceReinforcement Learning
🎯 What it does: This paper achieves self-organizing representations of low-level actions through learning task embeddings (LTE) and perception embeddings (LSE) in a multi-task policy network, which can be directly used as high-level instructions.
Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth
Filipe de Avila Belbute-Peres (Carnegie Mellon University), J Zico Kolter
Data SynthesisOptimizationImageVideoPhysics RelatedAudio
🎯 What it does: This paper proposes a simplified version of the sine network (SSN), replacing the asymmetric initialization of SIREN with a unified Kaiming normal initialization. It proves from the perspective of neural tangent kernel (NTK) theory that the NTK of this network approximates a low-pass filter, with the bandwidth controlled by the hyperparameter ω. Based on this analysis, the author provides empirical rules for tuning ω and conducts experiments in implicit modeling (images, videos, audio, SDF, Helmholtz equation) and physics-informed neural networks (PINN: Burgers, Navier-Stokes, Schrödinger) tasks, demonstrating superior performance compared to traditional SIREN or tanh MLP.
SIMPLE: A Gradient Estimator for k-Subset Sampling
Kareem Ahmed (University of California Los Angeles), Guy Van den Broeck
OptimizationAuto EncoderImageTabular
🎯 What it does: A new gradient estimator SIMPLE is proposed for end-to-end training in discrete layers with k-subset sampling. This method obtains samples through precise discrete sampling in the forward pass and uses the gradient of the k-subset marginal probabilities to approximate the true gradient in the backward pass, thus avoiding traditional relaxations or high-variance estimates.
SIMPLE: Specialized Model-Sample Matching for Domain Generalization
Ziyue Li (ShanghaiTech University), Dongsheng Li (ShanghaiTech University)
Domain AdaptationImageBenchmark
🎯 What it does: The SIMPLE framework is proposed to achieve domain generalization through a pool of pre-trained models and sample-specific matching, without the need for fine-tuning.
simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing
Zitao Liu (Guangdong Institute of Smart Education), Weiqi Luo (Guangdong Institute of Smart Education)
Recommendation SystemTabularTime SeriesSequential
🎯 What it does: This paper proposes SIMPLEKT, a simple knowledge tracing baseline model based on the Rasch model and ordinary dot-product attention.
Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
Samuel Lavoie (Mila Université de Montréal), Aaron Courville (Mila Université de Montréal)
ClassificationRepresentation LearningContrastive LearningImage
🎯 What it does: This paper introduces the Simplicial Embeddings (SEM) module in self-supervised learning, projecting the encoder output onto L sparse vectors of dimension V, using softmax for discretization, and utilizing this representation in downstream classification tasks.
Simplicial Hopfield networks
Thomas F Burns, Tomoki Fukai (OIST Graduate University)
Image
🎯 What it does: This paper proposes an extension of the Hopfield network to include Simplicial Hopfield networks with higher-order set connections, and conducts theoretical and experimental analyses of its memory capacity and performance.
Simplified State Space Layers for Sequence Modeling
Jimmy T.H. Smith (Stanford University), Scott Linderman
Time SeriesSequentialAudio
🎯 What it does: A simplified state space layer S5 is proposed for modeling long sequences.
Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective
Raj Ghugare (Visvesvaraya National Institute of Technology Nagpur), Russ Salakhutdinov
Representation LearningReinforcement Learning
🎯 What it does: A unified low-order model reinforcement learning algorithm ALM is proposed, which optimizes the observation encoder, latent space dynamics model, and policy simultaneously using a single lower bound objective, allowing the three to be trained collaboratively under the same goal.
Single-shot General Hyper-parameter Optimization for Federated Learning
Yi Zhou, Heiko Ludwig
OptimizationFederated LearningHyperparameter SearchMeta LearningTabular
🎯 What it does: Proposed a single-iteration hyperparameter optimization framework FLoRA in a federated learning environment;
SketchKnitter: Vectorized Sketch Generation with Diffusion Models
Qiang Wang (Beijing University of Posts and Telecommunications), Yi-Zhe Song (University of Surrey)
GenerationRetrievalRecurrent Neural NetworkDiffusion modelImage
🎯 What it does: Using a diffusion model to treat sketches as an inverse deformation process of stroke points, training the model to recover recognizable vectorized sketches from noise.
SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models
Ziyi Wu (University of Toronto), Animesh Garg (Google Research)
Object DetectionObject TrackingGenerationData SynthesisTransformerVideoPhysics Related
🎯 What it does: We propose SlotFormer, a Transformer-based autoregressive model that learns physical dynamics from videos using unsupervised object slot representations and generates high-quality future frames over long time steps; simultaneously, we transfer this model to VQA and planning tasks, enhancing downstream performance.
SLTUNET: A Simple Unified Model for Sign Language Translation
Biao Zhang (University of Edinburgh), Rico Sennrich (University of Zurich)
TransformerText
🎯 What it does: A unified end-to-end sign language translation model SLTUNET is proposed, capable of simultaneously handling Sign2Gloss, Sign2Text, Gloss2Text, Text2Gloss (excluded), and machine translation tasks, achieving multi-task joint training.
SMART: Self-supervised Multi-task pretrAining with contRol Transformers
Yanchao Sun (University of Maryland), Ashish Kapoor (Microsoft)
Robotic IntelligenceTransformerReinforcement LearningSequential
🎯 What it does: This paper proposes a self-supervised multi-task pre-training framework called SMART, which utilizes a Control Transformer for pre-training on multi-domain continuous control tasks, significantly improving learning efficiency and performance in downstream imitation learning or reinforcement learning tasks.
SMART: Sentences as Basic Units for Text Evaluation
Reinald Kim Amplayo (Google Research), Shashi Narayan (Google Research)
GenerationText
🎯 What it does: A text evaluation metric named SMART is proposed, which takes sentences as the basic unit of matching and uses sentence-level soft matching to evaluate generated text.