ICLR 2023 Papers — Page 14
International Conference on Learning Representations · 1573 papers
SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing
Sheng Li (University of Pittsburgh), Xulong Tang (Northeastern University)
OptimizationComputational EfficiencyConvolutional Neural NetworkSupervised Fine-TuningImageText
🎯 What it does: A layer freezing framework based on attention, SmartFRZ, has been designed and implemented to automatically freeze layers that do not need to be updated during training, significantly reducing training costs.
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Andy Zeng (Google), Pete Florence (Google)
GenerationRetrievalTransformerLarge Language ModelPrompt EngineeringVision Language ModelImageVideoTextMultimodality
🎯 What it does: This study investigates the construction of a Socratic Models system that can be directly used in multimodal reasoning tasks (such as image description, video retrieval, perspective understanding, etc.) by combining different pretrained multimodal models (such as VLM, LLM, ALM) in a zero-shot manner using language as an intermediate representation and prompt engineering.
Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learning
Chongjian GE, Ping Luo (University of Hong Kong)
Object DetectionSegmentationRepresentation LearningTransformerContrastive LearningImage
🎯 What it does: This paper proposes Soft Neighbor Contrastive Learning (SNCLR), which improves traditional binary instance discrimination contrastive learning by introducing a candidate neighbor set and a cross-attention mechanism to assign soft weights (positive support) to different instances.
Softened Symbol Grounding for Neuro-symbolic Systems
Zenan Li (Nanjing University), Jian L\"{u}
Reinforcement LearningSequential
🎯 What it does: A neural-symbolic learning framework is proposed to soften symbolic grounding, modeling symbolic states as a Boltzmann distribution, and achieving efficient sampling through projection MCMC and SMT solvers, gradually jointly training neural networks and symbolic constraints.
SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning
Hao Chen (Carnegie Mellon University), Marios Savvides (Carnegie Mellon University)
ClassificationData-Centric LearningImageText
🎯 What it does: This paper studies a new pseudo-label weighting method called SoftMatch, which aims to simultaneously improve both the quantity and quality of pseudo-labels in semi-supervised learning.
SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments
Tsun-Hsuan Wang (Massachusetts Institute of Technology), Chuang Gan (Massachusetts Institute of Technology)
OptimizationRobotic IntelligenceConvolutional Neural NetworkReinforcement LearningMeshBenchmark
🎯 What it does: A SoftZoo soft robot collaborative design platform is proposed, supporting various natural materials, six types of terrain, three categories of motion tasks, and providing a unified interface for robot geometry, elasticity, and muscle.
Solving Constrained Variational Inequalities via a First-order Interior Point-based Method
Tong Yang (University of California), Tatjana Chavdarova (University of California)
OptimizationGenerative Adversarial NetworkTabular
🎯 What it does: A first-order interior point method ACVI is proposed for solving constrained variational inequality (cVI) problems;
Solving Continuous Control via Q-learning
Tim Seyde (Massachusetts Institute of Technology), Markus Wulfmeier (DeepMind)
Robotic IntelligenceReinforcement LearningImageVideo
🎯 What it does: This paper proposes a Decoupled Q-Network (DecQN) that treats continuous control tasks as a cooperative multi-agent problem. By discretizing the action space into limit binary (bang-bang) and performing value decomposition in a global state, high-dimensional continuous control can be achieved using only Q-learning.
Solving stochastic weak Minty variational inequalities without increasing batch size
Thomas Pethick (École Polytechnique Fédérale de Lausanne), Volkan Cevher (École Polytechnique Fédérale de Lausanne)
OptimizationStochastic Differential Equation
🎯 What it does: This study investigates a stochastic outer gradient algorithm for weak Minty variational inequalities, proposing methods such as BC-SEG+ and BC-PSEG+, achieving convergence without increasing the batch size.
Sound Randomized Smoothing in Floating-Point Arithmetic
Vaclav Voracek, Matthias Hein (Tübingen AI Center University of Tübingen)
Adversarial AttackConvolutional Neural NetworkImage
🎯 What it does: This study investigates the unreliability of random smoothing under floating-point arithmetic and proposes a provably reliable random smoothing method under quantized inputs.
SP2 : A Second Order Stochastic Polyak Method
Shuang Li (University of California), Robert M. Gower (Simons Foundation)
OptimizationTabular
🎯 What it does: The SP2 method is proposed, which improves SP using local second-order approximation and implements incremental second-order stochastic optimization without convexity and positive definite Hessian.
Spacetime Representation Learning
Marc T. Law (NVIDIA), James Lucas (NVIDIA)
Representation LearningGraph Neural NetworkGraph
🎯 What it does: A general directed graph embedding framework based on spacetime (Lorentz spacetime) is proposed, utilizing causal structures to realize temporal relationships between nodes.
Sparse Distributed Memory is a Continual Learner
Trenton Bricken (Harvard University), Gabriel Kreiman (Harvard Medical School)
ClassificationRepresentation LearningConvolutional Neural NetworkImage
🎯 What it does: This study designs an improved multi-layer perceptron (SDMLP) based on sparse distributed memory (SDM), achieving 'organic' learning capabilities in continuous learning scenarios through neurobiologically inspired mechanisms such as Top-K activation, no bias, L2 normalization, and positive weight constraints.
Sparse Mixture-of-Experts are Domain Generalizable Learners
Bo Li (Nanyang Technological University), Ziwei Liu (Nanyang Technological University)
ClassificationDomain AdaptationTransformerMixture of ExpertsImage
🎯 What it does: A visual Transformer based on Sparse Mixture of Experts (GMoE) has been constructed, aimed at enhancing domain generalization performance.
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Tianlong Chen (University of Texas at Austin), Zhangyang Wang (University of Texas at Austin)
TransformerMixture of ExpertsText
🎯 What it does: A SMoE-Dropout training framework is proposed, which trains the Transformer in a sparse Mixture-of-Experts manner using a fixed random router and a gradually increasing number of active experts, avoiding parameter redundancy while allowing adaptive scaling based on resources during inference.
Sparse Random Networks for Communication-Efficient Federated Learning
Berivan Isik (Stanford University), Zorzi Michele
Federated LearningImage
🎯 What it does: This study investigates a framework that utilizes sparse random networks in federated learning and achieves communication efficiency through training probability masks.
Sparse Token Transformer with Attention Back Tracking
Heejun Lee (KAIST), Sung Ju Hwang (KAIST)
TransformerImageText
🎯 What it does: Sparse token pruning for Transformer is performed, proposing the importance assessment of attention backtracking (Attention Back-Tracking, ABT) from the final output back to the input, and implementing dynamic token removal by combining a lightweight attention approximation network (ApproxNet) and learnable thresholds (Concrete masking).
Sparse tree-based Initialization for Neural Networks
Patrick Lutz (Boston University), Erwan Scornet (Ecole Polytechnique)
ClassificationOptimizationExplainability and InterpretabilityHyperparameter SearchTabular
🎯 What it does: Convert pre-trained tree-based models (such as random forests, gradient boosting trees, or deep forests) into a sparse three-layer neural network, initializing the first two layers' weights for a multilayer perceptron (MLP), while randomly initializing the remaining layers, and then training using gradient descent.
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Aran Komatsuzaki (Google Research), Neil Houlsby (Google Research)
ClassificationRecognitionTransformerSupervised Fine-TuningMixture of ExpertsImageText
🎯 What it does: Transfer the pre-trained dense Transformer model to a sparse-activated Mixture-of-Experts (MoE) model, utilizing existing training costs and parameters by copying dense layer weights and adding expert layers and routers, achieving an increase in model capacity while maintaining relatively low additional computational overhead.
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!
Shiwei Liu (University of Texas at Austin), Zhangyang Wang (University of Texas at Austin)
OptimizationProtein Structure PredictionTransformerSupervised Fine-TuningTextBenchmark
🎯 What it does: This paper presents SMC-Bench, a benchmark that includes four categories of high-difficulty tasks (common sense reasoning, arithmetic reasoning, protein thermal stability prediction, and multilingual translation) to systematically evaluate the performance of sparse neural networks (SNN) on these tasks, revealing that existing sparse algorithms often fail in real-world scenarios;
Sparsity-Constrained Optimal Transport
Tianlin Liu (University of Basel), Mathieu Blondel (Google Research)
OptimizationMixture of ExpertsImage
🎯 What it does: This paper studies a sparse optimal transport method with explicit cardinality constraints for sparse expert model routing.
Spatial Attention Kinetic Networks with E(n)-Equivariance
Yuanqing Wang (Memorial Sloan Kettering Cancer Center), John Chodera
OptimizationComputational EfficiencyRepresentation LearningDrug DiscoveryGraph Neural NetworkReinforcement LearningGraphBenchmarkPhysics Related
🎯 What it does: A spatial attention function utilizing learnable linear combinations of edge vectors is proposed and embedded into a dynamic graph network (SAKE) with E(n) equivariance for energy prediction and dynamics simulation of physical systems.
Spatio-temporal point processes with deep non-stationary kernels
Zheng Dong (Georgia Institute of Technology), Yao Xie (Georgia Institute of Technology)
Time SeriesSequentialFinance Related
🎯 What it does: A low-rank deep non-stationary kernel model is proposed for efficient modeling of spatiotemporal point processes.
Specformer: Spectral Graph Neural Networks Meet Transformers
Deyu Bo (Beijing University of Posts and Telecommunications), Renjie Liao (University of British Columbia)
ClassificationGraph Neural NetworkTransformerGraph
🎯 What it does: This paper proposes Specformer, a spectral graph neural network based on Transformer, which can perform adaptive set-to-set filtering on the spectrum of the graph Laplacian operator.
Spectral Augmentation for Self-Supervised Learning on Graphs
Lu Lin (Pennsylvania State University), Hongning Wang (University of Virginia)
ClassificationRepresentation LearningAdversarial AttackGraph Neural NetworkContrastive LearningGraph
🎯 What it does: The paper proposes a topology enhancement method based on spectral transformation called SPAN (Spectral Augmentation for Graphs). It achieves more robust representations in graph contrastive learning by pre-computing the edge replacement probabilities that maximize changes in the spectrum (Laplace eigenvalues) of the graph.
Spectral Decomposition Representation for Reinforcement Learning
Tongzheng Ren (UT Austin), Bo Dai (Google Research)
Representation LearningReinforcement LearningSequential
🎯 What it does: This paper proposes a new spectral decomposition representation learning method called SPEDER, which can learn policy-independent spectral features from state-action transition dynamics and achieve online and offline reinforcement learning and imitation learning by combining optimistic/pessimistic exploration mechanisms.
SpeedyZero: Mastering Atari with Limited Data and Time
Yixuan Mei (Shanghai Qi Zhi Institute), Yi Wu (Shanghai Qi Zhi Institute)
Computational EfficiencyReinforcement LearningVideoBenchmark
🎯 What it does: On the extremely inefficient Atari 100k benchmark, SpeedyZero was built as a distributed model-based RL system that balances speed and sample efficiency using only 300k environment steps and 35 minutes of training time.
Spherical Sliced-Wasserstein
Clément Bonet (University of Bretagne Sud), Minh Tan Pham (IRISA)
GenerationData SynthesisComputational EfficiencyAuto EncoderImage
🎯 What it does: A new Spherical Sliced-Wasserstein (SSW) divergence is proposed on the sphere, and it is applied to distribution fitting, density estimation, and generative models for spherical data.
Spikformer: When Spiking Neural Network Meets Transformer
Zhaokun Zhou (Peking University), Li Yuan (Peking University)
ClassificationRecognitionComputational EfficiencySpiking Neural NetworkTransformerImage
🎯 What it does: This paper proposes a Spiking Transformer model called Spikformer, which combines the self-attention mechanism with spiking neural networks, providing an efficient inference framework based on Spike Self Attention (SSA) in spiking form.
Spiking Convolutional Neural Networks for Text Classification
Changze Lv (Fudan University), Xiaoqing Zheng (Fudan University)
ClassificationAdversarial AttackConvolutional Neural NetworkSpiking Neural NetworkText
🎯 What it does: A two-step 'conversion + fine-tuning' method is proposed, which generates Poisson spike trains using pre-trained word vectors to train spiking neural networks (SNN) suitable for text classification and conducts experimental validation.
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Gang Li (Google Research), Yang Li (Google Research)
TransformerVision Language ModelImageText
🎯 What it does: This paper proposes a completely vision-based mobile UI understanding framework called Spotlight, which utilizes screenshots and target areas (focus) for multi-task learning and few-shot inference.
SQA3D: Situated Question Answering in 3D Scenes
Xiaojian Ma (University of California Los Angeles), Siyuan Huang (Beijing Institute for General Artificial Intelligence)
Robotic IntelligenceTransformerLarge Language ModelVision Language ModelMultimodalityPoint Cloud
🎯 What it does: This paper proposes SQA3D, a context-based question answering task in 3D scenes, which requires the robot to first locate its position and orientation in the 3D scene through textual descriptions, and then perform contextual reasoning to answer questions.
Squeeze Training for Adversarial Robustness
Qizhang Li (Harbin Institute of Technology), Hao Chen (UC Davis)
OptimizationAdversarial AttackConvolutional Neural NetworkImage
🎯 What it does: This paper proposes a Squeeze Training (ST) method that uses collaborative samples (neighborhood samples with lower loss) together with adversarial samples for training, in order to regularize the loss landscape of the network and enhance robustness.
Stable Target Field for Reduced Variance Score Estimation in Diffusion Models
Yilun Xu (Massachusetts Institute of Technology), Tommi S. Jaakkola
GenerationData SynthesisComputational EfficiencyDiffusion modelScore-based ModelImageStochastic Differential EquationOrdinary Differential Equation
🎯 What it does: Proposes the Stable Target Field (STF) objective function, which uses self-normalized importance sampling with a large reference batch to reduce the variance of the diffusion model training objective and accelerate training.
StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not at Random
Haoxuan Li (Peking University), Peng Wu (Beijing Technology and Business University)
Recommendation SystemTabular
🎯 What it does: This paper proposes a stable double robust learning (StableDR) framework for unbiased prediction in recommendation systems with missing not at random (MNAR) data.
STaSy: Score-based Tabular data Synthesis
Jayoung Kim (Yonsei University), Noseong Park (Yonsei University)
GenerationData SynthesisScore-based ModelMultimodalityTabularStochastic Differential EquationOrdinary Differential Equation
🎯 What it does: A table data synthesis method STaSy based on score models is proposed, combined with self-paced learning and log probability fine-tuning;
Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
Dianbo Liu (Mila), Yoshua Bengio (Mila)
Reinforcement Learning
🎯 What it does: A cooperative multi-agent reinforcement learning environment called HECOGrid is proposed, which features adjustable coordination and environmental heterogeneity, and introduces the State Active Facilitator (SAF) method based on shared knowledge sources and a multi-strategy pool.
Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions
David Bieber (Google Research), Daniel Tarlow (Google Research)
Graph Neural NetworkText
🎯 What it does: This study proposes the task of predicting Python runtime errors in static scenarios where programs cannot be executed, and infers through training models on code and external resource descriptions;
Statistical Efficiency of Score Matching: The View from Isoperimetry
Frederic Koehler (Stanford University), Andrej Risteski (Carnegie Mellon University)
Score-based Model
🎯 What it does: This study investigates the performance of energy-based models (score matching) in terms of statistical efficiency and compares it with the efficiency of maximum likelihood estimation (MLE).
Statistical Guarantees for Consensus Clustering
Zhixin Zhou (City University of Hong Kong), Arash A Amini
🎯 What it does: A consensus clustering framework based on average association matrix and spectral clustering is proposed, and a local refinement step is designed to address the label permutation problem between different clusters.
Statistical Inference for Fisher Market Equilibrium
Luofeng Liao (Columbia University), Christian Kroer (Columbia University)
OptimizationTabularFinance Related
🎯 What it does: This paper constructs a statistical inference framework based on infinite-dimensional Fisher markets, using the observed equilibrium solutions of finite commodity markets to estimate long-term market indicators such as utility, speed multipliers, and social welfare.
Statistical Theory of Differentially Private Marginal-based Data Synthesis Algorithms
Ximing Li (Tsinghua University), Guang Cheng (University of California)
Data SynthesisSafty and PrivacyTabular
🎯 What it does: For high-dimensional data, a theoretically analyzed algorithm for differentially private synthetic data generation based on Bayesian networks (PrivBayes) is proposed, providing upper bounds for total variation distance, L2 distance, and TSTR error in downstream machine learning tasks, along with corresponding lower bounds.
Stay Moral and Explore: Learn to Behave Morally in Text-based Games
Zijing Shi (University of Technology Sydney), Yali Du (King's College London)
TransformerLarge Language ModelReinforcement LearningTextBenchmark
🎯 What it does: An algorithm named MorAL is proposed, allowing text game agents to learn and adhere to moral guidelines while performing tasks.
Stochastic Differentially Private and Fair Learning
Andrew Lowy (University of Southern California), Meisam Razaviyayn (University of Southern California)
OptimizationSafty and PrivacyImageTabularStochastic Differential Equation
🎯 What it does: A differential privacy fair learning algorithm DP-FERMI is proposed, which can converge under small batches.
Stochastic Multi-Person 3D Motion Forecasting
Sirui Xu (University of Illinois), Liangyan Gui
GenerationPose EstimationRecurrent Neural NetworkTransformerDiffusion modelGenerative Adversarial NetworkVideoTime SeriesBenchmark
🎯 What it does: A dual-layer generative framework (DuMMF) is proposed for the stochastic prediction of 3D motion of multiple humans, which can simultaneously consider the authenticity of individual movements, social interactions among multiple people, and the overall multimodality of motion.
Stochastic No-regret Learning for General Games with Variance Reduction
Yichi Zhou (Microsoft Research), Shuai Li (Shanghai Jiao Tong University)
OptimizationComputational EfficiencyReinforcement Learning
🎯 What it does: A randomized Optimistic Mirror Descent (OMD) algorithm is proposed, which can converge quickly in general games, with individual regret disappearing at a rate of O(1/T^{3/4}) and the total regret of all players disappearing at a rate of O(1/T).
Strategic Classification with Graph Neural Networks
Itay Eilat (Technion Israel Institute of Technology), Nir Rosenfeld (Technion Israel Institute of Technology)
ClassificationGraph Neural NetworkGraph
🎯 What it does: Research on strategic classification in the context of graph neural networks, considering users' adaptive modifications of features through social relationships.
STREET: A MULTI-TASK STRUCTURED REASONING AND EXPLANATION BENCHMARK
Danilo Neves Ribeiro, Dan Roth (Amazon Web Services)
TransformerLarge Language ModelSupervised Fine-TuningTextGraphBenchmark
🎯 What it does: Proposes the STREET benchmark, which requires models to provide multi-step structured reasoning graphs while answering questions, covering multiple tasks and domains.
Strong inductive biases provably prevent harmless interpolation
Michael Aerni (ETH Zurich), Fanny Yang (ETH Zurich)
Convolutional Neural NetworkImage
🎯 What it does: This paper studies the phenomenon that whether interpolation is harmful in over-parameterized models depends on the strength of inductive bias, proposing to adjust the strength of inductive bias through the size of convolutional kernel filters and proving a phase transition;
StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training
Yuechen Yu (Baidu Inc), Jingdong Wang (Baidu Inc)
ClassificationRecognitionConvolutional Neural NetworkTransformerVision Language ModelImageMultimodality
🎯 What it does: A non-OCR pre-training framework based on images, StrucTexTv2, is designed to perform image reconstruction and language prediction simultaneously through text region-level masking.
Structure by Architecture: Structured Representations without Regularization
Felix Leeb (Max Planck Institute for Intelligent Systems), Bernhard Schölkopf (Max Planck Institute for Intelligent Systems)
GenerationRepresentation LearningConvolutional Neural NetworkAuto EncoderImage
🎯 What it does: A new unsupervised Structural Autoencoder (SAE) was designed and evaluated, utilizing hierarchical linear transformations (Str-Tfm) in the decoder to achieve hierarchical encoding of latent variables, and a mixed sampling method based on variable independence was proposed.
STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables
Jaehyun Nam (Korea Advanced Institute of Science and Technology), Jinwoo Shin (Korea Advanced Institute of Science and Technology)
ClassificationMeta LearningTabularBenchmark
🎯 What it does: This paper proposes an unsupervised meta-learning framework called STUNT for a small amount of labeled tabular data. It utilizes unlabeled tables to randomly select columns and generates pseudo-labels through k-means clustering, forming diverse few-shot tasks, and employs a prototypical network for meta-training, ultimately fine-tuning on a small number of labeled samples.
StyleMorph: Disentangled 3D-Aware Image Synthesis with a 3D Morphable StyleGAN
Eric-Tuan Le (University College London), Iasonas Kokkinos (University College London)
GenerationData SynthesisGenerative Adversarial NetworkImage
🎯 What it does: A novel unsupervised generative model called StyleMorph is constructed based on a 3D deformable template for high-quality, separable control of image synthesis.
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks
Noam Wies (Hebrew University of Jerusalem), Amnon Shashua (Hebrew University of Jerusalem)
Recurrent Neural NetworkTransformerSupervised Fine-TuningSequential
🎯 What it does: The paper proves that introducing subtask decomposition and intermediate supervision in sequence-to-sequence models can make inherently unlearnable composite tasks learnable, and presents theoretical and experimental results on the parity of position subsets and arbitrary P-class functions.
Subquadratic Algorithms for Kernel Matrices via Kernel Density Estimation
Ainesh Bakshi (Massachusetts Institute of Technology), Samson Zhou (University of California Berkeley and Rice University)
Tabular
🎯 What it does: A sub-quadratic time algorithm for kernel matrices is achieved through kernel density estimation (KDE), addressing several fundamental problems such as low-rank approximation, spectral sparsification, and eigenvalue estimation.
Subsampling in Large Graphs Using Ricci Curvature
Shushan Wu (University of Georgia), Wenxuan Zhong (University of Georgia)
Graph Neural NetworkGraph
🎯 What it does: A graph sub-sampling algorithm based on the Ollivier-Ricci curvature gradient (ORG-sub) is proposed, which gradually constructs a subgraph by maximizing the curvature difference of adjacent edges, aiming to preserve the community structure of the original graph and accurately estimate the number of communities.
Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees
Swarnadeep Saha (University of North Carolina), Mohit Bansal (University of North Carolina)
GenerationExplainability and InterpretabilityTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: An interpretable abstract summarization framework called Summarization Program (SP) is proposed, which gradually generates summary sentences by constructing a binary tree composed of neural modules (compression, rewriting, fusion).
Supervision Complexity and its Role in Knowledge Distillation
Hrayr Harutyunyan (USC Information Sciences Institute), Sanjiv Kumar (Google Research)
Knowledge DistillationConvolutional Neural NetworkImage
🎯 What it does: This paper introduces the concept of 'supervision complexity' to construct a theoretical framework that explains why student models can achieve better generalization performance in knowledge distillation (KD). Based on this, an online distillation method is proposed, which gradually uses predictions from teachers at different stages as supervision signals during the training of the student, thereby achieving a better balance between target complexity and student capacity.
Suppressing the Heterogeneity: A Strong Feature Extractor for Few-shot Segmentation
Zhengdong Hu (University of Technology Sydney), Yi Yang (Zhejiang University)
SegmentationTransformerImage
🎯 What it does: This paper proposes a feature extractor for few-shot semantic segmentation called MuHS, which utilizes a Transformer backbone and combines three mechanisms: cross-sample attention, cross-region interaction, and mask image segmentation, effectively suppressing sample, region, and patch-level heterogeneity, and enhancing the intra-class compactness of features.
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
Yoonho Lee (Stanford University), Chelsea Finn (Stanford University)
Domain AdaptationConvolutional Neural NetworkTransformerSupervised Fine-TuningImage
🎯 What it does: The study focuses on transfer learning under distribution shift, proposing that fine-tuning only a small segment of continuous layers of a pre-trained model (i.e., surgical fine-tuning) can enhance performance in the target domain, and that different types of shifts correspond to different optimal layer intervals.
SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication
Marco Bornstein (University of Maryland), Furong Huang (University of Maryland)
Federated LearningImage
🎯 What it does: SWIFT is proposed, a wait-free decentralized federated learning algorithm that allows each client to perform local training and model communication at its own pace, avoiding synchronous waiting.
Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields
Zhenxing MI, Dan Xu (Hong Kong University of Science and Technology)
GenerationData SynthesisMixture of ExpertsNeural Radiance FieldPoint Cloud
🎯 What it does: This paper proposes Switch-NeRF, an end-to-end learnable sparse NeRF architecture for the decomposition and reconstruction of large-scale scenes.
Symbolic Physics Learner: Discovering governing equations via Monte Carlo tree search
Fangzheng Sun (Northeastern University), Hao Sun (Renmin University of China)
Reinforcement LearningTabularTime SeriesPhysics Related
🎯 What it does: This paper proposes a Symbolic Physics Learner (SPL) based on Monte Carlo Tree Search (MCTS) for mining analytical equations of nonlinear dynamics from limited noisy data.
Symmetric Pruning in Quantum Neural Networks
Xinbiao Wang (Wuhan University), Dacheng Tao (Wuhan University)
OptimizationGraphPhysics Related
🎯 What it does: This paper proposes an effective tool for the training dynamics of quantum neural networks—Effective Quantum Neural Tangent Kernel (EQNTK), and based on this theory, develops a Symmetric Pruning (SP) method to automatically construct symmetric ansatz to enhance the trainability and convergence speed of quantum neural networks in ground state preparation (GSP) tasks.
Symmetries, Flat Minima, and the Conserved Quantities of Gradient Flow
Bo Zhao (University of California, San Diego), Nima Dehmamy (IBM Research)
OptimizationAdversarial AttackConvolutional Neural NetworkImageOrdinary Differential Equation
🎯 What it does: This study proposes a general framework based on the equivariance of activation functions to find continuous symmetry transformations in the parameter space of neural networks, revealing many low-loss flat directions and introducing data-dependent nonlinear symmetry transformations. Furthermore, it derives conservation quantities of the gradient flow using these symmetries, which can parameterize flat extrema and demonstrate that the conservation quantities are related to convergence speed, sharpness of optimal points, and generalization ability. Finally, it shows that the model ensemble constructed through symmetry transformations enhances robustness against FGSM attacks without retraining.
SYNC: SAFETY-AWARE NEURAL CONTROL FOR STABILIZING STOCHASTIC DELAY-DIFFERENTIAL EQUATIONS
Jingdong Zhang (Fudan University), Wei Lin (Fudan University)
OptimizationSafty and PrivacyTime SeriesStochastic Differential Equation
🎯 What it does: A safety-aware control (SYNC) framework based on neural networks is designed to achieve stabilization and ensure safety constraints in stochastic delay differential equations (SDDE).
Synthetic Data Generation of Many-to-Many Datasets via Random Graph Generation
Kai Xu (Hazy), Luke Robinson (Hazy)
Data SynthesisGraph Neural NetworkGenerative Adversarial NetworkTabular
🎯 What it does: A differential privacy-preserving synthetic data generation framework for many-to-many relational databases is proposed, utilizing random graph generation and node and set embedding to achieve scalable model training and sampling.
Systematic Rectification of Language Models via Dead-end Analysis
Meng Cao (Mila - Quebec AI Institute), Samira Shabanian (Microsoft Research)
GenerationTransformerLarge Language ModelReinforcement LearningText
🎯 What it does: This paper proposes a method that utilizes a small auxiliary RL model to dynamically adjust the token selection probabilities during the generation process of language models, in order to reduce the toxicity risk of the final text.
TabCaps: A Capsule Neural Network for Tabular Data Classification with BoW Routing
Jintai Chen (Zhejiang University), Jian Wu (Zhejiang University)
ClassificationAuto EncoderTabular
🎯 What it does: A TABCAPS model is proposed for processing tabular data using capsule neural networks.
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
Noah Hollmann (Charité University Medicine Berlin), Frank Hutter (University of Freiburg)
ClassificationTransformerTabular
🎯 What it does: A pre-trained Transformer named TabPFN is proposed, capable of making predictions for small-scale tabular classification tasks (up to 1,000 training samples, 100 numerical features, 10 classes) in less than one second, without the need for hyperparameter tuning.
Tailoring Language Generation Models under Total Variation Distance
Haozhe Ji (Tsinghua University), Minlie Huang (NetEase Inc.)
GenerationData SynthesisTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Proposes a language generation training objective TaiLr based on Total Variation Distance (TVD) to address the issues of overfitting and text degeneration in MLE.
Taking a Step Back with KCal: Multi-Class Kernel-Based Calibration for Deep Neural Networks
Zhen Lin (University of Illinois), Jimeng Sun (University of Illinois)
ClassificationRecognitionTransformerImageBiomedical Data
🎯 What it does: This paper proposes a multi-class complete calibration method KCal based on kernel density estimation. It utilizes the intermediate embeddings of deep networks to learn a low-dimensional metric space, and then performs KDE inference on the calibration set to directly output predicted probabilities that satisfy probability distribution constraints.
TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization
Alan Jeffares (University of Cambridge), Mihaela van der Schaar (University of Cambridge)
ClassificationOptimizationTabular
🎯 What it does: A regularization method based on neural network gradient orthogonalization and specialization (TANGOS) is proposed to enhance the generalization performance of models for tabular data.
Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives
Shaokun Zhang (Pennsylvania State University), Qingyun Wu (Pennsylvania State University)
OptimizationHyperparameter SearchTabular
🎯 What it does: This paper proposes a targeted method for multi-objective hyperparameter optimization called LexiFlow, which can automatically find the optimal configuration across multiple objectives based on user-specified lexicographic priorities.
Task Ambiguity in Humans and Language Models
Alex Tamkin (Stanford University), Noah Goodman (Stanford University)
ClassificationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
🎯 What it does: A controllable ambiguity sentence classification benchmark, AmbiBench, is proposed to evaluate the performance of humans and various language models (including HFD and non-HFD) under different levels of task ambiguity. The study investigates how models resolve ambiguity through informative/non-informative instructions, few-shot examples, fine-tuning, and other methods.
Task-Aware Information Routing from Common Representation Space in Lifelong Learning
Prashant Shivaram Bhat (NavInfo Europe), Elahe Arani (Eindhoven University of Technology)
Auto EncoderSequential
🎯 What it does: A TAMiL method based on the global workspace theory is proposed, combining experience replay and task-specific attention modules to achieve lifelong learning.
Task-customized Masked Autoencoder via Mixture of Cluster-conditional Experts
Zhili LIU, James Kwok
ClassificationObject DetectionSegmentationTransformerMixture of ExpertsAuto EncoderImage
🎯 What it does: A self-supervised pre-training framework based on Mixture of Cluster-conditional Experts (MoCE) is designed, utilizing clustering groups and a gating network to allow each expert to learn only semantically similar data, thereby achieving a task-customized Masked Autoencoder.
TaskPrompter: Spatial-Channel Multi-Task Prompting for Dense Scene Understanding
Hanrong Ye (Hong Kong University of Science and Technology), Dan Xu (Hong Kong University of Science and Technology)
SegmentationDepth EstimationTransformerPrompt EngineeringImage
🎯 What it does: This paper proposes a Spatial-Channel Multi-Task Prompting (TaskPrompter) framework that utilizes spatial and channel task prompts in Transformers to learn task-general, task-specific, and cross-task interaction representations uniformly at each layer, thereby achieving multi-task dense scene understanding.
TDR-CL: Targeted Doubly Robust Collaborative Learning for Debiased Recommendations
Haoxuan Li (Peking University), Peng Wu (Beijing Technology and Business University)
Recommendation SystemTabular
🎯 What it does: A dual robust (TDR) estimator and collaborative learning (TDR-CL) framework is proposed to simultaneously reduce bias and variance in recommendation systems, achieving unbiased, low-variance debiased predictions.
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Manzil Zaheer (Google Research), Sanjiv Kumar (Google Research)
RetrievalKnowledge DistillationAuto EncoderGenerative Adversarial NetworkImageText
🎯 What it does: This paper proposes a Teacher-Guided Training (TGT) framework, which dynamically generates samples that conflict with the teacher's decision boundary in the latent space by utilizing the generator and classifier of a large model, thereby accelerating and enhancing the distillation effect of a small model.
TempCLR: Temporal Alignment Representation with Contrastive Learning
Yuncong Yang (Columbia University), Shih-Fu Chang (Columbia University)
RecognitionRetrievalRepresentation LearningTransformerContrastive LearningVideoText
🎯 What it does: This paper proposes TempCLR, a temporal alignment representation framework based on contrastive learning, designed to explicitly align long videos with their corresponding multi-sentence descriptions.
TEMPERA: Test-Time Prompt Editing via Reinforcement Learning
Tianjun Zhang (University of California Berkeley), Joseph E. Gonzalez (University of California Berkeley)
TransformerReinforcement LearningPrompt EngineeringText
🎯 What it does: This paper proposes a reinforcement learning-based test-time prompt editing method called TEMPERA, which automatically generates query-dependent discrete prompts for zero/few-shot text classification tasks using pre-trained language models.
Temperature Schedules for self-supervised contrastive methods on long-tail data
Anna Kukleva (MPI for Informatics), Christian Rupprecht (University of Oxford)
ClassificationRepresentation LearningContrastive LearningImage
🎯 What it does: The study investigates the performance of self-supervised contrastive learning on long-tailed distribution data and proposes a dynamic temperature (τ) scheduling method that alternates between emphasizing instance discrimination and group discrimination using cosine scheduling, thereby improving representation quality.
Temporal Coherent Test Time Optimization for Robust Video Classification
Chenyu Yi (Nanyang Technological University), Alex Kot
ClassificationOptimizationVideo
🎯 What it does: A temporal consistency testing optimization framework for video classification, TeCo, is proposed, which utilizes global information for entropy minimization and local information for attention-weighted temporal consistency regularization to enhance the model's robustness on corrupted test data.
Temporal Dependencies in Feature Importance for Time Series Prediction
Kin Kwan Leung (Layer 6 AI), Maksims Volkovs (Layer 6 AI)
Explainability and InterpretabilityTabularTime SeriesBiomedical DataElectronic Health Records
🎯 What it does: A time-window-based feature removal explanation method called WinIT is proposed, which can capture the importance of temporal features in time series forecasting.
Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning
Mhairi Dunion (University of Edinburgh), Stefano V Albrecht
Representation LearningReinforcement LearningImageTime Series
🎯 What it does: This paper studies robust representation learning in reinforcement learning and proposes a self-supervised auxiliary task named TED, which utilizes the temporal structure of continuous observations to learn decoupled image representations, thereby enhancing the generalization ability of RL agents when environmental variables change.
Temporal Domain Generalization with Drift-Aware Dynamic Neural Networks
Guangji Bai (Emory University), Liang Zhao (Emory University)
Domain AdaptationRecurrent Neural NetworkGraph Neural NetworkTime SeriesSequential
🎯 What it does: The DRAIN framework is proposed for domain generalization in environments with temporal domain drift, capable of predicting future model parameters and maintaining high performance without future data.
Tensor-Based Sketching Method for the Low-Rank Approximation of Data Streams.
Cuiyu Liu (Peking University), Chao Yang (Peking University)
CompressionComputational EfficiencyImageMagnetic Resonance Imaging
🎯 What it does: A sparse projection method based on tensor decomposition is proposed, which quickly generates low-rank approximation data flow projection matrices from the training set;
Test-Time Adaptation via Self-Training with Nearest Neighbor Information
Minguk Jang (Korea Advanced Institute of Science and Technology), Hye Won Chung (Korea Advanced Institute of Science and Technology)
Domain AdaptationImage
🎯 What it does: A self-training framework TAST based on nearest neighbor pseudo-labels is proposed, which can adapt to domain shift during testing;
Test-Time Robust Personalization for Federated Learning
Liangze Jiang (École Polytechnique Fédérale de Lausanne), Tao Lin (Westlake University)
Federated LearningConvolutional Neural NetworkTransformerSupervised Fine-TuningImageBenchmark
🎯 What it does: This paper addresses the issue of distribution shift during the testing phase in federated learning deployment and proposes the FedTHE+ method, which achieves online robust personalization without changing the training process.
Text Summarization with Oracle Expectation
Yumo Xu (University of Edinburgh), Mirella Lapata (University of Edinburgh)
GenerationTransformerSupervised Fine-TuningText
🎯 What it does: This paper proposes a soft label method based on oracle expectations, called OREO, to improve the generation of sentence labels in extractive summarization;
TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization
Bairu Hou (University of California Santa Barbara), Shiyu Chang (Massachusetts Institute of Technology IBM Watson Artificial Intelligence Lab)
OptimizationAdversarial AttackTransformerLarge Language ModelText
🎯 What it does: This paper proposes TEXTGRAD, a first-order optimization attack framework based on gradient-driven methods, addressing the constraints of discreteness and fluency in text attacks.
TextShield: Beyond Successfully Detecting Adversarial Sentences in text classification
Lingfeng Shen (Johns Hopkins University), Ying Chen (China Agricultural University)
ClassificationAdversarial AttackConvolutional Neural NetworkRecurrent Neural NetworkTransformerText
🎯 What it does: A detection-correction combined defense framework called TextShield is proposed, which uses Adaptive Word Importance (AWI) calculated from saliency to detect and correct word-level adversarial attacks.
Thalamus: a brain-inspired algorithm for biologically-plausible continual learning and disentangled representations
Ali Hummos (Massachusetts Institute of Technology)
Representation LearningRecurrent Neural NetworkTabular
🎯 What it does: A continuous learning algorithm inspired by the thalamocortical circuits of the brain, called Thalamus, is proposed, which dynamically generates contextual signals in the latent space using gradient optimization during inference.
That Label's got Style: Handling Label Style Bias for Uncertain Image Segmentation
Kilian Zepf (Technical University of Denmark), Aasa Feragen (Technical University of Denmark)
SegmentationAuto EncoderImage
🎯 What it does: In the uncertainty model for image segmentation, researchers found that different labeling tools or different experimenters produce 'label styles' that can lead to model bias; to address this, a model that conditions on label styles during training was proposed.
The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks
Daniel Kunin (Stanford University), Surya Ganguli (Stanford University)
OptimizationImage
🎯 What it does: This paper studies the implicit maximum margin bias produced by gradient flow under the exponential loss function for almost all modern feedforward neural networks (including residual, bias, normalization layers, etc.), proposes the definition of 'quasi-homogeneous' networks, and provides corresponding gradient dynamics analysis.
The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image
Yuki M Asano, Aaqib Saeed (Eindhoven University of Technology)
ClassificationKnowledge DistillationConvolutional Neural NetworkContrastive LearningImageMultimodality
🎯 What it does: This paper proposes a method that uses only a single image and its strong augmented data to train a student network through knowledge distillation, achieving high accuracy in multi-class classification tasks.
The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
Huancheng Chen (University of Texas), Haris Vikalo (University of Texas)
Federated LearningKnowledge DistillationImage
🎯 What it does: Proposes FedHKD, which utilizes data-independent super knowledge distillation to simultaneously enhance personalized and global models in federated learning;
The Curious Case of Benign Memorization
Sotiris Anagnostidis (ETH Zurich), Thomas Hofmann (ETH Zurich)
ClassificationConvolutional Neural NetworkImage
🎯 What it does: Research on deep networks trained with completely random labels found that adding data augmentation allows the network to simultaneously achieve random label memorization and learn useful features (referred to as 'benign memorization').
The Dark Side of AutoML: Towards Architectural Backdoor Search
Ren Pang (Pennsylvania State University), Ting Wang (Pennsylvania State University)
Adversarial AttackNeural Architecture SearchImage
🎯 What it does: This paper proposes to directly search for network architectures that inherently contain backdoors using Neural Architecture Search (NAS), thereby achieving backdoor attacks without contaminating training data or model parameters.