π― What it does: This study investigates the unreliability of random smoothing under floating-point arithmetic and proposes a provably reliable random smoothing method under quantized inputs.
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Tianlong Chen (University of Texas at Austin), Zhangyang Wang (University of Texas at Austin)
CodeTransformerMixture of ExpertsText
π― What it does: A SMoE-Dropout training framework is proposed, which trains the Transformer in a sparse Mixture-of-Experts manner using a fixed random router and a gradually increasing number of active experts, avoiding parameter redundancy while allowing adaptive scaling based on resources during inference.
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Aran Komatsuzaki (Google Research), Neil Houlsby (Google Research)
CodeClassificationRecognitionTransformerSupervised Fine-TuningMixture of ExpertsImageText
π― What it does: Transfer the pre-trained dense Transformer model to a sparse-activated Mixture-of-Experts (MoE) model, utilizing existing training costs and parameters by copying dense layer weights and adding expert layers and routers, achieving an increase in model capacity while maintaining relatively low additional computational overhead.
π― What it does: This paper presents SMC-Bench, a benchmark that includes four categories of high-difficulty tasks (common sense reasoning, arithmetic reasoning, protein thermal stability prediction, and multilingual translation) to systematically evaluate the performance of sparse neural networks (SNN) on these tasks, revealing that existing sparse algorithms often fail in real-world scenarios;
Spatial Attention Kinetic Networks with E(n)-Equivariance
Yuanqing Wang (Memorial Sloan Kettering Cancer Center), John Chodera
CodeOptimizationComputational EfficiencyRepresentation LearningDrug DiscoveryGraph Neural NetworkReinforcement LearningGraphBenchmarkPhysics Related
π― What it does: A spatial attention function utilizing learnable linear combinations of edge vectors is proposed and embedded into a dynamic graph network (SAKE) with E(n) equivariance for energy prediction and dynamics simulation of physical systems.
π― What it does: This paper proposes Specformer, a spectral graph neural network based on Transformer, which can perform adaptive set-to-set filtering on the spectrum of the graph Laplacian operator.
π― What it does: A new Spherical Sliced-Wasserstein (SSW) divergence is proposed on the sphere, and it is applied to distribution fitting, density estimation, and generative models for spherical data.
π― What it does: This paper proposes a Spiking Transformer model called Spikformer, which combines the self-attention mechanism with spiking neural networks, providing an efficient inference framework based on Spike Self Attention (SSA) in spiking form.
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Gang Li (Google Research), Yang Li (Google Research)
CodeTransformerVision Language ModelImageText
π― What it does: This paper proposes a completely vision-based mobile UI understanding framework called Spotlight, which utilizes screenshots and target areas (focus) for multi-task learning and few-shot inference.
π― What it does: This paper proposes a Squeeze Training (ST) method that uses collaborative samples (neighborhood samples with lower loss) together with adversarial samples for training, in order to regularize the loss landscape of the network and enhance robustness.
π― What it does: Proposes the Stable Target Field (STF) objective function, which uses self-normalized importance sampling with a large reference batch to reduce the variance of the diffusion model training objective and accelerate training.
Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
Dianbo Liu (Mila), Yoshua Bengio (Mila)
CodeReinforcement Learning
π― What it does: A cooperative multi-agent reinforcement learning environment called HECOGrid is proposed, which features adjustable coordination and environmental heterogeneity, and introduces the State Active Facilitator (SAF) method based on shared knowledge sources and a multi-strategy pool.
Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions
David Bieber (Google Research), Daniel Tarlow (Google Research)
CodeGraph Neural NetworkText
π― What it does: This study proposes the task of predicting Python runtime errors in static scenarios where programs cannot be executed, and infers through training models on code and external resource descriptions;
Strategic Classification with Graph Neural Networks
Itay Eilat (Technion Israel Institute of Technology), Nir Rosenfeld (Technion Israel Institute of Technology)
CodeClassificationGraph Neural NetworkGraph
π― What it does: Research on strategic classification in the context of graph neural networks, considering users' adaptive modifications of features through social relationships.
STREET: A MULTI-TASK STRUCTURED REASONING AND EXPLANATION BENCHMARK
Danilo Neves Ribeiro, Dan Roth (Amazon Web Services)
CodeTransformerLarge Language ModelSupervised Fine-TuningTextGraphBenchmark
π― What it does: Proposes the STREET benchmark, which requires models to provide multi-step structured reasoning graphs while answering questions, covering multiple tasks and domains.
StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training
Yuechen Yu (Baidu Inc), Jingdong Wang (Baidu Inc)
CodeClassificationRecognitionConvolutional Neural NetworkTransformerVision Language ModelImageMultimodality
π― What it does: A non-OCR pre-training framework based on images, StrucTexTv2, is designed to perform image reconstruction and language prediction simultaneously through text region-level masking.
STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables
Jaehyun Nam (Korea Advanced Institute of Science and Technology), Jinwoo Shin (Korea Advanced Institute of Science and Technology)
CodeClassificationMeta LearningTabularBenchmark
π― What it does: This paper proposes an unsupervised meta-learning framework called STUNT for a small amount of labeled tabular data. It utilizes unlabeled tables to randomly select columns and generates pseudo-labels through k-means clustering, forming diverse few-shot tasks, and employs a prototypical network for meta-training, ultimately fine-tuning on a small number of labeled samples.
π― What it does: The paper proves that introducing subtask decomposition and intermediate supervision in sequence-to-sequence models can make inherently unlearnable composite tasks learnable, and presents theoretical and experimental results on the parity of position subsets and arbitrary P-class functions.
Subquadratic Algorithms for Kernel Matrices via Kernel Density Estimation
Ainesh Bakshi (Massachusetts Institute of Technology), Samson Zhou (University of California Berkeley and Rice University)
CodeTabular
π― What it does: A sub-quadratic time algorithm for kernel matrices is achieved through kernel density estimation (KDE), addressing several fundamental problems such as low-rank approximation, spectral sparsification, and eigenvalue estimation.
Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees
Swarnadeep Saha (University of North Carolina), Mohit Bansal (University of North Carolina)
CodeGenerationExplainability and InterpretabilityTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: An interpretable abstract summarization framework called Summarization Program (SP) is proposed, which gradually generates summary sentences by constructing a binary tree composed of neural modules (compression, rewriting, fusion).
SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication
Marco Bornstein (University of Maryland), Furong Huang (University of Maryland)
CodeFederated LearningImage
π― What it does: SWIFT is proposed, a wait-free decentralized federated learning algorithm that allows each client to perform local training and model communication at its own pace, avoiding synchronous waiting.
Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields
Zhenxing MI, Dan Xu (Hong Kong University of Science and Technology)
CodeGenerationData SynthesisMixture of ExpertsNeural Radiance FieldPoint Cloud
π― What it does: This paper proposes Switch-NeRF, an end-to-end learnable sparse NeRF architecture for the decomposition and reconstruction of large-scale scenes.
Symbolic Physics Learner: Discovering governing equations via Monte Carlo tree search
Fangzheng Sun (Northeastern University), Hao Sun (Renmin University of China)
CodeReinforcement LearningTabularTime SeriesPhysics Related
π― What it does: This paper proposes a Symbolic Physics Learner (SPL) based on Monte Carlo Tree Search (MCTS) for mining analytical equations of nonlinear dynamics from limited noisy data.
π― What it does: This study proposes a general framework based on the equivariance of activation functions to find continuous symmetry transformations in the parameter space of neural networks, revealing many low-loss flat directions and introducing data-dependent nonlinear symmetry transformations. Furthermore, it derives conservation quantities of the gradient flow using these symmetries, which can parameterize flat extrema and demonstrate that the conservation quantities are related to convergence speed, sharpness of optimal points, and generalization ability. Finally, it shows that the model ensemble constructed through symmetry transformations enhances robustness against FGSM attacks without retraining.
SYNC: SAFETY-AWARE NEURAL CONTROL FOR STABILIZING STOCHASTIC DELAY-DIFFERENTIAL EQUATIONS
Jingdong Zhang (Fudan University), Wei Lin (Fudan University)
CodeOptimizationSafty and PrivacyTime SeriesStochastic Differential Equation
π― What it does: A safety-aware control (SYNC) framework based on neural networks is designed to achieve stabilization and ensure safety constraints in stochastic delay differential equations (SDDE).
Systematic Rectification of Language Models via Dead-end Analysis
Meng Cao (Mila - Quebec AI Institute), Samira Shabanian (Microsoft Research)
CodeGenerationTransformerLarge Language ModelReinforcement LearningText
π― What it does: This paper proposes a method that utilizes a small auxiliary RL model to dynamically adjust the token selection probabilities during the generation process of language models, in order to reduce the toxicity risk of the final text.
π― What it does: A pre-trained Transformer named TabPFN is proposed, capable of making predictions for small-scale tabular classification tasks (up to 1,000 training samples, 100 numerical features, 10 classes) in less than one second, without the need for hyperparameter tuning.
Tailoring Language Generation Models under Total Variation Distance
Haozhe Ji (Tsinghua University), Minlie Huang (NetEase Inc.)
CodeGenerationData SynthesisTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: Proposes a language generation training objective TaiLr based on Total Variation Distance (TVD) to address the issues of overfitting and text degeneration in MLE.
Taking a Step Back with KCal: Multi-Class Kernel-Based Calibration for Deep Neural Networks
Zhen Lin (University of Illinois), Jimeng Sun (University of Illinois)
CodeClassificationRecognitionTransformerImageBiomedical Data
π― What it does: This paper proposes a multi-class complete calibration method KCal based on kernel density estimation. It utilizes the intermediate embeddings of deep networks to learn a low-dimensional metric space, and then performs KDE inference on the calibration set to directly output predicted probabilities that satisfy probability distribution constraints.
TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization
Alan Jeffares (University of Cambridge), Mihaela van der Schaar (University of Cambridge)
CodeClassificationOptimizationTabular
π― What it does: A regularization method based on neural network gradient orthogonalization and specialization (TANGOS) is proposed to enhance the generalization performance of models for tabular data.
Alex Tamkin (Stanford University), Noah Goodman (Stanford University)
CodeClassificationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringTextBenchmark
π― What it does: A controllable ambiguity sentence classification benchmark, AmbiBench, is proposed to evaluate the performance of humans and various language models (including HFD and non-HFD) under different levels of task ambiguity. The study investigates how models resolve ambiguity through informative/non-informative instructions, few-shot examples, fine-tuning, and other methods.
Task-Aware Information Routing from Common Representation Space in Lifelong Learning
Prashant Shivaram Bhat (NavInfo Europe), Elahe Arani (Eindhoven University of Technology)
CodeAuto EncoderSequential
π― What it does: A TAMiL method based on the global workspace theory is proposed, combining experience replay and task-specific attention modules to achieve lifelong learning.
Task-customized Masked Autoencoder via Mixture of Cluster-conditional Experts
Zhili LIU, James Kwok
CodeClassificationObject DetectionSegmentationTransformerMixture of ExpertsAuto EncoderImage
π― What it does: A self-supervised pre-training framework based on Mixture of Cluster-conditional Experts (MoCE) is designed, utilizing clustering groups and a gating network to allow each expert to learn only semantically similar data, thereby achieving a task-customized Masked Autoencoder.
π― What it does: This paper proposes TempCLR, a temporal alignment representation framework based on contrastive learning, designed to explicitly align long videos with their corresponding multi-sentence descriptions.
π― What it does: This paper proposes a reinforcement learning-based test-time prompt editing method called TEMPERA, which automatically generates query-dependent discrete prompts for zero/few-shot text classification tasks using pre-trained language models.
CodeExplainability and InterpretabilityTabularTime SeriesBiomedical DataElectronic Health Records
π― What it does: A time-window-based feature removal explanation method called WinIT is proposed, which can capture the importance of temporal features in time series forecasting.
π― What it does: The DRAIN framework is proposed for domain generalization in environments with temporal domain drift, capable of predicting future model parameters and maintaining high performance without future data.
π― What it does: A sparse projection method based on tensor decomposition is proposed, which quickly generates low-rank approximation data flow projection matrices from the training set;
π― What it does: This paper addresses the issue of distribution shift during the testing phase in federated learning deployment and proposes the FedTHE+ method, which achieves online robust personalization without changing the training process.
π― What it does: This paper proposes a soft label method based on oracle expectations, called OREO, to improve the generation of sentence labels in extractive summarization;
TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization
Bairu Hou (University of California Santa Barbara), Shiyu Chang (Massachusetts Institute of Technology IBM Watson Artificial Intelligence Lab)
CodeOptimizationAdversarial AttackTransformerLarge Language ModelText
π― What it does: This paper proposes TEXTGRAD, a first-order optimization attack framework based on gradient-driven methods, addressing the constraints of discreteness and fluency in text attacks.
π― What it does: This paper proposes a method that uses only a single image and its strong augmented data to train a student network through knowledge distillation, achieving high accuracy in multi-class classification tasks.
π― What it does: This paper proposes to directly search for network architectures that inherently contain backdoors using Neural Architecture Search (NAS), thereby achieving backdoor attacks without contaminating training data or model parameters.
π― What it does: This paper systematically studies the Unified Open Set Recognition (UOSR) problem, analyzes its uncertainty distribution, and proposes a few-shot UOSR evaluation framework and the FS-KNNS method.
π― What it does: A new rotation target detection loss function KFIoU is proposed to approximate the difficult-to-derive SkewIoU, while maintaining consistency with conventional regression losses, thereby improving detection accuracy.
The Lie Derivative for Measuring Learned Equivariance
Nate Gruver (New York University), Andrew Gordon Wilson (New York University)
CodeConvolutional Neural NetworkTransformerImageBiomedical Data
π― What it does: This paper systematically evaluates the equivariance of hundreds of pre-trained visual models (CNN, ViT, Mixer) using Lie derivatives and hierarchical decomposition.
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
Zihui Xue (University of Texas at Austin), Hang Zhao (Tsinghua University)
CodeKnowledge DistillationMultimodality
π― What it does: This study investigates the cross-modal knowledge distillation mechanism, proposing the Modal Venn Diagram (MVD) and the Modal Focus Hypothesis (MFH) to explain and predict the effects of cross-modal KD.
The Surprising Computational Power of Nondeterministic Stack RNNs
Brian DuSell (University of Notre Dame), David Chiang (University of Notre Dame)
CodeRecognitionRecurrent Neural NetworkText
π― What it does: This paper proposes and experiments with a differentiable non-deterministic stack RNN (RNS-RNN) and its vector stack version (VRNS-RNN), demonstrating its ability to recognize all context-free languages, their intersections, and some non-context-free languages, and conducts natural language modeling experiments on the Penn Treebank.
The Tilted Variational Autoencoder: Improving Out-of-Distribution Detection
Griffin Floto (University of Toronto), Mihai Nica (University of Guelph)
CodeAnomaly DetectionAuto EncoderImage
π― What it does: Proposed and implemented a tilted Gaussian prior as a substitute for the standard Gaussian, used in variational autoencoders to improve the distribution coverage of high-dimensional latent space and enhance anomaly detection performance.
π― What it does: This study investigates the trade-off between the generalizability (applicability to multiple tasks) and label efficiency (rapid convergence with a small amount of labeled data) of contrastive learning pre-trained representations, and proposes a method of fine-tuning through contrastive regularization to alleviate this trade-off.
CodeClassificationExplainability and InterpretabilityImage
π― What it does: This paper proposes ProtoKNN, a model that extends ProtoPNet to use KNN as a similarity-based classifier while maintaining the interpretability of case-based reasoning.
Tier Balancing: Towards Dynamic Fairness over Underlying Causal Factors
Zeyu Tang (Carnegie Mellon University), Kun Zhang (Mohamed bin Zayed University of Artificial Intelligence)
CodeTabularFinance Related
π― What it does: Proposes and analyzes the dynamic fairness concept of Tier Balancing based on causal graphs, studying the interaction between decision-making and data generation processes to achieve long-term fairness;
π― What it does: The system studies the impact of time augmentation generated by natural interactions on self-supervised visual representation learning, and validates it in both 3D simulation environments and real video data.
π― What it does: The SOLOFusion framework is proposed, which combines long-term low-resolution and short-term high-resolution multi-view fusion for 3D object detection using only cameras, along with theoretical analysis and baseline implementation.
TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis
Haixu Wu (Tsinghua University), Mingsheng Long (Tsinghua University)
CodeAnomaly DetectionConvolutional Neural NetworkRecurrent Neural NetworkTransformerTime Series
π― What it does: A general time series analysis framework called TimesNet is proposed, which is based on multi-periodicity, converting one-dimensional time series into two-dimensional tensors and using TimesBlock for multi-scale 2D transformations.
Zhen Qin (SenseTime Research), Yiran Zhong (Shanghai AI Laboratory)
CodeTransformerTextSequential
π― What it does: This paper proposes a sequence modeling network based on Toeplitz matrices (TNN), which achieves token mixing through relative position encoding, eliminates the quadratic complexity of the attention matrix, and enables efficient processing of long sequences.
π― What it does: The Token Merging (ToMe) method is introduced in Vision Transformer (ViT), which reduces computational load and improves throughput by merging similar tokens in each Transformer block.
Topology-aware Robust Optimization for Out-of-Distribution Generalization
Fengchun Qiao (University of Delaware), Xi Peng (University of Delaware)
CodeDomain AdaptationOptimizationImageTime Series
π― What it does: This paper proposes Topology-Aware Robust Optimization (TRO), which combines distribution topology learning with robust optimization to address the out-of-distribution (OOD) generalization problem.
Toward Adversarial Training on Contextualized Language Representation
Hongqiu Wu (Shanghai Jiao Tong University), Min Zhang (Soochow University)
CodeOptimizationAdversarial AttackTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This paper studies adversarial training (AT) on pre-trained language models (PLM) and finds that traditional AT mainly perturbs the decoder with limited impact on the encoder. It subsequently proposes a new adversarial training methodβContextualized representation-Adversarial Training (CreAT), which aims to maximize the differences in contextualized representations, thereby achieving global worst-case optimization for the entire model. CreAT is applied in both the model pre-training and fine-tuning stages.
π― What it does: This paper proposes FedOV, a single-round federated learning framework designed for label skew, which utilizes open set voting to address the misclassification of unseen classes by local models.
π― What it does: This paper studies the selective classification problem and proposes a Softmax Response selection mechanism based on the classifier itself and entropy regularization to improve model accuracy.
Towards Inferential Reproducibility of Machine Learning Research
Michael Hagmann (Heidelberg University), Stefan Riezler (Heidelberg University)
CodeHyperparameter SearchTransformerText
π― What it does: In response to the randomness and unmeasurable noise in machine learning experiments, a statistical analysis framework based on Linear Mixed Effects Model (LMEM) and Generalized Likelihood Ratio Test (GLRT) is proposed to evaluate the significance of model performance differences, variance components, and reliability.
Towards Interpretable Deep Reinforcement Learning with Human-Friendly Prototypes
Eoin M. Kenny (Massachusetts Institute of Technology), Julie Shah
CodeExplainability and InterpretabilityReinforcement Learning
π― What it does: This paper proposes the Prototype-Wrapper Network (PW-Net), an 'explainable by design' method that allows any pre-trained deep reinforcement learning (RL) agent to make decisions using human-friendly prototypes.
Towards One-shot Neural Combinatorial Solvers: Theoretical and Empirical Notes on the Cardinality-Constrained Case
Runzhong Wang (Shanghai Jiao Tong University), Junchi Yan (Shanghai Jiao Tong University)
CodeOptimizationRecurrent Neural NetworkGraph Neural NetworkReinforcement LearningGraphTabularFinance Related
π― What it does: A differentiable cardinality-constrained neural network (CardNN) is designed to solve combinatorial optimization problems in one go through Sinkhorn or Gumbel-Sinkhorn layers, achieving end-to-end learning for prediction-optimization.
π― What it does: This paper studies methods for the robustness certification of Universal Perturbation (UP) in neural networks, proposing a combined framework of linear relaxation and integer linear programming, providing an optimal lower bound for UP, and offering error estimates for the entire data distribution.
Towards Stable Test-time Adaptation in Dynamic Wild World
Shuaicheng Niu (South China University of Technology), Mingkui Tan (South China University of Technology)
CodeDomain AdaptationOptimizationImage
π― What it does: This study improves the stability of Test-Time Adaptation (TTA) in wild testing scenarios (mixed perturbations, small batch sizes, online label imbalance) by proposing a Sharpness-Aware and Reliable Entropy Minimization (SAR) method, which can filter out high-gradient noise samples and approach flat minima, thereby suppressing model collapse and enhancing robustness.
π― What it does: Analyzes the phenomenon of representation dimension collapse caused by data heterogeneity in federated learning, provides a theoretical explanation, and proposes the addition of feature decorrelation regularization (FEDDECORR) during the local training phase to alleviate this issue.
π― What it does: This paper proposes a framework for information exchange among latent variables at different layers in Hierarchical Variational Autoencoders (HVAE), allowing for separate control of the bit rate at each layer, thus enabling customized optimization for different tasks (reconstruction, generation, representation learning).
Huan Wang (Northeastern University), Yun Fu (AInnovation Labs, Inc.)
CodeOptimizationConvolutional Neural NetworkImage
π― What it does: A structured pruning method based on preserving network trainability is proposedβTrainability Preserving Pruning (TPP). It maintains the trainability of the network during the pruning process through decorrelation regularization of the Gram matrices of the filters to be pruned and the filters to be retained, as well as regularization of the batch normalization parameters.
π― What it does: Proposes a Trainable Weight Averaging (TWA) method that optimizes historical model weights in a low-dimensional subspace during the early stages of training to achieve more efficient training and better generalization performance.
π― What it does: This paper explores the correlation between the deep understanding ability of language models and human brain representations by fine-tuning four baseline Transformer models (BART, LED, BigBird, LongT5) on the long narrative summarization task (BookSum) and comparing the fine-tuned models with baseline models in terms of fMRI brain activity while reading Chapter 9 of Harry Potter.
Jie Ren (Michigan State University), Jiliang Tang (Michigan State University)
CodeAdversarial AttackContrastive LearningImage
π― What it does: A transferable no-learning sample method (TUE) is proposed, which invalidates personal data in unauthorized training by generating perturbations.
Ruchi Guo (University of California), Long Chen (University of California)
CodeTransformerImagePhysics Related
π― What it does: A deep direct sampling method based on Transformer and PDE feature mapping is proposed for real-time reconstruction of the electrical impedance tomography (EIT) inverse problem.
Transformer-based model for symbolic regression via joint supervised learning
Wenqiang Li (Chinese Academy of Sciences), Songsong Tian (Chinese Academy of Sciences)
CodeTransformerContrastive LearningTabular
π― What it does: A Transformer-based symbolic regression model is proposed, using a residual MLP as a feature extractor, and enhancing the recovery rate of expression skeletons through joint supervised learning (cross-entropy + contrastive learning).
π― What it does: Construct a Transformer-XL based autoregressive world model (TWM) that utilizes imagined generated trajectories for training strategies, achieving efficient learning within 100,000 interactions on the Atari 100k benchmark.
CodeTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: To address the immediate correction of errors in large pre-trained language models (PLMs) that have already been deployed, we propose the Sequential Model Editing (SME) task and introduce the Transformer-Patcher method, which adds and trains a small number of learnable neurons (patches) in the FFN of the last layer of the Transformer to fix errors.
π― What it does: Under the condition of very limited real interaction experience, the IRIS agent is proposed, which compresses images into tokens using a discrete autoencoder, and then performs autoregressive prediction with a GPT-style Transformer to train policies in its 'imagined' world model, achieving efficient reinforcement learning.
Qian Lou (University of Central Florida), Bo Feng (Meta Platforms, Inc.)
CodeAdversarial AttackTransformerText
π― What it does: This paper proposes a test-time executable text backdoor attack method called TrojText, which can make trigger sentences recognized as the target category by performing a small number of bit flips on the model weights without training data.
Turning the Curse of Heterogeneity in Federated Learning into a Blessing for Out-of-Distribution Detection
Shuyang Yu (Michigan State University), Jiayu Zhou (Michigan State University)
CodeAnomaly DetectionFederated LearningImage
π― What it does: A new anomaly detection method in federated learning called FOSTER is proposed, which utilizes data heterogeneity to synthesize virtual external class anomaly samples, thereby improving anomaly detection performance.
π― What it does: A data-free structured pruning method called TVSPrune is proposed, which identifies and removes non-discriminative filters by utilizing the total variation distance of the class conditional distribution of the convolutional layer outputs, without the need for retraining or accessing the original data.
Yi Tay (Google Research), Donald Metzler (Google Research)
CodeGenerationRetrievalTransformerLarge Language ModelTextChain-of-Thought
π― What it does: This study proposes the UL2 (Unifying Language Learning Paradigms) framework, which combines various self-supervised denoising tasks (R-Denoiser, S-Denoiser, X-Denoiser) through a Mixture-of-Denoisers (MoD) mixed pre-training objective, and introduces a mode switching mechanism, allowing a single model to perform consistently across various NLP tasks (from text generation to reasoning, question answering, classification, etc.).
π― What it does: The study investigates how to learn unbiased contrastive learning representations in the presence of data bias, proposing a new Ξ΅-SupInfoNCE contrastive loss and a FairKL debiasing regularization term.
Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles
Martin Bjerke (University of Cambridge), Benjamin Adric Dunn
CodeOptimizationRepresentation LearningConvolutional Neural NetworkAuto EncoderTabularTime Series
π― What it does: This paper proposes a differentiable neural latent variable model (faeLVM) that learns low-dimensional latent variables of neuronal populations through feature sharing (shared tuning curves) and cluster detection (soft clustering), achieving separation and tuning curve fitting of different neural groups within the same framework.
Understanding new tasks through the lens of training data via exponential tilting
Subha Maity (University of Michigan), Yuekai Sun (University of Michigan)
CodeDomain AdaptationImageBenchmark
π― What it does: Proposes a method to assign weights to training samples through an exponential tilting model to approximate the target domain distribution, thereby achieving model evaluation, fine-tuning, and model selection in the unlabeled target domain.
π― What it does: This study investigates the covariance structure of large filters in depthwise separable convolutions and proposes a no-learning initialization method based on closed-form multivariate Gaussian distribution.
π― What it does: This paper presents Uni-Mol, a unified 3D molecular representation learning framework that can directly take 3D coordinates as input and output.
π― What it does: Using the large-scale LAION 400M dataset, we first performed k-means clustering on the visual and textual features from CLIP to generate approximately one million pseudo-classes. We then pre-trained ViT using a softmax loss with random negative classes and random feature subsets, resulting in a general and compressed image representation.
Unified Discrete Diffusion for Simultaneous Vision-Language Generation
Minghui Hu (Qatar University), Ponnuthurai N. Suganthan (Nanyang Technological University)
CodeGenerationData SynthesisTransformerVision Language ModelDiffusion modelImageTextMultimodality
π― What it does: This paper proposes a unified discrete diffusion model called UniD3, which can simultaneously perform joint generation of vision and language as well as bidirectional modality translation.
UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph
Jinhao Jiang (Renmin University of China), Ji-Rong Wen (Renmin University of China)
CodeRetrievalGraph Neural NetworkLarge Language ModelContrastive LearningTextGraphBenchmark
π― What it does: This paper proposes a unified model called UniKGQA, which combines retrieval and reasoning in two stages of knowledge graph question answering (KGQA). It achieves the sharing and transfer of retrieval and reasoning through a unified semantic matching and information propagation module.
Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matching
Donggyun Kim (Korea Advanced Institute of Science and Technology), Seunghoon Hong (Korea Advanced Institute of Science and Technology)
CodeSegmentationMeta LearningTransformerImage
π― What it does: A general few-shot dense prediction framework VTM is proposed, which can learn any dense prediction task with only a small amount of labeled data.
π― What it does: This paper proposes Meta-EGN, an unsupervised learning framework based on MAML, designed to learn model initializations that can be quickly fine-tuned for combinatorial optimization problems, thereby achieving high-quality solutions for individual instances.
CodeDomain AdaptationOptimizationRepresentation LearningProtein Structure PredictionMultimodalityBiomedical Data
π― What it does: For datasets without correspondence between two domains, a Joint MDS method is proposed to achieve unsupervised manifold alignment and low-dimensional embedding.
π― What it does: An unsupervised meta-learning framework PsCo is proposed, which utilizes momentum networks and momentum queues to online construct few-shot tasks, and trains the model through pseudo-supervised contrastive learning to achieve few-shot classification.
Unsupervised Model Selection for Time Series Anomaly Detection
Mononito Goswami (Carnegie Mellon University), Andrey Kan (Amazon Research)
CodeAnomaly DetectionRecurrent Neural NetworkTime Series
π― What it does: This paper studies the unsupervised model selection problem for anomaly detection on unlabeled time series data, proposing a model selection framework based on proxy metrics and performing robust ranking aggregation.