π― What it does: A continuous normalization flow model named FlowMM has been developed, utilizing Riemannian Flow Matching to generate periodic crystal structures and achieve two major tasks: crystal structure prediction (CSP) and novel material generation (DNG).
From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble
Qianlong Wen (University of Notre Dame), Yanfang Ye (University of Notre Dame)
CodeKnowledge DistillationRepresentation LearningDrug DiscoveryGraph Neural NetworkContrastive LearningGraphBiomedical Data
π― What it does: A plugin-based multi-granularity graph semantic integration framework MGSE is proposed, which utilizes knowledge distillation to learn and integrate representations from a single teacher model at different granularities;
From Neurons to Neutrons: A Case Study in Interpretability
Ouail Kitouni (Massachusetts Institute of Technology), Mike Williams (Massachusetts Institute of Technology)
CodeOptimizationExplainability and InterpretabilityKnowledge DistillationRepresentation LearningTabularPhysics Related
π― What it does: This study investigates the use of Mechanistic Interpretability methods to explain neural networks trained on nuclear physics data, uncovering low-dimensional representations within the network that match known semi-empirical mass formulas (SEMF) and shell models, and deriving an improved nuclear binding energy formula through symbolic regression.
FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning
Yuwei Fu (McGill University), Benoit Boulet (McGill University)
CodeRobotic IntelligenceTransformerReinforcement LearningVision Language ModelMultimodality
π― What it does: Utilizing a pre-trained Vision-Language Model (VLM) as a reward signal to improve sample efficiency in sparse reward reinforcement learning tasks, the FuRL method is proposed and validated on the Meta-world MT10.
Gated Linear Attention Transformers with Hardware-Efficient Training
Songlin Yang (Massachusetts Institute of Technology), Yoon Kim (Massachusetts Institute of Technology)
CodeTransformerLarge Language ModelText
π― What it does: Proposes the FLASHLINEARATTENTION hardware-efficient linear attention algorithm and introduces data-dependent gating (GLA) in the Transformer architecture, achieving performance competitive with traditional softmax Transformers from linear attention.
Gaussian Plane-Wave Neural Operator for Electron Density Estimation
Seongsu Kim (Pohang University of Science and Technology), Sungsoo Ahn (Pohang University of Science and Technology)
CodeGraph Neural NetworkTabularPhysics Related
π― What it does: This paper proposes a Gaussian Plane Wave Hybrid Basis Neural Operator (GPWNO) for efficient prediction of electronic density in molecules and solids.
Generalist Equivariant Transformer Towards 3D Molecular Interaction Learning
Xiangzhe Kong (Tsinghua University), Yang Liu (Tsinghua University)
CodeDrug DiscoveryGraph Neural NetworkTransformerGraphBiomedical Data
π― What it does: This paper proposes a unified representation of different molecules (proteins, small molecules, nucleic acids, etc.) as a geometric graph of sets, and designs a Generalist Equivariant Transformer (GET) to achieve efficient learning of this representation.
π― What it does: A generalization theory for clean-label backdoor attacks is proposed, along with an algorithm-independent error upper bound. Based on this theory, a trigger combining adversarial noise and indiscriminate poisoning is designed.
Generalizing Knowledge Graph Embedding with Universal Orthogonal Parameterization
Rui Li (Renmin University of China), Xu Chen (Renmin University of China)
CodeAdversarial AttackGraph Neural NetworkGraph
π― What it does: A general knowledge graph embedding framework called GoldE has been designed and implemented. This framework achieves arbitrary dimensional and geometric (Euclidean, elliptical, hyperbolic) relationship transformations through unified orthogonal parameterization, and integrates different geometries on product manifolds to adapt to the heterogeneous topology of graphs.
GeoAB: Towards Realistic Antibody Design and Reliable Affinity Maturation
Haitao Lin (Zhejiang University), Stan Z. Li (Westlake University)
CodeOptimizationDrug DiscoveryGraph Neural NetworkNeural Radiance FieldGraphBiomedical Data
π― What it does: GeoAB proposes a generation-optimization framework for the co-design of antibody CDRs and affinity maturation, which includes Geo-Initializer, Geo-Refiner, and Geo-Optimizer. It can generate CDR structures that comply with physical geometric constraints and accurately predict mutation ΞΞG.
GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model
Ling Li (Hong Kong University of Science and Technology), Wei Zeng (Hong Kong University of Science and Technology)
CodeSegmentationRetrievalTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodality
π― What it does: This study investigates geographic localization based on street view images and embeds reasoning capabilities in a large visual-language model (LVLM), proposing the GeoReasoner framework.
Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
Harry Dong (Carnegie Mellon University), Beidi Chen (Meta AI)
CodeCompressionComputational EfficiencyTransformerLarge Language ModelText
π― What it does: A constant-size low-rank cache called LESS is proposed, which is integrated with an eviction-based sparse KV cache strategy to significantly compress the KV cache during LLM inference while maintaining performance.
GiLOT: Interpreting Generative Language Models via Optimal Transport
Xuhong Li (Baidu Inc), Haoyi Xiong (Baidu Inc)
CodeGenerationExplainability and InterpretabilityTransformerLarge Language ModelText
π― What it does: The GILOT method is proposed, which explains the importance of input tokens by calculating the optimal transport distance on the output distribution generated by large language models.
Gradient Compressed Sensing: A Query-Efficient Gradient Estimator for High-Dimensional Zeroth-Order Optimization
Ruizhong Qiu (University of Illinois Urbana-Champaign), Hanghang Tong (University of Illinois Urbana-Champaign)
CodeOptimizationGraph
π― What it does: A new zero-order gradient estimator, GraCe, is proposed to significantly reduce query complexity for sparse gradient problems under high-dimensional constraints.
Gradient-based Visual Explanation for Transformer-based CLIP
Chenyang ZHAO, Antoni B. Chan (City University of Hong Kong)
CodeRetrievalExplainability and InterpretabilityTransformerVision Language ModelImage
π― What it does: A gradient visualization explanation method for the CLIP visual-language model, called Grad-ECLIP, is proposed to generate high-quality heatmaps for image-text matching results.
Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Method
Kishaan Jeeveswaran (Eindhoven University of Technology), Bahram Zonooz (Eindhoven University of Technology)
CodeDomain AdaptationContrastive LearningImage
π― What it does: A three-stage domain incremental learning method named DARE is proposed, aimed at alleviating catastrophic forgetting caused by representation drift.
π― What it does: This paper proposes a graph distillation method called GDEM based on spectral basis matching, which generates synthetic graphs that allow GNNs to achieve the same performance on small graphs as on the original large graphs without relying on specific GNNs.
π― What it does: A graph generation framework called GruM based on diffusion mixing is proposed, which directly captures the graph topology by learning the mixture of the final graph structure, achieving fast convergence and high-quality generation.
Jiate Li (Nanchang University), Binghui Wang (Illinois Institute of Technology)
CodeExplainability and InterpretabilityAdversarial AttackGraph Neural NetworkGraph
π― What it does: This study investigates the robustness of GNN interpreters against adversarial perturbations in graph structures and proposes two attack methods based on loss and deduction.
π― What it does: This paper proposes and trains a general Graph Position and Structure Encoder (GPSE) that can learn and reconstruct various Position and Structure Encodings (PSE) from graph structures in a self-supervised manner, which can then be directly used as feature enhancement for any GNN or graph Transformer model.
Graph-based Forecasting with Missing Data through Spatiotemporal Downsampling
Ivan Marisca (UniversitΓ‘ della Svizzera italiana), Filippo Maria Bianchi (UiT the Arctic University of Norway)
CodeGraph Neural NetworkGraphTime Series
π― What it does: A hierarchical spatiotemporal downsampling-based graph neural network framework is proposed, which directly predicts on synchronized spatiotemporal sequences with missing values without the need for prior imputation.
GRATH: Gradual Self-Truthifying for Large Language Models
Weixin Chen (University of Illinois at Urbana-Champaign), Bo Li (University of Chicago)
CodeTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
π― What it does: The GRATH method is proposed, which enhances the model's truthfulness under OOD queries by allowing the LLM to generate pairs of true and false answers itself and using DPO for iterative fine-tuning.
π― What it does: A fluid prediction framework called HelmFluid is proposed based on the Helmholtz theorem, which models fluid dynamics by learning potential and stream functions, and utilizes a multi-scale multi-head integration architecture for future fluid field predictions.
π― What it does: In response to the scenario selection and ranking problem in two-stage stochastic programming (2SP), the HGCN2SP model is proposed, which encodes scenarios using a hierarchical graph structure and trains within a reinforcement learning framework.
Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks
Arjun Karuvally (University of Massachusetts Amherst), Hava T Siegelmann
CodeRecurrent Neural NetworkTransformerSequential
π― What it does: This paper proposes a wave-based working memory model (TWM) and connects its theory with recurrent neural networks (RNN) and self-attention Transformers, demonstrating that RNNs naturally form hidden propagating waves when learning linear history-dependent systems (HDS), thereby improving gradient propagation.
Hierarchical Integral Probability Metrics: A distance on random probability measures with low sample complexity
Marta Catalano (Luiss University), Hugo Lavenant (Bocconi University)
Code
π― What it does: A new distance (Hierarchical Integral Probability Metric, HIPM) is proposed to measure the similarity of random probability distributions.
π― What it does: This paper proposes the Highway Value Iteration Network (Highway VIN), which integrates highway value iteration into the planning module of VIN and adds an aggregate gate, exploration module, and filter gate to enhance long-term planning capabilities.
π― What it does: This study investigates how to accelerate the training and inference of Neural ODEs from the perspective of control theory through minimum time control and Lyapunov methods.
How Interpretable Are Interpretable Graph Neural Networks?
Yongqiang Chen (Chinese University of Hong Kong), James Cheng (Chinese University of Hong Kong)
CodeExplainability and InterpretabilityGraph Neural NetworkGraph
π― What it does: This paper proposes a theoretical framework to analyze the expressive power of explainable graph neural networks (XGNNs) and finds that existing attention-based XGNNs exhibit significant errors when approximating subgraph multilinear extensions (SubMT), leading to compromised interpretability and out-of-distribution (OOD) generalization. Based on this, the authors design a new XGNN architecture called Graph Multilinear Net (GMT), which more accurately approximates SubMT through linearization or random subgraph sampling, and further retrains the classifier using a frozen subgraph extractor after training to enhance performance.
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningTabular
π― What it does: This paper proposes an offline imitation learning method called ILID, which utilizes diverse behaviors from non-expert demonstrations to select beneficial actions from noisy data and learn a policy.
How Universal Polynomial Bases Enhance Spectral Graph Neural Networks: Heterophily, Over-smoothing, and Over-squashing
Keke Huang (National University of Singapore), Pietro Lio
CodeGraph Neural NetworkGraph
π― What it does: A unified polynomial basis (UniBasis) and a graph convolutional network based on it, called UniFilter, are proposed to handle different heterogeneous graphs and suppress over-smoothing and over-compression.
π― What it does: A novel inverse reinforcement learning framework is proposedβHybrid Inverse Reinforcement Learning (Hybrid IRL), which significantly reduces exploration and interaction requirements by simultaneously utilizing online data and expert demonstrations within the policy search.
Hybrid$^2$ Neural ODE Causal Modeling and an Application to Glycemic Response
Bob Junyi Zou (Stanford University), Emily Fox
CodeBiomedical DataOrdinary Differential Equation
π― What it does: A hybrid model combining ODE mechanisms and neural networks is proposed, optimizing both prediction and causal effects through a mixed loss function.
π― What it does: This paper proposes a hierarchical attention network for audio-visual speech separation, IIANet, which achieves higher quality separation by cross-utilizing intra-modal and inter-modal attention in the audio and visual pathways.
π― What it does: This paper proposes injecting heavy-tailed noise with infinite variance into SGD and proves that the weight matrix obtained after training in a hyperparameterized single hidden layer network can achieve compressibility with high probability at any compression ratio.
π― What it does: This paper proposes an implicit neural representation method based on operator learning (O-INR), treating the coordinate-value mapping as an operator between function spaces, rather than traditional MLP predictions.
Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning
Wei Li (Zhejiang University), Mohan Kankanhalli (National University of Singapore)
CodeRetrievalTransformerLarge Language ModelContrastive LearningImageTextMultimodality
π― What it does: This paper proposes a Multimodal Combination Learning (MCL) method that utilizes a frozen LLM and a CLIP visual encoder. By training on generated Multimodal Combination data (MMC), it aims to improve the understanding and generation of complex multimodal queries through visual-language mapping and context retrieval tasks.
π― What it does: This paper presents a unified perspective that equates the implementation of all zero-copy diffusion models in inverse problems to the use of an isotropic Gaussian approximation for the intractable denoising posterior, and based on this, optimizes the posterior covariance through maximum likelihood estimation to improve existing methods.
Improving Equivariant Graph Neural Networks on Large Geometric Graphs via Virtual Nodes Learning
Yuelin Zhang (Renmin University of China), Wenbing Huang (Renmin University of China)
CodeGraph Neural NetworkGraph
π― What it does: We propose FastEGNN, which learns a set of ordered virtual nodes to enable E(3)-equivariant GNNs to maintain global message passing on large-scale geometric graphs while balancing efficiency and accuracy.
π― What it does: A gradient-guided nested sampling algorithm GGNS is designed, combining techniques such as differentiable programming, Hamiltonian slice sampling, clustering decomposition, and dynamic parallelization to achieve efficient sampling and model comparison in high-dimensional multimodal Bayesian inference.
π― What it does: The GIC (Group Inference via data Comparison) method is proposed, which accurately infers group labels by utilizing the distribution differences between the training set and the comparison set, thereby improving the worst group accuracy in tasks with spurious correlations.
π― What it does: An adaptive decoding algorithm is proposed, which dynamically determines the size of the candidate set for the next token based on entropy and confidence increments to enhance the diversity and coherence of open-ended text generation.
π― What it does: This paper proposes an improved token-based world model, designs and implements a Parallel Observation Prediction (POP) mechanism, and constructs a new REM agent.
Improving Transformers with Dynamically Composable Multi-Head Attention
Da Xiao (Beijing University of Posts and Telecommunications), xingyuan yuan
CodeTransformerLarge Language ModelImageText
π― What it does: A dynamic composable multi-head attention (DCMHA) mechanism is proposed to replace the traditional MHA, enhancing the expressive power of the Transformer and reducing head redundancy.
In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation
Shiqi Chen (City University of Hong Kong), Junxian He (Hong Kong University of Science and Technology)
CodeTransformerLarge Language ModelText
π― What it does: This paper proposes 'in-context sharpness' as an unsupervised metric for identifying and correcting hallucinations by analyzing the activation patterns in the hidden states of large language models, and designs an Activation Decoding method based on this metric.
Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning
Xinran Li (Hong Kong University of Science and Technology), Jun Zhang (Hong Kong University of Science and Technology)
CodeReinforcement LearningSequential
π― What it does: This paper proposes an Individual Contribution-based Exploration Scaffold (ICES) method, which evaluates the impact of each agent's actions on global potential state transitions to guide their exploration behavior in multi-agent reinforcement learning.
Inferring Change Points in High-Dimensional Linear Regression via Approximate Message Passing
Gabriel Arpino (University of Cambridge), Ramji Venkataramanan (University of Cambridge)
CodeAnomaly DetectionOptimizationTime Series
π― What it does: A high-dimensional linear regression change point detection method based on Approximate Message Passing (AMP) is proposed, which can simultaneously estimate signal parameters and change point locations, and provide uncertainty quantification.
Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experiments
Allen Tran (Netflix), Nathan Kallus (Cornell University)
CodeReinforcement LearningTabularElectronic Health Records
π― What it does: The study investigates how to infer the long-term causal effects of sustained interventions based solely on short-term experimental data;
Information-Directed Pessimism for Offline Reinforcement Learning
Alec Koppel (J.P. Morgan), Sumitra Ganesh
CodeReinforcement LearningTabular
π― What it does: This paper proposes an offline reinforcement learning lazy policy optimization framework IDP based on discrete Stein mismatch degree, and presents two algorithms: IDP-VI and IDP-Q.
Jingxuan He (ETH Zurich), Martin Vechev (ETH Zurich)
CodeGenerationSafty and PrivacyAI Code AssistantTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This paper proposes SafeCoder, which combines security refinement and instruction tuning to enhance the security of code generated by LLMs during the instruction tuning phase while maintaining functional correctness.
Integrated Hardware Architecture and Device Placement Search
Irene Wang (Georgia Institute of Technology), Divya Mahajan (Georgia Institute of Technology)
CodeOptimizationTransformerLarge Language ModelText
π― What it does: This study proposes the PHAZE framework, which achieves joint optimization of hardware accelerator architecture and distributed training model allocation strategies, automatically generating optimal accelerator configurations and device placement schemes.
π― What it does: A 3D molecular diffusion model based on interactive retrieval enhancement, IRDIFF, is proposed for drug molecule generation targeting protein sites.
π― What it does: This paper proposes InterLUDE, a semi-supervised learning framework that interacts with labeled and unlabeled samples in the embedding space and loss function.
InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretation
Jacob Yoke Hong Si (University of Toronto), Rahul Krishnan
CodeExplainability and InterpretabilityKnowledge DistillationTransformerLarge Language ModelPrompt EngineeringAuto EncoderTabular
π― What it does: This paper proposes InterpreTabNet, an improved TabNet model that utilizes Gumbel-Softmax sampling and KL divergence regularization to generate sparse, non-overlapping attention masks, thereby enhancing the interpretability of tabular data; it also combines GPT-4 to provide natural language explanations for the masks.
π― What it does: A generalizable routing problem solver based on Transformer, INViT, is proposed, which can achieve near-optimal solutions on larger-scale and differently distributed instances while being trained only on small-scale, uniformly distributed instances.
Is Kernel Prediction More Powerful than Gating in Convolutional Neural Networks?
Lorenz K Muller
CodeRestorationConvolutional Neural NetworkImage
π― What it does: This paper explores the equivalence and practicality of two methods for dynamic weight prediction in convolutional neural networks: HyperNetwork and gating mechanisms, and applies them to the task of image denoising.
π― What it does: A new diffusion model called Isometric Diffusion is proposed, which guides the model to learn a geometrically reasonable latent space through geometric regularization, achieving better decoupling of the latent space.
π― What it does: Proposes an Iterative Regularization Policy Optimization (IRPO) framework that alternates between offline imitation learning and online reinforcement learning, using the policies obtained from online learning to generate new demonstrations, thereby continuously improving policy quality under imperfect demonstrations.
Iterative Search Attribution for Deep Neural Networks
Zhiyu Zhu (University of Sydney), Jun Shen (University of Wollongong)
CodeClassificationExplainability and InterpretabilityConvolutional Neural NetworkImage
π― What it does: This paper proposes and implements an Iterative Search Attribution method (ISA) based on gradient ascent and descent, which improves attribution accuracy by gradually pruning unimportant features and introducing scale factors.
π― What it does: This paper proposes a neural Granger causality learning method based on input-output Jacobian matrix regularization, which can simultaneously estimate summary and full-time causal graphs within a single model.
Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization
Haocheng Xi (Tsinghua University), Jun Zhu (Tsinghua University)
CodeComputational EfficiencyTransformerImageText
π― What it does: Jetfire is proposed, a full quantization training method based on INT8 data streams and block-level quantization for Transformer pre-training.
π― What it does: This study proposes the Kepler Codebook, treating codebook training as a Kepler sphere packing problem. It designs Irwin-Hall distribution regularization and codebook partition strategies to address the issues of codebook collapse and low utilization in traditional VQ training, further achieving high-quality image reconstruction and generation.
Kernel Debiased Plug-in Estimation: Simultaneous, Automated Debiasing without Influence Functions for Many Target Parameters
Brian M Cho, Ivana Malenica (Harvard University)
Code
π― What it does: A method called Kernel Debiased Plug-in Estimation (KDPE) is proposed to eliminate plug-in bias in nonparametric models and achieve efficient estimation without relying on influence functions.
π― What it does: Proposes Kernel Semi-Implicit Variational Inference (KSIVI), which explicitly solves the lower-level minimization problem of SIVI-SM in RKHS, obtaining the KSD objective and achieving semi-implicit variational inference without lower-level optimization.
KernelSHAP-IQ: Weighted Least Square Optimization for Shapley Interactions
Fabian Fumagalli (Bielefeld University), Barbara Hammer (Bielefeld University)
CodeOptimizationExplainability and InterpretabilityConvolutional Neural NetworkTransformerImageTextTabular
π― What it does: The KernelSHAP-IQ method is proposed for approximating high-order Shapley interactions (SII), treating SII as a solution to a weighted least squares (WLS) optimization problem.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
Zirui Liu (Rice University), Xia Hu (Rice University)
CodeCompressionOptimizationComputational EfficiencyTransformerLarge Language ModelText
π― What it does: A two-bit KV cache quantization method called KIVI is proposed, which uses channel-wise quantization for the key cache and token-wise quantization for the value cache.
Knowledge-aware Reinforced Language Models for Protein Directed Evolution
Yuhao Wang (Zhejiang University), Huajun Chen (Zhejiang University)
CodeDrug DiscoveryReinforcement LearningBiomedical Data
π― What it does: A protein directed evolution model KnowRLM based on amino acid knowledge graph and reinforcement learning is proposed, which can efficiently search and screen high-fitness mutants.
π― What it does: In collaborative multi-agent reinforcement learning, a method is proposed to accelerate learning by generating goal-achieving trajectories in the latent space and providing intrinsic rewards guided by potential goals.
LangCell: Language-Cell Pre-training for Cell Identity Understanding
Suyuan Zhao (Tsinghua University), Zaiqing Nie (Tsinghua University)
CodeClassificationRecognitionData-Centric LearningTransformerContrastive LearningTextBiomedical Data
π― What it does: The LangCell framework is proposed, which combines single-cell transcriptome data with natural language descriptions for joint pre-training, achieving cross-modal understanding of cell identities.
Fenghao Lei (Zhejiang University), Chaoyi Pang (NingboTech University)
CodeSafty and PrivacyReinforcement LearningSequentialStochastic Differential Equation
π― What it does: Proposes Langevin Policy and Langevin Actor-Critic (LAC) for safe reinforcement learning, directly approximating the safe optimal policy through sampling and achieving efficient training.
Large Scale Dataset Distillation with Domain Shift
Noel Loo (Massachusetts Institute of Technology), Daniela Rus
CodeDomain AdaptationKnowledge DistillationImage
π― What it does: A scalable large-scale dataset distillation algorithm D3S is proposed, reframing the dataset distillation problem as a domain shift problem, and optimizing the distribution and labels of the distilled data by approximating KL divergence;
Larimar: Large Language Models with Episodic Memory Control
Payel Das (IBM AI Research), Pin-Yu Chen
CodeGenerationTransformerLarge Language ModelText
π― What it does: A memory module called Larimar has been designed, which allows for external writing and reading, enabling large language models to update and forget knowledge during the testing phase through one-time writing without the need for retraining.
CodeOptimizationConvolutional Neural NetworkTransformerLarge Language ModelImageText
π― What it does: The LASER algorithm is proposed, which performs low-rank linear compression and transmission of gradients over wireless noisy channels, enhancing the communication efficiency of distributed training.
Jianke Yang (University of California San Diego), Rose Yu (University of California San Diego)
CodeGenerationData SynthesisAuto EncoderGenerative Adversarial NetworkTime Series
π― What it does: A generative model named LaLiGAN is proposed, which can automatically discover nonlinear symmetries from high-dimensional data and learn the corresponding linear latent space transformations.
Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning
Jinsoo Yoo, Geoff Pleiss (University of British Columbia)
CodeOptimizationImage
π― What it does: Proposes Layerwise Proximal Replay (LPR), which combines experience replay with proximal point optimization in online continual learning to stabilize network updates.
Learning Causal Dynamics Models in Object-Oriented Environments
Zhongwei Yu (Institute of Automation Chinese Academy of Sciences), Dengpeng Xing (Institute of Automation Chinese Academy of Sciences)
CodeTabular
π― What it does: Proposes the Object-Oriented Causal Dynamics Model (OOCDM), which can learn causal dynamics models in large-scale objectified environments.
π― What it does: An end-to-end deep reinforcement learning method is designed to achieve coverage path planning in unknown environments using continuous control signals.
Learning Decision Policies with Instrumental Variables through Double Machine Learning
Daqian Shao (University of Oxford), Marta Kwiatkowska (University of Oxford)
CodeOptimizationReinforcement LearningTabular
π― What it does: Using instrumental variables (IV) and a double machine learning (DML) framework in offline data, a nonlinear IV regression method DML-IV is proposed for learning causal effects and deriving decision strategies.
Learning Decision Trees and Forests with Algorithmic Recourse
Kentaro Kanamori (Fujitsu Limited), Yuichi Ike (Kyushu University)
CodeClassificationOptimizationExplainability and InterpretabilityAdversarial AttackTabularFinance Related
π― What it does: This study investigates a method for learning accurate tree models and forest models while ensuring executable algorithmic regression.
Learning from Integral Losses in Physics Informed Neural Networks
Ehsan Saleh (University of Illinois Urbana-Champaign), Matthew West (University of Illinois Urbana-Champaign)
CodeReinforcement LearningTabularPhysics Related
π― What it does: For partial integral equations that include integral terms, this paper proposes three methods for training Physics-Informed Neural Networks (PINN): deterministic sampling, double-sampling technique, and delayed target method, with a regularization improvement on the latter;
Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs
Jordan Dotzel (Cornell University), Zhiru Zhang (Cornell University)
CodeConvolutional Neural NetworkTransformerLarge Language ModelText
π― What it does: This paper first conducts a distribution analysis of the weights and activations of 30 large language models and convolutional networks, finding that most conform to the Student's t-distribution. Based on this distribution, a new four-bit floating-point quantization format, Student Float (SF4), is derived, which improves accuracy by about 0.7% compared to the existing NF4 across various LLMs. Furthermore, 'super-normal' support (SR, SP) is added to non-integer formats such as E2M1 and APoT, further enhancing precision and reducing hardware area consumption. Finally, a precision-area Pareto curve is drawn to illustrate the trade-offs of different data types.
Yoni Choukroun (Tel Aviv University), Lior Wolf (Tel Aviv University)
CodeTransformer
π― What it does: An end-to-end differentiable coding-decoding joint training framework is proposed, capable of simultaneously learning the generator matrix of binary linear block codes and the corresponding Transformer neural decoder.
Learning Mixtures of Gaussian Processes through Random Projection
Emmanuel Akeweje (Trinity College Dublin), Mimi Zhang (I-Form Advanced Manufacturing Research Centre)
CodeTime Series
π― What it does: This paper proposes an ensemble clustering framework that first randomly projects functional data to one dimension and then fits a unidimensional Gaussian mixture model, aiming to efficiently learn high-dimensional Gaussian process mixture models.
Learning Pseudo-Contractive Denoisers for Inverse Problems
Deliang Wei (East China Normal University), Fang Li (East China Normal University)
CodeRestorationSuper ResolutionImage
π― What it does: A pseudo-contractive deep denoiser is proposed, and a convergent plug-and-play recovery algorithm is designed based on this, achieving global convergence using Ishikawa iteration;
π― What it does: A Scale-Aware Spatio-Temporal Network (SASNet) is proposed to achieve event-driven motion deblurring at arbitrary spatial and temporal scales.
Learning Shadow Variable Representation for Treatment Effect Estimation under Collider Bias
Baohong Li (Zhejiang University), Kun Kuang (Zhejiang University)
CodeRepresentation LearningAuto EncoderTabular
π― What it does: The study investigates how to estimate causal effects in the presence of collider bias in observational data and proposes a method for automatically learning shadow variable representations without prior knowledge.
π― What it does: A learning enhancement method based on Transformer, called SAWT, has been developed for efficiently solving the Quadratic Assignment Problem (QAP).
Learning Surrogates for Offline Black-Box Optimization via Gradient Matching
Minh Hoang (Princeton University), Trong Nghia Hoang (Washington State University)
CodeOptimizationTabularBenchmark
π― What it does: This paper proposes a gradient matching-based offline black-box optimization surrogate model learning method (MATCH-OPT) and provides a theoretical proof of the upper bound of the surrogate gradient error and optimization performance loss.
π― What it does: A meta continual learning framework based on Bayesian principles (SB-MCL) is proposed, which learns a neural network as a data mapper during the meta-learning phase, allowing for the use of exponential family Bayesian sequential updates during the continual learning phase, thereby eliminating catastrophic forgetting.
Learning to Infer Generative Template Programs for Visual Concepts
R. Kenny Jones (Brown University), Daniel Ritchie (Adobe Research)
CodeSegmentationGenerationTransformerImage
π― What it does: A neural symbolic framework called Template Programs is proposed to infer generalizable generative programs from a small number of visual samples.
David Steinmann (TU Darmstadt), Kristian Kersting (TU Darmstadt)
CodeClassificationRecognitionImage
π― What it does: This paper proposes the Concept Bottleneck Memory Model (CB2M), which adds two layers of memory to the traditional concept bottleneck model to continuously utilize and reuse human interventions.
Learning to Route Among Specialized Experts for Zero-Shot Generalization
Mohammed Muqeeth (Massachusetts Institute of Technology International Business Machines), Colin Raffel (University of Toronto)
CodeTransformerLarge Language ModelSupervised Fine-TuningMixture of ExpertsTextBenchmark
π― What it does: This paper proposes a post-hoc adaptive token-level routing method called PHATGOOSE, which enables dynamic routing among a large number of parameter-efficient fine-tuned expert models (such as LoRA modules) to enhance the zero-shot generalization of pre-trained language models on unseen tasks.
Less is More: on the Over-Globalizing Problem in Graph Transformers
Yujie Xing (Beijing University of Posts and Telecommunications), Chuan Shi (Beijing University of Posts and Telecommunications)
CodeGraph Neural NetworkTransformerGraph
π― What it does: This paper reveals the problem of excessive globalization in graph Transformers and proposes the CoBFormer architecture to alleviate this issue.
CodeClassificationDomain AdaptationRepresentation LearningTransformerVision Language ModelContrastive LearningImage
π― What it does: A completely unsupervised transfer framework named TURTLE is proposed, which can automatically discover the true labels of a dataset from a pre-trained base model without using any labels.
Lie Neurons: Adjoint-Equivariant Neural Networks for Semisimple Lie Algebras
Tzu-Yuan Lin (University of Michigan), Maani Ghaffari (University of Michigan)
CodePoint Cloud
π― What it does: A neural network for adjoint equivariance of semisimple Lie algebras, called Lie Neurons, is proposed, which can handle inputs from any finite-dimensional semisimple Lie algebra.
Tuomas Oikarinen (University of California San Diego), Tsui-Wei Weng (University of California San Diego)
CodeExplainability and InterpretabilityConvolutional Neural NetworkTransformerImageMultimodality
π― What it does: A method called Linear Explanations is proposed, which views the activation of a single neuron as a linear combination of concepts, generating explanations through a concept activation matrix and sparse linear regression, while also introducing a simulation-based automatic evaluation framework.
Listwise Reward Estimation for Offline Preference-based Reinforcement Learning
Heewoong Choi (Seoul National University), Taesup Moon (Seoul National University)
CodeReinforcement LearningSequential
π― What it does: This paper proposes a new offline preference-based reinforcement learning method called LiRE, which constructs a complete trajectory ranking list (RLT) and effectively utilizes second-order preference information through the same ternary feedback to achieve more accurate reward function estimation.