These 737 ICLR 2023 papers come with a code repository. Each shows an AI one-line summary below β get the verified repo link + the full 6-part summary (innovation, method, data, results, limitations) and search every ICLR 2023 paper, free trial on arXivSub.
BEEF: Bi-Compatible Class-Incremental Learning via Energy-Based Expansion and Fusion
Fu-Yun Wang (Nanjing University), Peilin Zhao (Tencent AI Lab)
CodeClassificationContrastive LearningImage
π― What it does: The BEEF framework is proposed to address catastrophic forgetting in class-incremental learning, using an energy-based method for independent training and fusion of modules, while considering the compatibility of new and old tasks.
π― What it does: This paper proposes a no-training 'denoised random smoothing' method that combines pre-trained diffusion models (Diffusion Probabilistic Models) with existing classifiers to achieve provable robustness of images based on the L2 norm.
$\Lambda$-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells
Sajad Movahedi (University of Tehran), Babak N Araabi (University of Tehran)
CodeOptimizationNeural Architecture SearchImage
π― What it does: In the gradient-based differentiable architecture search method DARTS, the authors analyze the convergence issues caused by the weight-sharing framework and propose the 'Layer Alignment (Ξ)' metric. They design two regularization terms to enhance the consistency of layer gradients, thereby alleviating the performance collapse problem, and introduce the Ξ-DARTS method.
$\mathcal{O}$-GNN: incorporating ring priors into molecular modeling
Jinhua Zhu (University of Science and Technology of China), Tie-Yan Liu (Microsoft Research AI4Science)
CodeDrug DiscoveryGraph Neural NetworkGraph
π― What it does: A ring-enhanced graph neural network (O-GNN) is proposed, explicitly modeling molecular rings and iteratively updating them along with atoms and bonds.
π― What it does: A neighborhood subgraph-based Weisfeiler-Leman hierarchy (N-WL) is proposed to measure the expressiveness of graph neural networks, and based on this, the Graph Neighbourhood Neural Network (G3N) model is designed;
$k$NN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference
Benfeng Xu (University of Science and Technology of China), Yongdong Zhang (Institute of Artificial Intelligence)
CodeClassificationTransformerLarge Language ModelPrompt EngineeringText
π― What it does: A kNN-based prompting method called k NN Prompting is proposed, which utilizes the complete language model distribution generated by LLM as representation, constructs a local data storage, and makes predictions through nearest neighbor matching;
π― What it does: Designed and implemented TargetDiff, a SE(3)-equivariant diffusion model for generating 3D molecules that match protein targets, which can be used for affinity ranking and prediction.
π― What it does: A 2Dβ3D knowledge distillation method based on panoramic images is proposed, utilizing a pre-trained 2D segmentation network to enhance 3D voxel semantic segmentation, and a 3D Segmenter network is introduced.
π― What it does: This paper proposes 3D UX-Net, a lightweight 3D convolutional network that uses large kernel depthwise separable convolutions to simulate hierarchical Transformers for medical image segmentation.
π― What it does: This paper proposes a unified evaluation framework to compare different failure detection methods, revealing three major flaws in current evaluations, and validates through large-scale experiments that softmax response remains the best baseline.
π― What it does: A single-loop Bregman Alternating Projection Gradient (BAPG) algorithm is proposed for approximately solving the Gromov-Wasserstein distance, balancing computational efficiency and matching accuracy.
A GNN-Guided Predict-and-Search Framework for Mixed-Integer Linear Programming
Qingyu Han (Shandong University), Xiaodong Luo (Shenzhen Research Institute of Big Data)
CodeOptimizationGraph Neural NetworkTabular
π― What it does: This paper proposes a prediction-search framework based on graph neural networks (GNN) to quickly find high-quality feasible solutions for mixed-integer linear programming (MILP).
A Holistic View of Label Noise Transition Matrix in Deep Learning and Beyond
LIN Yong, Bo Han (Hong Kong Baptist University)
CodeClassificationOptimizationImage
π― What it does: This paper proposes a learning framework for statistically consistent classifiers in the presence of label noise by estimating the noise transition matrix (T). It first summarizes existing T estimation methods as the Minimum Geometric Envelope Operator (MGEO) and points out its inconsistency under posterior estimation errors. Subsequently, it introduces the Robust Bilevel Optimization (ROBOT) framework, which utilizes bilevel optimization and robust loss to achieve identifiable, consistent, and finite sample generalization guarantees for T without requiring perfect posterior estimation and anchor point assumptions.
π― What it does: A method called Detectron is proposed, based on a Constraint Inconsistency Classifier (CDC) and ensemble learning, for the rapid detection of harmful covariate shifts during deployment.
π― What it does: This paper proposes a memory-efficient class-incremental learning method called MEMO, and for the first time compares various CIL methods under a unified memory budget;
A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classification
Xiang Hu (Ant Group), Kewei Tu (ShanghaiTech University)
CodeClassificationExplainability and InterpretabilityTransformerText
π― What it does: A Symbolic-Neural model is proposed, which learns single-label or multi-label classification of text through a structured language model (Structured LM) without span labels, and predicts span-level labels at internal nodes, forming an interpretable label tree.
A probabilistic framework for task-aligned intra- and inter-area neural manifold estimation
Edoardo Balzani (New York University), Cristina Savin (New York University)
CodeTime Series
π― What it does: A probabilistic framework TAME-GP based on Gaussian process priors is proposed, which can separate shared and private variations within and outside of neural populations in a single trial and align them with task variables.
A Self-Attention Ansatz for Ab-initio Quantum Chemistry
Ingrid von Glehn (DeepMind), David Pfau (DeepMind)
CodeTransformerTabularPhysics Related
π― What it does: This paper proposes a self-attention-based wave function transformer (Psiformer) that can serve as an approximate Ansatz for the multi-electron SchrΓΆdinger equation.
π― What it does: This paper proposes a concise visual room rearrangement framework that utilizes two voxel-based semantic maps (target state and current state) along with a semantic search strategy to first locate the objects to be rearranged, then infer the target positions through map differences, and finally plan movements to achieve the rearrangement.
π― What it does: This study focuses on deep active learning and proposes the use of Snapshot Ensembles to estimate uncertainty and build efficient active learning algorithms.
A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks
Marc Anton Finzi (New York University), Andrew Gordon Wilson (University of British Columbia)
CodePhysics RelatedOrdinary Differential Equation
π― What it does: A local time method called Neural-IVP is proposed, which uses neural networks to approximate the solution of initial value PDEs, avoiding the catastrophic forgetting caused by traditional global minimization.
A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation
Hiroki Furuta (University of Tokyo), Shixiang Shane Gu (Google Research)
CodeKnowledge DistillationRobotic IntelligenceReinforcement Learning from Human FeedbackGraph Neural NetworkTransformerReinforcement LearningAgentic AIGraphBenchmark
π― What it does: This paper proposes a unified Morphology-Task Graph (MTG) representation and implements offline behavior distillation for multiple morphologies and tasks in the MxT-Bench environment, thereby training a single policy that can generalize across various morphologies and tasks.
π― What it does: This paper reinterprets soft threshold pruning as an implicit ISTA optimization problem and, based on this theoretical framework, designs learning rate adaptive threshold scheduling (LATS), simplified threshold scheduling (S-LATS), and continuation strategies (PGH scheduling), achieving efficient pruning during the sparsity increase process.
π― What it does: A variational autoencoder for Transformers (NVAE) is proposed, which treats the multi-vector embeddings of Transformers as a variable-sized mixture distribution and uses non-parametric variational information bottleneck (NVIB) for regularization.
π― What it does: This study constructed the AVFS dataset by collecting 638,180 triplet judgments of facial similarity and learned a facial embedding space that aligns with human perception.
π― What it does: This paper proposes a framework for the automated generation and use of auxiliary objectives, and based on this, designs the AANG algorithm for task-aware multi-task learning on specific end tasks, thereby improving the model's performance on low-resource NLP tasks.
Accurate Bayesian Meta-Learning by Accurate Task Posterior Inference
Michael Volpp (Karlsruhe Institute of Technology), Gerhard Neumann (Karlsruhe Institute of Technology)
CodeMeta LearningImageTabular
π― What it does: This paper proposes a new Bayesian Meta-Learning (BML) model GMM-NP, which achieves precise inference of task posteriors through Gaussian Mixture Models (GMM) and Trust Region Natural Gradient Variational Inference (TRNG-VI).
π― What it does: This paper proposes an Attention Retractable Transformer (ART) network for image super-resolution, denoising, and JPEG compression artifact removal, utilizing alternating modules of dense and sparse attention to expand the receptive field and improve recovery quality.
π― What it does: A message passing mechanism based on Allen-Cahn phase transition dynamics and attractive/repulsive forces (ACMP) is designed and implemented as a trainable neural ODE solver.
Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by Aggregation
Marius-Constantin Dinu (Johannes Kepler University Linz), Werner Zellinger (Austrian Academy of Sciences)
CodeDomain AdaptationImageTextTime Series
π― What it does: In unsupervised domain adaptation, for a given sequence of models obtained from different hyperparameters, this paper proposes a linear aggregation method based on importance-weighted least squares to solve for the optimal linear combination that minimizes target domain error.
π― What it does: This paper proposes a method based on Adversity (ADVERSITY) to train cooperative strategies that are high-performing, reasonable, and different from known agents in environments like Hanabi and Dec-POMDP.
Aleksandar Taranovic (Karlsruhe Institute of Technology), Gerhard Neumann (Bosch Center for Artificial Intelligence)
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningGenerative Adversarial NetworkTabular
π― What it does: A new adversarial imitation learning method (AILP) is proposed, which learns control policies by utilizing both demonstrations and preference feedback.
π― What it does: A self-supervised AE-FLOW model is proposed, which combines the reconstruction error of the autoencoder with the log-likelihood of the regularized flow for anomaly detection in medical images.
AGRO: Adversarial discovery of error-prone Groups for Robust Optimization
Bhargavi Paranjape (University of Washington), Hannaneh Hajishirzi (University of Washington)
CodeOptimizationTransformerTextBenchmarkAgriculture Related
π― What it does: We propose AGRO, an end-to-end method for group distributionally robust optimization that combines a learning model with a soft grouper without the need for pre-defined groups.
π― What it does: Aligning the IT layer representation of the neural network model with the IT neural data of gorillas to make it more similar to brain representations, thereby enhancing the consistency of human behavior and adversarial robustness.
Almost Linear Constant-Factor Sketching for $\ell_1$ and Logistic Regression
Alexander Munteanu (TU Dortmund University), David Woodruff
CodeOptimizationTabular
π― What it does: For high-dimensional large-scale data, a new oblivious linear projection (sketch) method is proposed for approximately solving β1 regression and logistic regression (including variance regularization versions), and it provides nearly linear sketch dimensions with constant factor approximation guarantees.
Alternating Differentiation for Optimization Layers
Haixiang Sun (ShanghaiTech University), Dacheng Tao (JD Explore Academy)
CodeOptimizationAuto EncoderTabularTime Series
π― What it does: This paper proposes a solver named Alt-Diff, which efficiently performs backpropagation for convex optimization layers with polyhedral constraints in deep neural networks.
An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation
Yuqiao Wen (University of Alberta), Lili Mou (University of Alberta)
CodeGenerationTransformerText
π― What it does: A method called EqHard-EM, an equal-sized hard EM algorithm, is proposed, which combines multi-decoder and multi-adapter architectures to generate diverse and high-quality dialogue responses.
π― What it does: The Bit Diffusion method is proposed to encode discrete data into binary bits and map them to continuous values (Analog Bits), enabling continuous time diffusion models to generate discrete images and text, and introduces Self-Conditioning and Asymmetric Time Intervals to enhance sampling quality.
Anamnesic Neural Differential Equations with Orthogonal Polynomial Projections
Edward De Brouwer (KU Leuven), Rahul G Krishnan
CodeTime SeriesBiomedical DataElectronic Health RecordsOrdinary Differential Equation
π― What it does: This paper proposes PolyODE, a temporal model that embeds orthogonal polynomial projections into neural ODEs, capable of maintaining long-term memory and performing backward reconstruction under irregular sampling.
π― What it does: This paper proposes the Anti-Symmetric Deep Graph Network (A-DGN), which views graph neural networks as continuous differential equations on graphs. By using an anti-symmetric weight matrix, it ensures stability and non-dissipation, thereby maintaining long-term node dependencies in deep structures and avoiding gradient vanishing/explosion.
Approximate Nearest Neighbor Search through Modern Error-Correcting Codes
Noam Touitou (Amazon), Nissim Halabi
CodeRetrievalOptimizationAuto EncoderTabular
π― What it does: A novel approximate nearest neighbor search algorithm, PCNN, is proposed, which utilizes Polar Codes for clustering in high-dimensional binary embedding space and achieves multi-probe through list decoding.
π― What it does: This study investigates the bottleneck of increasing the depth of graph Transformers and proposes the DeepGraph model, which enhances graph representation performance at deeper layers through substructure tokens and local attention.
Ask Me Anything: A simple strategy for prompting language models
Simran Arora (Stanford University), Christopher Re
CodeTransformerLarge Language ModelPrompt EngineeringText
π― What it does: Proposes an aggregation method that utilizes multiple imperfect but effective prompts to improve the zero/few-shot reasoning performance of large language models.
Yang Jiao (Tongji University), Chengtao Jian (Tongji University)
CodeOptimizationTabular
π― What it does: An Asynchronous Distributed Bi-level Optimization algorithm (ADBO) is proposed, which achieves bi-level optimization under non-convex upper and lower objective functions through single-loop updates, and provides theoretical proofs of convergence and iteration complexity.
π― What it does: Proposes a method to measure sample similarity by retaining augmented distribution overlap, and introduces the Augmentation Component Analysis (ACA) method to effectively implement this idea.
Aaron Palmer (University of Connecticut), Jinbo Bi (University of Connecticut)
CodeGenerationData SynthesisAuto EncoderImage
π― What it does: This paper proposes a novel generative autoencoder called the Goodness-of-Fit Autoencoder (GoFAE), which introduces a normality GoF test as a regularization objective at the mini-batch level and uses higher criticism at the global level to automatically select the regularization coefficient, thereby achieving statistical matching between the latent space distribution and the Gaussian prior class.
π― What it does: This paper proposes a self-supervised framework called ACT, which transforms pre-trained 2D visual Transformers or language Transformers into cross-modal teachers through an autoencoder, using their latent space features as the masking modeling target for a 3D point cloud Transformer student, achieving cross-modal knowledge distillation.
π― What it does: This paper proposes the AutoGT framework, which automatically searches for the optimal combination of graph Transformer structures and graph encoding strategies to complete graph classification tasks.
π― What it does: An automated graph data augmentation framework called GraphAug is proposed for graph classification tasks, generating augmented samples while keeping the labels unchanged.
Automatic Chain of Thought Prompting in Large Language Models
Zhuosheng Zhang (Shanghai Jiao Tong University), Alex Smola (Amazon Web Services)
CodeOptimizationTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
π― What it does: This paper proposes Auto-CoT, which utilizes large language models (LLMs) to automatically generate chain-of-thought (CoT) examples, achieving a few-shot prompting that does not rely on manual design, thereby providing performance comparable to manually crafted examples in reasoning tasks.
Automating Nearest Neighbor Search Configuration with Constrained Optimization
Philip Sun (Google Research), Sanjiv Kumar (Google Research)
CodeRetrievalOptimizationTabular
π― What it does: This paper proposes a constraint optimization-based quantization ANN hyperparameter tuning method that can automatically generate parameter configurations close to the Pareto front by only specifying the desired search cost or recall rate.
π― What it does: This paper proposes deploying existing Conditional Neural Processes (CNP) in an autoregressive (AR) manner to construct joint predictive distributions and generate coherent function samples without changing the model architecture and training process.
π― What it does: This paper proposes AUTOTRANSFER, which utilizes a task-model repository and Fisher Information Matrix-driven task embeddings to achieve cross-task knowledge transfer in graph neural network AutoML searches, significantly reducing search costs.
Satoshi Hara (Osaka University), Yuichi Yoshida (National Institute of Informatics)
CodeClassificationOptimizationTabular
π― What it does: A decision tree learning algorithm that is stable against training data perturbations has been designed, along with theoretical proofs and experimental validation.
Avoiding spurious correlations via logit correction
Sheng Liu (New York University), Carlos Fernandez-Granda (New York University)
CodeClassificationData-Centric LearningImage
π― What it does: This paper proposes a logit-corrected loss function (LC) and a Group MixUp data augmentation method to suppress spurious correlations in samples during training, thereby improving the balanced accuracy of the model across different subgroups.
π― What it does: This paper studies a binarized image restoration network and proposes a Basic Binarized Convolution Unit (BBCU). It designs different variants for the four modules of the network: head, body, upsampling, and tail, achieving an efficient and lightweight image restoration model.
π― What it does: The paper proposes a framework that utilizes a pure video (without action labels) pre-training model to enhance the sample efficiency of models based on reinforcement learning.
π― What it does: The Behavior Prior Representation (BPR) method is proposed, which learns state representations by behavior cloning the behavior policy from offline data, and applies any offline RL algorithm for policy learning after freezing that representation.
π― What it does: This paper proposes a BPPO algorithm for offline reinforcement learning based on PPO, which utilizes offline data to monotonically improve the behavior policy, thus achieving offline training without additional constraints.
π― What it does: This paper proposes a benchmark framework for Inverse Constraint Reinforcement Learning (ICRL) and builds reproducible experimental environments for two real tasks: robot control and autonomous driving. It also introduces a variational method, VICRL, to learn the posterior distribution of constraints. The performance of different algorithms in constraint inference and execution is evaluated by comparing various ICRL baselines.
π― What it does: Proposes the FedCIL framework to address the problem of federated incremental learning for clients continuously learning new categories, and implements memoryless forgetting suppression through generative replay.
π― What it does: A Dynamic Prior Knowledge (DPK) framework is proposed, utilizing teacher features as input to mix with students and adjusting the ratio through dynamic masking for knowledge distillation.
π― What it does: The BETTY automatic differentiation library has been developed to specifically address the high complexity and implementation difficulties of gradient computation in multi-layer optimization (MLO), enabling scalable training for large-scale MLO.
π― What it does: Utilizing BEV space to achieve cross-modal knowledge distillation, with the LiDAR detector as the teacher and the multi-view camera detector as the student, feature alignment is realized through a unified BEV representation for distillation of the student.
Beyond calibration: estimating the grouping loss of modern neural networks
Alexandre Perez-Lebel (Inria), Gael Varoquaux
CodeClassificationImageText
π― What it does: A method for estimating the Grouping Loss of modern neural networks is proposed, constructing a theoretical decomposition of the Grouping Loss and an estimable lower bound.
Hongyan Chang (National University of Singapore), Reza Shokri (National University of Singapore)
CodeFederated LearningTabular
π― What it does: This paper reveals how federated learning (FL) systems propagate group fairness biases among different participants through experimental and theoretical analysis, and provides specific manifestations of these biases in model parameters.
π― What it does: This paper presents a large-scale trained universal neural vocoder, BigVGAN, which can achieve zero-shot high-fidelity audio synthesis in unseen speaker, language, recording environment, and even musical contexts.
Zhoujun Cheng (University of Hong Kong), Tao Yu (University of Hong Kong)
CodeAI Code AssistantTransformerLarge Language ModelPrompt EngineeringMultimodalityTabular
π― What it does: The BINDER framework is proposed, which maps natural language input to executable programs and inserts a unified language model API (such as Codex) to extend the functionality of existing programming languages (SQL/Python), achieving untrained neural-symbolic integration.
π― What it does: This paper proposes Bit-Pruning, a framework for achieving multiply-less sparse dot products through bit-wise pruning during the training phase;
π― What it does: A bit-rate constrained distributionally robust optimization method (BR-DRO) is proposed, which can achieve robustness against unknown group shifts and label noise without relying on the labels of the training sample group.
Emiel Hoogeboom (Google Research), Tim Salimans (Google Research)
CodeGenerationData SynthesisDiffusion modelImage
π― What it does: A diffusion model that combines fuzziness (thermal diffusion) with Gaussian noise is proposed, called Blurring Diffusion Models, providing a probabilistic framework under anisotropic noise in the frequency domain.
Boosting Causal Discovery via Adaptive Sample Reweighting
An Zhang (National University of Singapore), Tat-Seng Chua (National University of Singapore)
CodeTabularBiomedical Data
π― What it does: A model-agnostic adaptive sample weighting framework called ReScore is proposed to enhance the performance of differentiable causal structure learning methods.
Boosting Multiagent Reinforcement Learning via Permutation Invariant and Permutation Equivariant Networks
Jianye HAO, Zhen Wang (Northwestern Polytechnical University)
CodeReinforcement Learning
π― What it does: Two network structures utilizing permutation invariance (PI) and permutation equivariance (PE) are proposed (Dynamic Permutation Network DPN and Hypernetwork HPN), which can significantly compress the state-action space and improve sample efficiency by minimally modifying the feedforward layer of the existing MARL algorithm.
π― What it does: This study enhances the capability of subgraph MPNN in counting cycles and paths at the node level, proposing I2-GNNs that significantly improve counting accuracy by assigning unique identifiers to the root node and its neighbors.
Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraint
Borui Zhang (Tsinghua University), Jiwen Lu (Tsinghua University)
CodeOptimizationExplainability and InterpretabilityConvolutional Neural NetworkTransformerImage
π― What it does: A novel optimizer Bort is proposed, which improves the interpretability and reversibility of neural networks by applying bounded orthogonal constraints to the network weights.
BrainBERT: Self-supervised representation learning for intracranial recordings
Christopher Wang (Massachusetts Institute of Technology), Andrei Barbu (Massachusetts Institute of Technology)
CodeRepresentation LearningTransformerTime SeriesBiomedical Data
π― What it does: Developed BrainBERT, a self-supervised model based on Transformer that converts SEEG brain recordings into context-aware embeddings, which can then be quickly and accurately used by a linear decoder to complete various neural decoding tasks;
π― What it does: The paper proposes a method to calibrate offset spikes by adjusting the initial membrane potential, achieving high-precision and low-latency conversion from ANN to SNN.
BSTT: A Bayesian Spatial-Temporal Transformer for Sleep Staging
Yuchen Liu (Chinese Academy of Sciences), Ziyu Jia (Chinese Academy of Sciences)
CodeClassificationExplainability and InterpretabilityTransformerTime SeriesBiomedical Data
π― What it does: A model combining Bayesian relational reasoning and Transformer, called BSTT, is proposed for automatic sleep staging, along with an interpretable spatial-temporal relationship graph.
π― What it does: This study investigates the reliability issues of sparse training models and proposes the CigL method to generate more reliable sparse networks with comparable performance.
Calibrating Transformers via Sparse Gaussian Processes
Wenlong Chen (Imperial College London), Yingzhen Li (Imperial College London)
CodeTransformerImageTextGraph
π― What it does: By replacing the softmax in the multi-head attention of the Transformer with a kernel function and utilizing Sparse Variational Gaussian Processes (SVGP) for Bayesian inference in the attention output space, we achieve uncertainty quantification and calibration for the Transformer.
Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems
Yewen Fan (Carnegie Mellon University), Kun Zhang (Mohamed bin Zayed University of Artificial Intelligence)
CodeRecommendation SystemTabular
π― What it does: This paper addresses the issue of maximization bias in advertising recommendation systems and proposes a variance-adjusted debiasing calibration method (VAD) that implements linear adjustments to prediction results to enhance calibration effectiveness.
Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories
Li-Cheng Lan (University of California), Cho-Jui Hsieh (University of California)
CodeReinforcement Learning
π― What it does: This paper proposes a 'Relay-Evaluation' assessment framework to test the generalization ability of reinforcement learning (RL) agents in a fixed environment regarding controllable states (i.e., states with high-reward strategies) and points out that many mainstream algorithms exhibit a high failure rate in this evaluation; it then introduces the Self-Trajectory Augmentation (STA) method, which explores starting from high-value states in its historical trajectory during training, significantly improving the success rate under Relay-Evaluation without reducing conventional testing performance.
Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study
Mingxu Tao (Peking University), Dongyan Zhao (Peking University)
CodeClassificationRepresentation LearningTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
π― What it does: This study investigates the encoding capability of BERT in task-incremental learning through probing and analyzes the topological structure of the representation subspace, demonstrating that BERT can retain knowledge of old tasks over the long term even without experience replay.
π― What it does: This paper proposes three simple design improvements for CNNs through the disassembly and experimentation of the Vision Transformer structure: patchifying input, increasing convolution kernel size, and reducing normalization and activation layers, thereby significantly enhancing the robustness of CNNs against out-of-distribution data.
CodeTransformerLarge Language ModelPrompt EngineeringText
π― What it does: This paper studies the transfer performance of using AutoPrompt to automatically generate discrete prompts across different pre-trained language models, and proposes the simultaneous use of multiple models (generator + evaluator) during prompt induction to enhance the cross-model generality of prompts.
CodeOptimizationExplainability and InterpretabilityConvolutional Neural NetworkImageTabular
π― What it does: This paper proposes a method to evaluate and learn optimal baseline values through interpretable causal patterns to enhance the credibility of Shapley value explanations for deep neural networks.
π― What it does: By optimizing adversarial queries (canary) for the Shadow model, the effectiveness of black-box membership inference attacks is enhanced.
CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning
Samuel Maddock (University of Warwick), Pierre Stock (Meta AI)
CodeFederated LearningSafty and PrivacyAdversarial AttackImageText
π― What it does: This paper proposes an empirical attack method called CANIFE, which evaluates model privacy leakage in federated learning through the careful design of 'suspicious samples' (canary), capable of measuring privacy leakage in each round under real threat models.
Capturing the Motion of Every Joint: 3D Human Pose and Shape Estimation with Independent Tokens
Sen Yang (Southeast University), Gang YU
CodePose EstimationTransformerImageVideo
π― What it does: The paper proposes a Transformer framework based on independent Tokens for estimating 3D human pose and shape from single frames or videos.
Xinyi Wang (University of California), William Yang Wang (University of California)
CodeDomain AdaptationAuto EncoderImageBenchmark
π― What it does: A lightweight mini-batch sampling strategy based on causal balancing is proposed for domain generalization tasks. By learning latent covariates and matching balanced scores, unbiased training batches are constructed to suppress spurious correlations.
Causal Estimation for Text Data with (Apparent) Overlap Violations
Lin Gui (University of Chicago), Victor Veitch (Google Research)
CodeTransformerSupervised Fine-TuningText
π― What it does: This paper studies how to estimate the causal effect of a certain attribute in text data, such as the impact of polite versus rude emails on response time. The authors propose a method to handle causal identification and obtain robust causal estimates in the presence of apparent overlap violations.
Causal Reasoning in the Presence of Latent Confounders via Neural ADMG Learning
Matthew Ashman (University of Cambridge), Cheng Zhang (Microsoft Research)
CodeFlow-based ModelAuto EncoderTabularSequential
π― What it does: A learning framework for ADMG based on neural autoregressive flow (N-ADMG) is proposed, achieving gradient-learnable causal inference and reasoning for causal models with latent confounders and nonlinear structures.
Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems
Phillip Lippe (University of Amsterdam), Efstratios Gavves (University of Amsterdam)
CodeRepresentation LearningFlow-based ModelAuto EncoderTime Series
π― What it does: The iCITRIS method is proposed, extending CITRIS to simultaneously learn multidimensional causal variables and causal graphs containing instantaneous causal relationships in time series.
Certifiably Robust Policy Learning against Adversarial Multi-Agent Communication
Yanchao Sun (University of Maryland), Furong Huang (University of Maryland)
CodeAdversarial AttackReinforcement Learning
π― What it does: A provably robust multi-agent communication defense framework named Ablated Message Ensemble (AME) is proposed, capable of withstanding attacks where up to half of the messages are arbitrarily tampered with during testing.
Certified Defences Against Adversarial Patch Attacks on Semantic Segmentation
Maksym Yatsura (Robert Bosch GmbH), Jan Hendrik Metzen (Robert Bosch GmbH)
CodeSegmentationAdversarial AttackImage
π― What it does: A general authentication defense framework called DEMASKED SMOOTHING is proposed to protect semantic segmentation models against adversarial patch attacks.
π― What it does: A certification training method called SABR is proposed, which is based on selecting small and precise sub-regions in the input space for interval propagation;
Zizhang Chen (Brandeis University), Pengyu Hong (Brandeis University)
CodeGraph Neural NetworkGraph
π― What it does: A method based on Influence Functions is proposed to estimate the impact of removing nodes or edges on model parameters and prediction performance in Graph Convolutional Networks (GCN);