π― What it does: This paper explores the behavior of ERM and OOD objectives in feature learning through theoretical analysis and experiments, and proposes the Feature Augmented Training (FeAT) iterative method to obtain richer features and enhance OOD generalization performance.
Understanding and Mitigating Copying in Diffusion Models
Gowthami Somepalli (University of Maryland), Tom Goldstein (University of Maryland)
CodeGenerationDiffusion modelImageText
π― What it does: Analyzes the replication behavior of diffusion models under text conditions and proposes various de-duplication strategies during training and inference.
π― What it does: By viewing contrastive learning as distributionally robust optimization, this paper provides a theoretical analysis that explains the tolerance of contrastive learning to negative sample sampling bias and proposes a new weighted InfoNCE lossβADNCEβto alleviate issues of excessive conservativeness and sensitivity to outliers.
Understanding Deep Gradient Leakage via Inversion Influence Functions
Haobo Zhang (Michigan State University), Jiayu Zhou (Michigan State University)
CodeSafty and PrivacyAdversarial AttackConvolutional Neural NetworkTransformerImageText
π― What it does: The problem of Deep Gradient Leakage (DGL) is analyzed by proposing and validating the Inverse Influence Function (I2F), providing both empirical and theoretical results.
Understanding Few-Shot Learning: Measuring Task Relatedness and Adaptation Difficulty via Attributes
Minyang Hu (Institute of Computing Technology, Chinese Academy of Sciences), Xilin CHEN
CodeMeta LearningImage
π― What it does: Proposed and validated the Task Attribute Distance (TAD) metric to measure the correlation between training tasks and new tasks, as well as the adaptation difficulty of new tasks;
Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry
Yong-Hyun Park (Seoul National University), Youngjung Uh (Seoul National University)
CodeGenerationDiffusion modelImageText
π― What it does: This paper analyzes the latent space of diffusion models through pullback metrics, extracts local latent bases, and utilizes them to achieve image editing in the latent space at a single moment. It further studies the evolution of the latent structure with diffusion steps and the impact of text prompts.
Undirected Probabilistic Model for Tensor Decomposition
Zerui Tao (Tokyo University of Agriculture and Technology), Qibin Zhao (RIKEN AIP)
CodeContrastive LearningMultimodalityTime Series
π― What it does: A framework for undirected tensor decomposition is constructed through deep energy-based models (EBM) to jointly learn tensor observations and latent factors, thereby achieving probabilistic modeling of non-Gaussian, multimodal data.
π― What it does: This paper proposes Uni-ControlNet, a unified framework that can simultaneously utilize various local (such as edges, depth, segmentation, etc.) and global (such as CLIP image embeddings) control signals within a single model, enabling composable text-to-image diffusion model control.
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Can Qin (Northeastern University), Ran Xu (Salesforce AI Research)
CodeSegmentationGenerationData SynthesisMixture of ExpertsDiffusion modelImageText
π― What it does: We propose and train UniControl, a unified diffusion model capable of handling multiple visual conditions (edges, segmentation, depth, skeletons, etc.) and text prompts to achieve controllable image generation.
Zheyun Qin (Shandong University), Xiankai Lu (Shandong University)
CodeSegmentationTransformerPoint Cloud
π― What it does: A prototype-based unified framework called PROTOSEG is proposed, which unifies semantic, instance, and panoptic segmentation tasks into classification problems of different granularities, using Transformers to extract point embeddings and achieve classification through dynamic prototype association and updating.
π― What it does: Proposes a Feature Multiplexing framework that allows multiple classification features to share the same embedding space, and based on this, designs a Unified Embedding that significantly reduces model parameters and latency.
π― What it does: Unified the problem of Off-Policy Learning to Rank as a Markov Decision Process (MDP) and directly learned the optimal ranking policy through offline reinforcement learning (RL).
Unified Segment-to-Segment Framework for Simultaneous Sequence Generation
Shaolei Zhang (Chinese Academy of Sciences), Yang Feng (Chinese Academy of Sciences)
CodeRecognitionGenerationTransformerTextAudio
π― What it does: A unified paragraph-to-paragraph framework (Seg2Seg) is proposed, which introduces latent paragraphs as a bridge to learn source-target adaptive mapping in real-time sequence generation (such as streaming ASR, synchronous MT, and synchronous ST) and achieves multi-task learning.
π― What it does: A unified threshold integrated sample-to-sample loss (USS loss) is proposed, which learns a unified threshold to distinguish between positive and negative face pairs;
CodeGraph Neural NetworkPrompt EngineeringGraphBiomedical Data
π― What it does: A general prompt tuning method for pre-trained graph neural networks (Graph Prompt Feature, GPF and its variant GPF-plus) is proposed, which adapts to downstream tasks by adding learnable prompt vectors in the graph node feature space without modifying the model itself.
π― What it does: A cross-modal 3D object detection framework UPIDet is proposed, which utilizes image information to enhance point cloud detection performance.
Unleashing the Full Potential of Product Quantization for Large-Scale Image Retrieval
Yu Liang (Hunan University), Xiaoyu Wang (Hong Kong University of Science and Technology)
CodeRetrievalConvolutional Neural NetworkImage
π― What it does: A product quantization framework FPPQ based on deep learning is proposed, which utilizes a softmax differentiable PQ branch to learn category-level PQ codes, aiming to enhance the performance of large-scale image retrieval.
π― What it does: A framework named Adversarial Invariant Augmentation (AIA) is proposed for graph data augmentation to address the covariate shift problem in graph classification tasks; it enhances environmental features differentially while keeping stable features unchanged, improving the model's generalization performance in unseen environments.
Unleashing the Power of Randomization in Auditing Differentially Private ML
Krishna Pillutla (Google Research), Sewoong Oh (University of Washington)
CodeSafty and PrivacyGaussian SplattingTabular
π― What it does: Audit differential privacy machine learning algorithms and propose adding multiple random 'canaries' to the dataset for multiple statistical tests.
Unlimiformer: Long-Range Transformers with Unlimited Length Input
Amanda Bertsch (Carnegie Mellon University), Matthew R. Gormley (Carnegie Mellon University)
CodeTransformerTextRetrieval-Augmented Generation
π― What it does: The existing pre-trained encoder-decoder Transformer is modified to use k-NN retrieval to focus only on the top k most relevant keys in the cross-attention, enabling the processing of inputs of infinite length.
Unlocking Feature Visualization for Deep Network with MAgnitude Constrained Optimization
Thomas FEL, Thomas Serre (Brown University)
CodeGenerationOptimizationExplainability and InterpretabilityConvolutional Neural NetworkTransformerImage
π― What it does: A new feature visualization method called MACO is proposed, which utilizes amplitude constraints of the Fourier spectrum and phase optimization to generate natural image explanations.
π― What it does: This paper proposes the UNSSOR method, which utilizes overdetermined mixed signals (more microphones than speakers) to train a neural network under unsupervised conditions by constructing a mixed constraint loss, achieving speech separation; after training, it can be used for single-microphone (underdetermined) separation.
π― What it does: This paper proposes an unsupervised learning to reject framework REJEX, which sets a constant threshold in anomaly detection using the EXCEED stability measure, thereby enabling the rejection of predictions for highly uncertain samples.
π― What it does: A novel unsupervised optical flow estimation method for event cameras, USFlow, is proposed, which utilizes multi-layer dilated convolutions to achieve dynamic temporal representation, extracts features from multi-scale temporal windows, and constructs a self-supervised loss through optical flow-guided light intensity fusion.
π― What it does: Proposes an unsupervised multi-energy neural representation (Polyner) that directly recovers metal artifact-free images from CT projections affected by metal;
π― What it does: This paper proposes an unsupervised video domain adaptation framework based on a separable variational autoencoder (TranSVAE), specifically for action recognition tasks.
π― What it does: A method called UP-NeRF is proposed, which jointly optimizes camera poses and neural radiance fields to achieve high-quality view synthesis without camera pose priors and in the presence of inconsistent lighting and transient occlusions.
Juyeon Heo (University of Cambridge), Adrian Weller (Alan Turing Institute)
CodeExplainability and InterpretabilityImage
π― What it does: This paper rephrases the framework of Model Explanation (MLX) as a robustness problem, utilizing human-provided explanation masks to define the perturbation space, thereby training models that are robust to irrelevant features without requiring strong parameter smoothing.
Devon R. Graham (University of British Columbia), Tim Roughgarden (Columbia University)
CodeOptimizationTabular
π― What it does: This paper proposes a utility function-based algorithm configuration method called Utilitarian Procrastination, addressing the shortcomings of traditional expected runtime minimization and providing theoretical guarantees.
π― What it does: A minimalist convolutional network called VanillaNet is proposed, which removes depth, shortcut connections, and self-attention, using only a minimalist module of 1Γ1 convolution + BN + activation. A deep training strategy and a series of activation functions are employed during training to enhance non-linearity.
π― What it does: This paper proposes an online unbiased, low-variance evolutionary strategy gradient estimation method called NRES, which addresses the high variance and slow convergence issues of traditional online ES.
π― What it does: The VAG-CO method is proposed, which solves combinatorial optimization problems using a self-regressive variational adaptive graph model.
Variational Inference with Gaussian Score Matching
Chirag Modi (Flatiron Institute), Lawrence K. Saul (Flatiron Institute)
CodeOptimizationScore-based ModelTabular
π― What it does: A variational inference method based on score matching (GSM-VI) is proposed, which achieves closed-form iterative updates of Gaussian family variational distributions by minimizing the KL distance at each step and enforcing the matching of the posterior and the gradient of the variational distribution.
π― What it does: This paper proposes Variational Weighted Kernel Density Estimation (VWKDE), which reduces bias in density ratio estimation by applying position-dependent weight functions to the kernel, and applies this method to posterior probability and KL divergence interpolation estimation.
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Sihan Chen (University of Chinese Academy of Sciences), Jing Liu (University of Chinese Academy of Sciences)
CodeGenerationRetrievalTransformerLarge Language ModelContrastive LearningVideoTextMultimodalityAudio
π― What it does: A multimodal video subtitle dataset VAST-27M with a scale of 27M has been constructed, and a foundational model VAST capable of perceiving visual, audio, subtitle, and text modalities has been trained.
π― What it does: This paper proposes the VIP-Token Focused Compression (VCC) scheme, which compresses and decompresses long sequences between Transformer layers, significantly reducing the computational and memory requirements for sequence length.
VeriX: Towards Verified Explainability of Deep Neural Networks
Min Wu (Stanford University), Clark Barrett (Stanford University)
CodeAutonomous DrivingExplainability and InterpretabilityImage
π― What it does: This paper proposes VERIX, which generates optimal robust explanations and counterfactuals on decision boundaries based on constraint solving and feature sensitivity ranking.
π― What it does: By combining text prompts with a two-dimensional diffusion model (Instruct-Pix2Pix) and utilizing NeRF depth information to edit key views, a 3D editing framework called ViCA-NeRF is proposed, which propagates edits to the panorama through projection and mixing methods while maintaining viewpoint consistency.
Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution
Ying Wang (New York University), Andrew Gordon Wilson (New York University)
CodeExplainability and InterpretabilityTransformerContrastive LearningImageTextMultimodality
π― What it does: Proposes an explainability method based on multimodal information bottleneck (M2IB) for generating attribution maps for image-text pairs;
Haotian Liu (University of Wisconsin Madison), Yong Jae Lee (Columbia University)
CodeRecognitionGenerationRetrievalTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodalityBenchmark
π― What it does: This study proposes LLaVA (Large Language and Vision Assistant), a multimodal model capable of visual question answering and chatting, by connecting the CLIP visual encoder with the Vicuna LLM through linear projection and performing end-to-end instruction tuning on visual instruction-following data generated by GPT-4.
VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models
Ziyi Yin (Pennsylvania State University), Fenglong Ma (Pennsylvania State University)
CodeRecognitionAdversarial AttackTransformerVision Language ModelImageTextMultimodality
π― What it does: This paper studies adversarial attacks on downstream fine-tuning models using only publicly available pre-trained vision-language models under black-box conditions, and proposes the VLATTACK method.
Alessandro Conti (University of Trento), Elisa Ricci (Fondazione Bruno Kessler)
CodeClassificationRetrievalTransformerVision Language ModelImageMultimodality
π― What it does: Proposes the Vocabulary-free Image Classification (VIC) task and develops the CaSED method to achieve vocabulary-free image classification.
VoxDet: Voxel Learning for Novel Instance Detection
Bowen Li (Carnegie Mellon University), Sebastian Scherer (Carnegie Mellon University)
CodeObject DetectionPoint Cloud
π― What it does: This paper proposes a voxel learning-based 3D geometric perception framework called VoxDet, designed for detecting unseen instances under multi-view template conditions.
VPGTrans: Transfer Visual Prompt Generator across LLMs
Ao Zhang (National University of Singapore), Tat-Seng Chua (National University of Singapore)
CodeTransformerLarge Language ModelPrompt EngineeringVision Language ModelImageTextMultimodality
π― What it does: A two-stage visual prompt generator (VPG) transfer framework called VPGTrans is proposed, which can efficiently transfer VPG between different LLM sizes and types, significantly reducing training costs.
π― What it does: A 3D generation method based on voxel-point progressive generation (VPP) is proposed, which can efficiently generate multi-category, high-resolution point clouds and supports downstream tasks such as editing and completion.
π― What it does: A post-hoc OOD detection method called VRA (Variational Rectified Activation) is proposed, which designs an activation function through variational methods to suppress abnormally low/high activations and amplify intermediate activations.
Vulnerabilities in Video Quality Assessment Models: The Challenge of Adversarial Attacks
Aoxiang Zhang, Yuan-Gen Wang (Guangzhou University)
CodeAdversarial AttackVideo
π― What it does: A robustness evaluation of no-reference video quality assessment (NR-VQA) models is conducted, systematically studying their vulnerability to adversarial attacks for the first time, and proposing a white-box attack method based on Score-Reversed Boundary Loss and a black-box attack method based on patch random search.
π― What it does: This paper proposes a framework based on Wasserstein Distributionally Robust Optimization (W-DRO) to unify the study of adversarial attacks and robust training of neural networks, and extends it to distributed threat models.
π― What it does: Through weak supervision, knowledge distillation from CLIP and DINO to NeRF is achieved for open vocabulary segmentation of 3D scenes.
π― What it does: Utilize SAM to generate pseudo-masks under sparse annotations (points/strokes) and train a weakly supervised hidden object segmentation model through multi-scale feature grouping (MFG).
π― What it does: This study evaluates the vulnerable 'non-learnable' datasets, revealing their impact on deep learning models and proposing new cracking methods.
What Do Deep Saliency Models Learn about Visual Attention?
Shi Chen (University of Minnesota), Qi Zhao (University of Minnesota)
CodeExplainability and InterpretabilityConvolutional Neural NetworkImage
π― What it does: An interpretable framework has been developed to decompose the implicit features of deep saliency models into trainable bases, and to quantitatively measure the positive and negative contributions of each semantic to saliency prediction through probability mapping and semantic alignment, thereby enabling a systematic analysis of model behavior.
What functions can Graph Neural Networks compute on random graphs? The role of Positional Encoding
Nicolas Keriven (Centre National de la Recherche Scientifique), Samuel Vaiter (Centre National de la Recherche Scientifique)
CodeGraph Neural NetworkGraph
π― What it does: This study investigates the expressible function space of graph neural networks (GNNs) on large random graphs for node tasks and analyzes the impact of positional encoding on expressiveness.
π― What it does: This study investigates how to enhance context learning effects in visual large models by automatically retrieving good examples, proposing both unsupervised and supervised retrieval frameworks.
π― What it does: This paper proposes an offline inverse reinforcement learning framework based on maximum likelihood estimation, which jointly learns the reward function and optimal policy using a world model and a conservative policy.
π― What it does: This paper systematically analyzes the memory and credit assignment capabilities of Transformers in reinforcement learning, proposes formal definitions for memory length and credit assignment length, and designs configurable toy tasks (such as Passive/Active T-Maze) to decouple these two capabilities. It then evaluates Transformer-based and LSTM-based RL algorithms in various POMDP environments (including custom T-Maze, Passive/Active Visual Match, Key-to-Door, PyBullet Benchmarks, etc.) to explore their performance under different memory/credit assignment lengths.
π― What it does: A visual prompt tuning framework for source-free domain adaptive semantic segmentation, Uni-UVPT, is proposed, which adapts a frozen Transformer model using only a small number of learnable parameters.
Where Did I Come From? Origin Attribution of AI-Generated Images
Zhenting Wang (Rutgers University), Shiqing Ma (Sony AI)
CodeGenerationData SynthesisOptimizationImage
π― What it does: A model-agnostic AI image source attribution method has been developed without modifications, utilizing reverse engineering to reconstruct inputs and distinguishing whether an image was generated by a specific model through reconstruction loss;
π― What it does: This study investigates the phenomenon of perceptual alignment of robust model gradients, proposing discrete manifold robustness to explain why gradients lie on the signal manifold, and introduces signal-interference decomposition.
Why think step by step? Reasoning emerges from the locality of experience
Ben Prystawski (Stanford University), Noah Goodman
CodeTransformerLarge Language ModelChain-of-Thought
π― What it does: The effectiveness of chain reasoning in language models was studied, and it was demonstrated that the local structure of training data allows intermediate reasoning steps to reduce bias.
π― What it does: A coverage-based window distribution shift detection method (Coverage-Based Detection, CBD) is proposed, which monitors whether the input stream undergoes distribution changes by calculating the lower bound of coverage based on the confidence (such as entropy) of the pre-trained model.
π― What it does: A population-based reinforcement learning method called Poppy is proposed to solve NP-hard combinatorial optimization problems by training a set of complementary strategies to improve the quality of solutions during inference.
Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model
Zirui Liu (Rice University), Xia Hu (Rice University)
CodeTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: A new unbiased sampling matrix multiplication estimation method WTA-CRS is proposed to significantly reduce activation storage during Transformer training while maintaining gradient unbiasedness.
WITRAN: Water-wave Information Transmission and Recurrent Acceleration Network for Long-range Time Series Forecasting
Yuxin Jia (Beijing Jiaotong University), Huaiyu Wan (Beijing Jiaotong University)
CodeRecurrent Neural NetworkTransformerTime Series
π― What it does: A new time series long-short term information transmission framework called WIT (Water-wave Information Transmission) is proposed, and based on this, a Recursive Accelerated Network (RAN) is constructed to efficiently capture global/local associations and long/short cycle repetitive semantics for long/ultra-long sequence prediction.
Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis
Alexander Meulemans (ETH ZΓΌrich), Greg Wayne
CodeReinforcement LearningContrastive Learning
π― What it does: This paper proposes a model-based long-term credit allocation method called COCOA (Counterfactual Contribution Analysis), which achieves more accurate policy gradient estimation by inferring the contribution of actions to future rewards.
π― What it does: The YOCO (You Only Condense Once) method is proposed, which allows for flexible scaling (trimming) of the synthesized dataset based on different computational resource requirements after a single dataset compression, without the need for an additional compression process.
π― What it does: A zero-shot anomaly detection method based on batch normalization and meta-training, ACR, is proposed, which can complete anomaly detection under new distributions without retraining.
CodeMeta LearningDrug DiscoveryTabularBiomedical DataElectronic Health Records
π― What it does: A zero-shot causal learning framework, CaML, is proposed, which can predict the causal effects of individuals on new interventions without any historical intervention data.
Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models
Lin Li (Zhejiang University), Long Chen (The Hong Kong University of Science and Technology)
CodeRecognitionObject DetectionComputational EfficiencyTransformerLarge Language ModelVision Language ModelImageChain-of-Thought
π― What it does: Proposes the RECODE method, which utilizes large language models to generate composite descriptive prompts (subject, object, space) to improve zero-shot visual relationship detection in CLIP;
ZipLM: Inference-Aware Structured Pruning of Language Models
Eldar Kurtic (IST Austria), Dan Alistarh (Neural Magic)
CodeCompressionComputational EfficiencyKnowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This paper proposes ZipLMβa reasoning-aware structured pruning method that can batch-generate various compressed BERT/GPT models under given reasoning environments and speed-up targets.
π― What it does: This paper proposes a visual tracking method called ZoomTrack based on non-uniform scaling, which enhances tracking accuracy while maintaining a small input size and high speed.