π― What it does: A verifiable robust conformal prediction framework (VRCP) based on neural network verification is proposed, which can maintain coverage guarantees of the prediction set under adversarial perturbations.
Sahil Bhatia (University of California Berkeley), Alvin Cheung (University of California Berkeley)
CodeAI Code AssistantTransformerLarge Language ModelPrompt EngineeringText
π― What it does: A verification enhancement framework based on large language models, LLMLIFT, has been developed, which can automatically convert source language programs into target DSLs and generate formal equivalence proofs.
Verified Safe Reinforcement Learning for Neural Network Dynamic Models
Junlin Wu (Washington University in St. Louis), Yevgeniy Vorobeychik (Washington University in St. Louis)
CodeAutonomous DrivingSafty and PrivacyReinforcement LearningTime Series
π― What it does: A safety reinforcement learning framework based on finite-step reachability verification is proposed, capable of training control policies that can operate within specified safety boundaries on neural network dynamic models.
VideoTetris: Towards Compositional Text-to-Video Generation
Ye Tian (Peking University), Bin CUI
CodeGenerationData SynthesisLarge Language ModelDiffusion modelOptical FlowVideoText
π― What it does: The VideoTetris framework is proposed to achieve text-based composable video generation, supporting spatial positioning and temporal changes of multiple objects.
Virtual Scanning: Unsupervised Non-line-of-sight Imaging from Irregularly Undersampled Transients
Xingyu Cui (Tianjin University), Jingyu Yang (Tianjin University)
CodeRestorationImageVideo
π― What it does: An unsupervised framework is proposed that utilizes virtual scanning and a SURE-based denoiser to reconstruct non-line-of-sight scenes from irregularly under-sampled time-delay signals.
VISA: Variational Inference with Sequential Sample-Average Approximations
Heiko Zimmermann (University of Amsterdam), Jan-Willem van de Meent (University of Amsterdam)
CodeOptimizationTabular
π― What it does: A variational inference method named VISA is proposed, which utilizes Sequential Sample Average Approximation (SAA) for forward KL inference in non-differentiable and computationally expensive models, significantly reducing the number of model evaluations.
π― What it does: A visual model pre-training method utilizing Layer Compression Learning (LCL) is proposed, which can train the visual encoder from scratch on mixed image-text sequences.
CodeClassificationAnomaly DetectionTransformerPrompt EngineeringVision Language ModelImageMultimodality
π― What it does: The DEFT framework is proposed, which detects noisy labels by learning positive and negative dual text prompts on a pre-trained vision-language model, and uses the filtered clean samples to complete the final model fine-tuning.
Vision-Language Navigation with Energy-Based Policy
Rui Liu (Zhejiang University), Yi Yang (Zhejiang University)
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackTransformerReinforcement LearningVision-Language-Action ModelMultimodality
π― What it does: This paper proposes an energy-based navigation strategy (ENP) that guides the decision-making of visual-language navigation (VLN) agents by learning the state-action joint distribution from expert demonstrations;
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
Jiannan Wu (OpenGVLab), Jifeng Dai (Tsinghua University)
CodeRecognitionSegmentationGenerationPose EstimationTransformerLarge Language ModelVision Language ModelDiffusion modelImageTextMultimodality
π― What it does: Proposes VisionLLM v2, constructing an end-to-end general multimodal large model that unifies visual perception, understanding, and generation, supporting hundreds of visual-visual language tasks;
Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model
Haogeng Liu (Institute of Automation, Chinese Academy of Sciences), Hongxia Yang (Institute of Automation, Chinese Academy of Sciences)
CodeComputational EfficiencyRepresentation LearningTransformerLarge Language ModelVision Language ModelImageTextMultimodality
π― What it does: The concept of visual anchors is proposed, and based on this, the Anchor Former (AcFormer) is designed as a visual-language connector for multimodal large language models, capable of significantly reducing the number of visual tokens and computational costs while maintaining or even improving accuracy.
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
Keyu Tian (Peking University), Liwei Wang (Peking University)
CodeGenerationTransformerAuto EncoderImage
π― What it does: A new visual autoregressive framework VAR is proposed, which achieves efficient image generation through multi-scale 'next-scale' prediction.
π― What it does: Construct a conceptual knowledge graph of visual data, diagnose the graph to discover concept co-occurrence biases, and achieve data debiasing by generating images of missing concept combinations, ultimately enhancing the generalization ability of downstream classifiers.
π― What it does: A zero-shot visual decoding and reconstruction framework based on EEG has been developed, which includes an ATM EEG encoder and a two-stage EEG-guided diffusion generation method.
Runjia Zeng (Rochester Institute of Technology), Dongfang Liu (Rochester Institute of Technology)
CodeClassificationDomain AdaptationExplainability and InterpretabilityTransformerPrompt EngineeringImage
π― What it does: This paper proposes a Visual Fourier Prompt Tuning (VFPT) method that embeds 2D Fast Fourier Transform (FFT) into visual prompts, allowing for efficient transfer across different datasets with only a small number of parameters fine-tuned.
Visual Perception by Large Language Modelβs Weights
Feipeng Ma (University of Science and Technology of China), Xiaoyan Sun (University of Science and Technology of China)
CodeRecognitionComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodality
π― What it does: VLoRA is proposed, which achieves multimodal perception by converting visual features into low-rank model weights (perceptual weights) and integrating them with LLM weights without introducing visual tokens.
π― What it does: To address the catastrophic forgetting problem in visual prompt tuning within ViT, an orthogonal projection of the prompt gradient in the null space is proposed to achieve interference-free updates for learned tasks.
VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance
Divyansh Srivastava (University of California San Diego), Tsui-Wei Weng (University of California San Diego)
CodeClassificationObject DetectionConvolutional Neural NetworkLarge Language ModelImage
π― What it does: A Vision-Language Guided Concept Bottleneck Model (VLG-CBM) is constructed, which generates candidate concepts through LLM and uses open-domain location-based object detectors (such as Grounding-DINO) to produce visually recognizable and locatable concept annotations, thereby achieving end-to-end visual and language guidance during the training of CBM.
Voila-A: Aligning Vision-Language Models with User's Gaze Attention
Kun Yan (Beihang University), Shuai Ma (Beihang University)
CodeRetrievalExplainability and InterpretabilityTransformerLarge Language ModelVision Language ModelImageTextMultimodality
π― What it does: Voila-A is proposed, which aligns visual language models using user gaze information to enhance their interpretability and effectiveness in real-world scenarios.
π― What it does: Utilizing a VQ-VAE-like generative model to convert BEV maps into discrete tokens, constructing a codebook as a prior; aligning camera perspective (PV) features to the discrete BEV space through a dedicated token decoder, and subsequently generating high-quality BEV semantic maps using the codebook, thereby addressing challenges such as occlusion and low resolution.
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models
Jinghan Jia (Michigan State University), Sijia Liu (IBM Research)
CodeOptimizationSafty and PrivacyTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This paper proposes a weight attribution-based gradual forgetting framework for large language models (WAGLE), which enhances the forgetting effect of large language models while maintaining their original functionality by identifying the model weights that have the most significant impact on the forgetting task.
Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning
Takuo Matsubara (University of Edinburgh)
CodeClassificationOptimizationTabular
π― What it does: This paper proposes a gradient boosting framework based on Wasserstein gradient flow (WGBoost), which can predict the probability distribution of input (a set of particles) and achieve distribution prediction of tree models in evidential learning.
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Wenkai Yang (Renmin University of China), Xu Sun (Peking University)
CodeAdversarial AttackRobotic IntelligenceTransformerLarge Language ModelSupervised Fine-TuningAgentic AIText
π― What it does: This paper studies and experiments with backdoor attacks on agents driven by large language models, proposing a general attack framework divided into three types: Query-Attack, Observation-Attack, and Thought-Attack.
CodeTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: The study investigates whether watermark text used for LLM fine-tuning leaves detectable watermark traces (i.e., 'radioactivity') in the model.
CodeClassificationDomain AdaptationTransformerPrompt EngineeringVision Language ModelImage
π― What it does: This paper proposes the WATT method, which achieves adaptation during CLIP testing through multi-template text prompts and weight averaging, improving zero-shot classification performance in cross-domain tasks.
π― What it does: A backdoor attack method called WaveAttack based on Discrete Wavelet Transform (DWT) is proposed, which generates covert triggers using high-frequency components and employs asymmetric frequency obfuscation with different amplitudes during training and inference phases to balance attack effectiveness and sample quality.
Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles
Qi Chen (Australian Institute for Machine Learning, University of Adelaide), Qi Wu (Australian Institute for Machine Learning, University of Adelaide)
CodeTransformerLarge Language ModelPrompt EngineeringTextBenchmark
π― What it does: The SPLAT benchmark is proposed to evaluate and stimulate the lateral thinking ability of large language models through situational puzzles.
CodeGenerationOptimizationTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningText
π― What it does: Without training large language models, alignment with large language models is achieved through greedy search of the log probability differences between small fine-tuned models and un-tuned models during inference.
Weight Diffusion for Future: Learn to Generalize in Non-Stationary Environments
Mixue Xie (Beijing Institute of Technology), Chengwei Zhu (Tencent)
CodeDomain AdaptationDiffusion modelTabular
π― What it does: In the domain incremental setting, the evolution of domain generalization is studied, and the Weight Diffusion (W-Diff) framework is proposed. It utilizes a conditional diffusion model to learn the evolution patterns of classifier parameters and generates customized classifiers for future domains; simultaneously, it learns a domain-shared feature encoder to suppress overfitting; during inference, robustness is enhanced through a weight ensemble.
π― What it does: This paper proposes a method to expand feature representation by applying random perturbations (WeiPer) to the weights of the final fully connected layer in the network, and implements post-hoc OOD detection using MSP or KL divergence scores.
Boxuan Zhang (Wuhan University), Bo Han (Hong Kong Baptist University)
CodeDomain AdaptationAnomaly DetectionConvolutional Neural NetworkTransformerVision Language ModelImage
π― What it does: The CoVer framework is proposed, which expands the input dimensions using common image corruptions (90 types) and averages the multi-dimensional confidence to enhance the detection capability for OOD samples.
π― What it does: The paper studies and dissects graph homophily into three dimensions: labels, structure, and features, proposing a new composite metric called Tri-Hom.
What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Samyak Jain (Five AI), Puneet K. Dokania (Five AI)
CodeData SynthesisSafty and PrivacyAdversarial AttackTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: A synthetic data generation framework is proposed for systematically studying the mechanisms of safe fine-tuning and its adversarial attacks;
π― What it does: Conduct systematic controlled experiments on the robustness of CLIP with long-tail pre-training data, exploring the impact of language supervision, pretext tasks, data distribution, scale, and open-world concepts, and transferring findings to supervised and self-supervised learning.
π― What it does: This paper conducts a systematic study on the ease of unlearning in machine learning models, analyzing the mixing degree of the forget set and the retain set in the embedding space, as well as the memorization degree of the forget set on different algorithms. It proposes a refined RUM framework to enhance unlearning performance.
What Matters in Graph Class Incremental Learning? An Information Preservation Perspective
Jialu Li (Tianjin University), Qinghua Hu (Tianjin University)
CodeGraph Neural NetworkGraph
π― What it does: This paper proposes a Graph Space Information Preservation framework (GSIP) for graph-based incremental learning, which aligns the low-frequency local-global information and high-frequency neighbor similarity from the outputs of the old model to reduce semantic and structural shifts, thereby alleviating catastrophic forgetting.
What matters when building vision-language models?
Hugo Laurençon (Hugging Face), Victor Sanh (Hugging Face)
CodeRecognitionGenerationRetrievalTransformerSupervised Fine-TuningVision Language ModelImageTextMultimodality
π― What it does: This paper investigates and systematically evaluates the core design choices of Visual Language Models (VLM) and develops an 8B parameter Idefics2 VLM based on experimental results.
CodeTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: Analyzed the impact of RoPE weight vector angles on attention, revealing that non-orthogonal weights are sensitive to basic grammatical information, while orthogonal weights are sensitive to advanced semantic information; based on this, proposed the Angle Weight Masking (AWM) method, which updates only the orthogonal weight pairs during fine-tuning, significantly reducing the number of trainable parameters while maintaining or improving performance.
Miguel Lazaro-Gredilla, Dileep George (Google Deepmind)
CodeOptimizationReinforcement Learning from Human FeedbackReinforcement LearningTabular
π― What it does: This paper rephrases the planning task as a probabilistic inference problem, deriving a variational formula for planning, and proposes Value Belief Propagation (VBP) and scalable VI-LP approximation methods for sparse state factor MDPs.
π― What it does: This study investigates the transferability of pre-trained deep networks under different conditions and their generalization ability to OOD (out-of-distribution) data. It systematically evaluates the impact of variables such as network architecture, training data diversity, image resolution, and data augmentation on the 'tunnel effect' and OOD performance.
π― What it does: This paper studies the generality and statistical accuracy issues of learning dynamical systems from time series data, proposing a novel generalization definition based on ergodic theory, and theoretically and experimentally verifying that Jacobian matching training can enhance the statistical accuracy of the model.
π― What it does: This paper proposes a unified, label-free embedding model evaluation method that constructs comparable metrics using the concepts of sufficiency, informativeness, and defects from information theory.
π― What it does: This paper evaluates the performance of the multicalibration post-processing method across various models and datasets through large-scale experiments and provides practical guidance.
When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search
Xuan Chen (Purdue University), Xiangyu Zhang (Purdue University)
CodeAdversarial AttackTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
π― What it does: A black-box jailbreak attack method based on deep reinforcement learning, called RLbreaker, is proposed for automatically generating and optimizing Jailbreaking Prompts.
π― What it does: This paper proposes a Time-Adaptive Control and Perception Framework (TACOS), which transforms continuous-time RL problems (including interaction costs or interaction count limitations) into equivalent discrete-time MDPs, thereby utilizing existing RL algorithms to achieve fewer system interactions.
π― What it does: In a controlled experimental setup, the impact of different initial learning rates on the final solutions of neural networks was systematically studied. It was proposed that a learning rate within the 'sub-interval 2A', slightly above the convergence threshold, can locate low valleys of high-quality solutions during the pre-training phase, and achieve optimal generalization performance after fine-tuning or weight averaging. Additionally, this learning rate range encourages the network to learn sparser and more targeted features. The experiments were extended to practical training settings, validating the universality of the conclusions.
Whoβs Gaming the System? A Causally-Motivated Approach for Detecting Strategic Adaptation
Trenton Chang (University of Michigan), Jenna Wiens (University of Michigan)
CodeAnomaly DetectionOptimizationTabular
π― What it does: A framework based on causal inference is proposed to rank strategic game behaviors (such as upcoding) in a multi-agent environment and identify the agents most likely to cheat.
π― What it does: This paper studies the hierarchical mismatch problem caused by full model updates in federated learning and proposes the FedPart method, which updates only part of the network layers in each round and designs a layer selection strategy, significantly improving model convergence speed and accuracy.
π― What it does: This study investigates the retention of community structure by the metric backbone (the union of all shortest paths) in the weighted random block model, and based on this, proposes practical methods for graph sparsification and construction using the metric backbone.
π― What it does: This paper conducts a fine-grained analysis of the Hessian block spectrum of Transformer and CNN, proposing that the block heterogeneity in Transformers is the reason for the performance gap between SGD and Adam.
Wide Two-Layer Networks can Learn from Adversarial Perturbations
Soichiro Kumano (University of Tokyo), Toshihiko Yamasaki (University of Tokyo)
CodeAdversarial AttackTabular
π― What it does: This paper theoretically proves that in sufficiently wide two-layer networks, perturbation learning can successfully learn discriminative features from mislabelled adversarial samples that are the same as those from the original clean samples, and it demonstrates that adversarial perturbations actually encompass the class features of the entire training set.
π― What it does: The Wild-GS method is proposed, adapting 3D Gaussian Splatting to unconstrained photo collections for real-time novel view synthesis.
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
Liwei Jiang (University of Washington), Nouha Dziri (Allen Institute for Artificial Intelligence)
CodeSafty and PrivacyAdversarial AttackTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: The WILDTEAMING framework is proposed to automatically mine jailbreak techniques from real user interactions with chatbots and combine them to generate diverse adversarial attacks, thereby constructing a large-scale secure training dataset called WILDJAILBREAK.
Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
Avelina Asada Hadji-Kyriacou (University of St Andrews), Ognjen Arandjelovic (University of St Andrews)
CodeTransformerLarge Language ModelReinforcement LearningText
π― What it does: This paper proposes Direct Preference Heads (DPH), which align the language model by scoring candidate outputs through an additional reward head during inference, avoiding direct modification of the generation distribution.
Maximilian Beck (Johannes Kepler University Linz), Sepp Hochreiter (Johannes Kepler University Linz)
CodeRecurrent Neural NetworkText
π― What it does: An extended LSTM model called xLSTM is proposed, which combines exponential gating and matrix memory to address the storage, correction, and parallelism limitations of traditional LSTMs.
xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Julius Hense (Berlin Institute for the Foundations of Learning and Data), Klaus Robert Muller
CodeExplainability and InterpretabilityBiomedical Data
π― What it does: This paper proposes the xMIL framework and implements xMIL-LRP to generate interpretable heatmaps for the predictions of multi-instance learning (MIL) models in digital pathology.
CodeRetrievalCompressionComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningTextMultimodalityRetrieval-Augmented Generation
π― What it does: This paper proposes an extreme context compression method called xRAG, which utilizes retrieval embeddings as multimodal features to compress an entire document into a single token input for a language model.
π― What it does: A real-time end-to-end YOLOv10 object detection framework has been developed, eliminating NMS and comprehensively optimizing the model structure.
π― What it does: The YOLA framework and Illumination-Invariant Module (IIM) are proposed to enhance object detection performance under low-light conditions by learning illumination-invariant features based on the Lambertian model.
π― What it does: This paper proposes Generalized Contrastive Alignment (GCA), which reinterprets contrastive learning as a distribution alignment problem and constructs a customizable alignment loss using optimal transport, allowing for fine-grained control of positive and negative sample relationships in self-supervised learning.
π― What it does: A self-supervised Contrastive Diffusion Loss (CDL) is proposed, which enhances the model's denoising performance in out-of-distribution (OOD) regions by treating the diffusion model as a noise classifier, thereby improving and accelerating the sequential and parallel sampling processes.
π― What it does: A zero-training event-intensity asymmetric stereo matching framework called ZEST is proposed, which aligns visual cues from events and frames, allowing image domain pre-trained stereo matching and monocular depth estimation models to be directly applied to event-frame scenes.
Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
Jieren Deng (Institute of Automation Chinese Academy of Sciences), Yunkuan Wang (Institute of Automation Chinese Academy of Sciences)
CodeObject DetectionTransformerVision Language ModelImageMultimodality
π― What it does: This paper proposes the Incremental Visual Language Object Detection (IVLOD) task and designs an incremental learning method that maintains zero-shot generalization.
Benjamin Minixhofer (University of Cambridge), Ivan VuliΔ (University of Edinburgh)
CodeTransformerLarge Language ModelSupervised Fine-TuningTextMultimodalityBenchmark
π― What it does: A method is proposed to transfer pre-trained language models to any new tokenizer using zero-shot tokenizer transfer (ZeTT) without training a new model.
π― What it does: This study investigates a zero-shot transfer method for learning neural ODE basis functions through functional encoders, enabling adaptive control systems to quickly identify new dynamics without retraining.
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
Yefei He (Zhejiang University), Bohan Zhuang (Monash University)
CodeGenerationRetrievalCompressionComputational EfficiencyTransformerLarge Language ModelText
π― What it does: This paper proposes ZipCache, which performs accurate and efficient mixed-precision quantization compression for KV cache in large language models, significantly reducing memory usage and inference latency.
Sho Yokoi (Tohoku University / RIKEN), Hidetoshi Shimodaira (Kyoto University / RIKEN)
CodeText
π― What it does: This paper proposes a whitening method (Zipfian whitening) that uses actual word frequency (Zipfian frequency) for expectation calculation in the word vector space, and provides corresponding symmetry evaluation metrics. By applying this method to post-process pre-trained word vectors such as GloVe, Word2Vec, and fastText, it significantly improves the performance of downstream tasks like sentence similarity.
π― What it does: A zero-shot offline panoramic perception framework ZOPP is proposed, utilizing multimodal (multi-view cameras + LiDAR) to automatically generate 3D semantic, instance, detection, and occupancy labels;