Conference on Neural Information Processing Systems Β· 2283 papers
Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
Gleb Rodionov, Dan Alistarh
CodeGenerationOptimizationTransformerLarge Language ModelPrompt EngineeringText
π― What it does: This paper proposes a parallel inference framework called Hogwild! Inference, which allows multiple LLM instances to synchronize in real-time and collaborate spontaneously to solve complex reasoning tasks through shared Key-Value caches.
π― What it does: This paper presents MixGS, a global optimization framework for large-scale scene reconstruction that achieves high-quality novel view synthesis by mixing high-resolution and original 3D Gaussians.
π― What it does: The paper proposes a network called InstaFormer, which can directly predict the occlusion and depth order matrix of all instances in a scene from RGB images in a single forward pass.
HoliTom: Holistic Token Merging for Fast Video Large Language Models
Kele Shao (Zhejiang University), Huan Wang (Westlake University)
CodeCompressionComputational EfficiencyTransformerLarge Language ModelVision Language ModelVideo
π― What it does: A training-independent HoliTom framework is proposed, which significantly compresses the visual tokens of video LLMs through global spatiotemporal segmentation and dual token merging, enhancing inference efficiency.
HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning
Chuhao Zhou (Nanyang Technological University), Jianfei Yang (Nanyang Technological University)
CodeRecognitionTransformerLarge Language ModelVision Language ModelVideoMultimodality
π― What it does: This paper proposes HoloLLM, a model that integrates scarce multimodal perceptions (LiDAR, infrared, millimeter-wave radar, WiFi) into a multimodal large language model to achieve language-based intelligent perception and reasoning.
HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios
Kunyu Peng (Karlsruhe Institute of Technology), Rainer Stiefelhagen (Karlsruhe Institute of Technology)
CodeObject DetectionSegmentationRecurrent Neural NetworkTransformerVision Language ModelDiffusion modelVideoTextMultimodality
π― What it does: Proposed and implemented the Referring Human Action Segmentation (RHAS) task, which can segment the actions of target individuals in multi-person videos based on textual descriptions.
HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
Haoran Li (Carnegie Mellon University), Ruiwen Xu (Xiaohongshu Inc.)
CodeRetrievalTransformerVision Language ModelVideoMultimodality
π― What it does: A hybrid position encoding (HoPE) suitable for long-context visual-language models is proposed to enhance long video understanding and retrieval performance.
π― What it does: In this paper, the authors conduct experiments on complex, long-horizon robotic tasks by constructing a large-scale (up to 1 billion transitions) offline reinforcement learning dataset. They systematically evaluate the scalability of existing offline RL algorithms and find that the 'horizon' is the main reason for the limited scalability of the algorithms. They then propose to enhance scalability by reducing the value and policy horizon (e.g., n-step returns, hierarchical policies) and design a simple scalable offline RL method called SHARSA based on this.
CodeAnomaly DetectionGraph Neural NetworkAuto EncoderGraphTime Series
π― What it does: This paper proposes HoT-VI, a reparameterized variational inference framework that utilizes k-order associations to model instance-level high-order correlations.
How Benchmark Prediction from Fewer Data Misses the Mark
Guanhua Zhang (Max Planck Institute for Intelligent Systems), Moritz Hardt (Max Planck Institute for Intelligent Systems)
CodeBenchmark
π― What it does: This paper conducts large-scale experiments on 11 existing and newly proposed benchmark prediction methods across 19 different benchmarks, systematically evaluating their effectiveness in predicting complete benchmark performance when only a small number of samples are assessed.
How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPs
Samet Demir (Koç University), Zafer Dogan (Koç University)
CodeTransformerText
π― What it does: This study investigates the contextual learning performance of pre-trained Transformers using a nonlinear MLP head under mixed multi-source data, and proves its equivalence to polynomial predictors in a high-dimensional asymptotic framework.
How Different from the Past? Spatio-Temporal Time Series Forecasting with Self-Supervised Deviation Learning
Haotian Gao (University of Tokyo), Renhe Jiang (University of Tokyo)
CodeAnomaly DetectionRecurrent Neural NetworkGraph Neural NetworkContrastive LearningTime Series
π― What it does: This paper proposes a spatiotemporal sequence prediction framework called ST-SSDL based on self-supervised bias learning, using historical averages as anchors and further discretizing the latent space through learnable prototypes, achieving relative distance consistency through contrastive learning and bias loss.
Jiaran Ye (Tsinghua University), Juanzi Li (Tsinghua University)
CodeTransformerLarge Language ModelText
π― What it does: In a controlled symbolic environment, training a Transformer to learn implicit reasoning from scratch reveals a three-stage developmental trajectory from memory to generalization.
How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
Xin Lu (Harbin Institute of Technology), Ting Liu (Harbin Institute of Technology)
CodeTransformerLarge Language ModelText
π― What it does: This paper proposes and validates the impact of sequence modeling architectures on base capabilities in pre-trained language models. It reveals the degradation of base capabilities in stateful sequence architectures (such as Mamba and RWKV) through a limited pre-training + OOV testing framework. It concludes that 'full sequence arbitrary selection' is the key design principle to avoid base capability degradation and designs a minimal Top-1 Element/Chunk Selection architecture for validation.
π― What it does: The paper studies the impact of topological bias on message passing in graph recommendation from the perspective of Dirichlet energy, and proposes a Test-time Simplicial Propagation (TSP) method that uses higher-order Simplicial Complex for message passing during the inference phase to mitigate this bias.
How Memory in Optimization Algorithms Implicitly Modifies the Loss
Matias D. Cattaneo (Princeton University), Boris Shigida (Princeton University)
CodeOptimizationTransformerImageText
π― What it does: A general method is proposed to approximate optimization algorithms with exponentially decaying memory as memoryless algorithms, and the memory effect is explained as an implicit disturbance to the loss function through a memory correction term, thereby analyzing the implicit regularization effect of memory on optimization dynamics.
π― What it does: A direct learning flow-map framework based on self-distillation is proposed, unifying and extending various accelerated sampling methods such as consistency models and progressive distillation, avoiding the need for a pre-trained teacher.
π― What it does: The HPSERec framework is proposed, utilizing hierarchical partitioning, expert networks, knowledge distillation, and the Sinkhorn OT feedback mechanism to model and enhance long-tail users and long-tail items in sequential recommendation.
π― What it does: This paper proposes a query-free high-quality cross-modal adversarial attack method called HQA-VLAttack, which achieves a higher attack success rate against vision-language pre-trained models.
Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection
Cong Zeng, zhiqiang xu
CodeAnomaly DetectionLarge Language ModelContrastive LearningText
π― What it does: Proposes transforming the detection of LLM-generated text from a binary classification problem into an outlier detection problem, considering human text as out-of-distribution samples.
Human-assisted Robotic Policy Refinement via Action Preference Optimization
Wenke Xia (Renmin University of China), Di Hu (Renmin University of China)
CodeOptimizationRobotic IntelligenceReinforcement Learning from Human FeedbackLarge Language ModelReinforcement LearningVision Language ModelVision-Language-Action ModelMultimodalitySequential
π― What it does: Collect interaction trajectories through human-machine collaboration in real environments and simulations, and iteratively refine the Vision-Language-Action (VLA) model using Action Preference Optimization (APO) to enable learning from failure examples and achieve continuous improvement.
Hybrid Latent Reasoning via Reinforcement Learning
Zhenrui Yue (University of Illinois Urbana-Champaign), Dong Wang
CodeOptimizationTransformerLarge Language ModelReinforcement LearningTextChain-of-Thought
π― What it does: A hybrid implicit reasoning framework HRPO based on reinforcement learning is proposed, allowing LLM to use both discrete words and continuous hidden states for reasoning during the generation process.
π― What it does: A robust attribute graph clustering framework RAGC is proposed, which integrates node-level and edge-level embeddings with hybrid collaborative enhancement and adaptive differentiated contrastive sample perception.
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Zhijian Zhuo (Peking University), Jinwen Ma
CodeTransformerMixture of ExpertsText
π― What it does: This paper proposes the HybridNorm structure, which uses QKV normalization in the attention layer of the Transformer and Post-Norm in the feedforward network, balancing the gradient stability of Pre-Norm with the performance advantages of Post-Norm.
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
Ting Sun (University of Illinois Urbana-Champaign), Fan Lai (University of Illinois Urbana-Champaign)
CodeTransformerLarge Language ModelText
π― What it does: The HyGen system is proposed to achieve elastic co-location for online and offline LLM tasks, maintaining the SLO of online requests while significantly improving overall throughput.
Wenyuan Li (Hokkaido University), Miki Haseyama (Hokkaido University)
CodeData SynthesisKnowledge DistillationImage
π― What it does: This paper introduces hyperbolic geometry into dataset distillation, utilizing the negative curvature of hyperbolic space to hierarchically embed the original data, and constructs a smaller synthetic dataset that maintains model performance by matching centroids in hyperbolic space.
Menglin Yang (Hong Kong University of Science and Technology), Rex Ying (Chinese University of Hong Kong)
CodeTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This study investigates the non-Euclidean characteristics of LLM word embeddings and proposes the HypLoRA method for low-rank fine-tuning in hyperbolic space.
HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representation
Haoran Luo (Beijing University of Posts and Telecommunications), Anh Tuan Luu
CodeGenerationRetrievalGraph Neural NetworkLarge Language ModelTextBiomedical DataAgriculture RelatedRetrieval-Augmented Generation
π― What it does: A new hypergraph-based retrieval-augmented generation method, HyperGraphRAG, is proposed to improve knowledge retrieval and generation by representing n-ary relational facts through hyperedges.
HyperMARL: Adaptive Hypernetworks for Multi-Agent RL
Kale-ab Tessera (University of Edinburgh), Stefano V. Albrecht (DeepFlow)
CodeReinforcement LearningAgentic AIBenchmark
π― What it does: This paper proposes HyperMARL, a shared policy framework using agent-conditioned hypernetworks that can adaptively achieve specialization, homogenization, or mixed behaviors in multi-agent reinforcement learning.
HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions
Rafael Bischof (ETH Zurich), Bernd Bickel (ETH Zurich)
CodeTransformerPhysics Related
π― What it does: A multi-physics neural operator HyPINO based on hypernetworks is proposed, which can zero-shot generalize to various two-dimensional linear PDEs without task-specific fine-tuning, and further improve accuracy through iterative residual-driven refinement.
HypoBootstrap: A Bootstrapping Framework for Inductive Reasoning
Si Chen (Beihang University), Richong Zhang (Beihang University)
CodeTransformerLarge Language ModelText
π― What it does: Developed the HypoBootstrap framework for inductive reasoning in large language models, employing a bootstrapping hypothesis generation and bootstrapping confirmation method to reduce model hallucinations.
I2-NeRF: Learning Neural Radiance Fields Under Physically-Grounded Media Interactions
Shuhong Liu (University of Tokyo), Tatsuya Harada (University of Tokyo)
CodeRestorationDepth EstimationNeural Radiance FieldImagePhysics Related
π― What it does: This paper proposes IΒ²-NeRF, which models light propagation in participating media by combining physical laws, and implements unified rendering of objects and media within the NeRF framework, supporting 3D reconstruction and enhancement in underwater, foggy, and low-light scenes.
CodeAnomaly DetectionGraph Neural NetworkAuto EncoderGraphFinance Related
π― What it does: A zero-shot general graph anomaly detection framework IA-GGAD is proposed, which can detect anomalous nodes in graphs from different domains without any fine-tuning of the target graph.
IDOL: Meeting Diverse Distribution Shifts with Prior Physics for Tropical Cyclone Multi-Task Estimation
HantingYan, Cong Bai (Zhejiang University of Technology)
CodeGraph Neural NetworkTransformerTime SeriesPhysics Related
π― What it does: Proposes the IDOL framework, which utilizes prior physical knowledge to impose identity distribution constraints for multi-task estimation of tropical cyclones.
IF-Guide: Influence Function-Guided Detoxification of LLMs
Zachary Coalson (Oregon State University), Sanghyun Hong (Anthropic)
CodeOptimizationComputational EfficiencyTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This paper proposes a method that uses Influence Functions to actively identify and suppress content in training data that leads large language models (LLMs) to exhibit toxic behavior, thereby reducing toxicity during the training phase.
IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation
Zijie Lin (National University of Singapore), Tat-Seng Chua (National University of Singapore)
CodeRecommendation SystemTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This paper measures the decisiveness of each token when LLM generates recommendations using Information Gain (IG) and proposes the IGD (IG-based Decisiveness-aware Token Handling) strategy, which re-weights low decisiveness tokens during the fine-tuning and inference stages to correct the token bias of existing models and improve recommendation effectiveness.
Image Stitching in Adverse Condition: A Bidirectional-Consistency Learning Framework and Benchmark
Zengxi Zhang (University of Tokyo), Jinyuan Liu (Dalian University of Technology)
CodeRestorationGaussian SplattingImageBenchmark
π― What it does: This study investigates the problem of image stitching in harsh environments such as low light, haze, and underwater, proposing a bidirectional consistency learning framework and a motion-tolerant seamless fusion network.
π― What it does: This work proposes a generalizable confidence mask method applicable to any black-box generative image super-resolution model, providing reliable and intuitive uncertainty estimates for generated images using conformal prediction techniques.
Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
Weixing Wang (Hasso Plattner Institute University of Potsdam), Haojin Yang (Hasso Plattner Institute University of Potsdam)
CodeRecognitionGenerationGraph Neural NetworkVision Language ModelContrastive LearningImageMultimodality
π― What it does: This paper proposes to identify and suppress visual priors that may lead to hallucinations by clustering the co-occurrence relationships of image tokens in discrete image tokenizers, thereby reducing the hallucination phenomenon in large visual language models.
π― What it does: This paper proposes a framework based on targeted feature suppression to quantify the dependence of CNNs on shape, texture, and color features, and re-evaluates the texture bias hypothesis of ImageNet pre-trained CNNs through this framework.
π― What it does: This paper studies a distributed robust optimization-based autoencoder DRAG, which learns a latent space representation that covers the complete state space from pixel inputs in online reinforcement learning, thereby enhancing the exploration and control performance of goal-oriented RL.
Imitation Beyond Expectation Using Pluralistic Stochastic Dominance
Ali Farajzadeh (University of Illinois Chicago), Brian D Ziebart
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningTabular
π― What it does: A multi-modal imitation learning method is proposedβMulti-dimensional Stochastic Dominance (PSD), which ensures that the reward distribution of the imitation policy is probabilistically superior to the demonstration distribution under all possible convex combinations of reward functions.
π― What it does: This study investigates the impact of dataset attributes (number of samples per class, number of classes) on the susceptibility of membership inference attacks (MIA) in deep transfer learning. It theoretically derives and experimentally verifies that the susceptibility to MIA decreases according to a power law as the number of samples per class increases.
Impact of Layer Norm on Memorization and Generalization in Transformers
Rishi Singhal (North Carolina State University), Jung-Eun Kim (North Carolina State University)
CodeClassificationTransformerImageText
π― What it does: This study investigates the impact of LayerNorm (LN) on memorization and learning in Pre-LN and Post-LN Transformers, explores the effect of removing LN parameters on model performance, and explains the different roles of LN in the two architectures through gradient analysis.
π― What it does: An Implicit Transferability Modeling (ITM) framework is proposed, which uses implicit variational approximation methods to quickly estimate the transfer performance of visual foundation models.
π― What it does: A closed-loop control-based variant of Bilinear RNN called Comba is proposed and implemented, combining state feedback and output correction, using scalar + low-rank (SPLR) state transformation;
Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
Yifan Sun (University of Illinois Urbana-Champaign), Huan Zhang (University of Illinois Urbana-Champaign)
CodeTransformerLarge Language ModelReinforcement LearningText
π― What it does: Proposed two techniques, Difficulty-targeted Online Data Selection (DOTS) and Rollout Replay (RR), which significantly improve the data efficiency of LLM reinforcement learning fine-tuning.
π― What it does: This paper proposes a general three-module (sampler, corrector, noise generator) canonical form that unifies existing diffusion inverse problem algorithms, and based on this framework, designs a learnable linear extrapolation method (LLE) to enhance the performance of inverse problem solving within a limited number of steps.
Improving Generalization of Neural Combinatorial Optimization for Vehicle Routing Problems via Test-Time Projection Learning
Yuanyao Chen (Southern University of Science and Technology), Zhenkun Wang (Southern University of Science and Technology)
CodeOptimizationTransformerLarge Language ModelGraph
π― What it does: This paper proposes a reasoning-time projection learning framework (TTPL) and multi-view decision fusion (MVDF) to enhance the generalization ability of neural combinatorial optimization models for large-scale vehicle routing problems.
π― What it does: By incorporating past state-based negative guidance (Self-Guidance) and adaptive chunking that dynamically switches between open-loop and closed-loop control based on action similarity during the denoising process of the Diffusion Policy, the action quality and real-time responsiveness of robot behavior cloning are improved.
Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
Zhoutong Wu (Peking University), Zhouchen Lin (Peking University)
CodeGenerationOptimizationRepresentation LearningTransformerLarge Language ModelText
π― What it does: This paper proposes SkipV1Former, a variant of cross-layer skip connections that uses the first layer's Value head in the deep layers of the Transformer, significantly reducing KV cache and enhancing representational capacity.
π― What it does: This paper studies the improvement of model-based reinforcement learning (MBRL) by encouraging the flatness of model training loss and proposes a method to integrate Sharpness-Aware Minimization (SAM) into world model training to enhance the performance of control policies.
π― What it does: This paper studies and improves Monte Carlo Tree Search (MCTS) for symbolic regression, proposing an extreme multi-armed bandit allocation strategy and evolution-inspired state transition actions.
Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration
Thomas Decker (Siemens AG), Florian Buettner (German Cancer Research Center)
CodeExplainability and InterpretabilityConvolutional Neural NetworkTransformerImageTabular
π― What it does: This study investigates the relationship between uncertainty calibration and perturbation-based explanations, proposing the ReCalX method to enhance model calibration under explanation-related perturbations, thereby improving explanation quality.
Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning
Yiqun Chen (Renmin University of China), Jiaxin Mao (Renmin University of China)
CodeRetrievalOptimizationLarge Language ModelReinforcement LearningTextRetrieval-Augmented Generation
π― What it does: This study proposes a joint optimization method for a multi-module retrieval-augmented generation (RAG) system, called MMOA-RAG. By treating modules such as query rewriting, document selection, and answer generation as multi-agents that share rewards and employing multi-agent PPO (MAPPO) for reinforcement learning, the method achieves collaborative optimization of the entire RAG pipeline.
Improving Reward Models with Proximal Policy Exploration for Preference-Based Reinforcement Learning
Yiwen Zhu (Zhejiang University), Bo An (Nanyang Technological University)
CodeReinforcement LearningSequentialBenchmark
π― What it does: A Proximal Policy Exploration (PPE) method is proposed to actively expand the coverage of the Preference Buffer to enhance the reliability of the reward model and accelerate preference-driven reinforcement learning.
Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting
Haoyu Zhang (Chinese University of Hong Kong), Tianshu Yu (Chinese University of Hong Kong)
CodeKnowledge DistillationRepresentation LearningTransformerLarge Language ModelPrompt EngineeringTextMultimodalityAudio
π― What it does: This paper proposes a teacher-student framework guided by MLLM (MMSLF), which trains the teacher model using context prompts generated by a general MLLM, and then distills knowledge to the student model, enabling it to perform multimodal sentiment analysis without using prompts.
π― What it does: This paper studies the issue of multiscale singularity in the score function when generating data using the Euclidean diffusion model under a known manifold structure, and proposes two methods to alleviate this problem.
Improving the Generation and Evaluation of Synthetic Data for Downstream Medical Causal Inference
Harry Amad (University of Cambridge), Mihaela van der Schaar (University of Cambridge)
CodeGenerationData SynthesisFlow-based ModelGenerative Adversarial NetworkTabularBiomedical Data
π― What it does: A theoretical framework for generating and evaluating synthetic data containing treatment variables is proposed, and based on this, the STEAM method is developed to generate synthetic medical data that meets the needs of causal inference.
π― What it does: A training method combining first-order and zero-order gradients, called FOGZO, is proposed to improve Quantization-Aware Training (QAT), particularly for low-precision weight quantization.
π― What it does: A model-agnostic post-processing framework called PIR is proposed, which improves the reliability of time series forecasting by identifying instance-level prediction errors and correcting them using local context (covariates, external variables) and global historical retrieval.
Antonio Orvieto (ELLIS Institute TΓΌbingen), Robert M. Gower (Flatiron Institute)
CodeOptimizationHyperparameter SearchTransformerLarge Language ModelText
π― What it does: A comprehensive evaluation of the training of Transformer language models in large-scale experiments, comparing the performance of Adam with various simplified optimizers (such as SignSGD, Signum, SGD, RMSprop), and revealing the key factors behind Adam's effectiveness through fine-grained hyperparameter search;
π― What it does: An Indirect Neural Corrector (INC) is proposed, embedding the learned correction term into the right side of the PDE, thereby achieving long-term accuracy improvement and stability enhancement in mixed operator solvers;
Fuxiang Zhang (Nanyang Technological University), Bo An (Nanyang Technological University)
CodeTransformerLarge Language ModelReinforcement LearningText
π― What it does: This paper proposes a self-verification framework that unifies answer generation and self-verification training through reinforcement learning, enhancing the performance of large language models in mathematical reasoning tasks and achieving scalability during testing.
Incomplete Multi-view Clustering via Hierarchical Semantic Alignment and Cooperative Completion
Xiaojian Ding (Nanjing University of Finance and Economics), Xiaoying Zhu (Nanjing University of Finance and Economics)
CodeAuto EncoderImage
π― What it does: A missing multi-view clustering framework based on hierarchical semantic alignment and collaborative completion, called HSACC, is proposed.
Incomplete Multi-view Deep Clustering with Data Imputation and Alignment
Jiyuan Liu (National University of Defense Technology), Kehua Guo (Central South University)
CodeRepresentation LearningAuto EncoderImage
π― What it does: The IMDC-DIA method is proposed, which utilizes a unique latent representation to reconstruct, complete, and cluster missing multi-view data;
InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction
Bin Lei (University of Minnesota), Caiwen Ding (Cisco Research)
CodeLarge Language ModelAgentic AIVision Language ModelImageVideoTextMultimodalityBenchmarkAudio
π― What it does: A multimodal general agent INFANTAGENT-NEXT is proposed, capable of interacting with computers through text, images, audio, and video;
Inference-Time Personalized Alignment with a Few User Preference Queries
Victor-Alexandru PΔdurean (Max Planck Institute for Software Systems), Adish Singla (Max Planck Institute for Software Systems)
CodeGenerationRecommendation SystemReinforcement Learning from Human FeedbackLarge Language ModelImageText
π― What it does: A method called USERALIGN is proposed to quickly obtain user preferences through a small number of pairwise comparisons during inference and to personalize the output of generative models.
Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search
Yuta Oshima (University of Tokyo), Hiroki Furuta (Google DeepMind)
CodeGenerationData SynthesisComputational EfficiencyVision Language ModelDiffusion modelVideoTextStochastic Differential EquationOrdinary Differential Equation
π― What it does: A beam search method for inference during the reverse process of diffusion models (Diffusion Latent Beam Search, DLBS) is proposed, which combines a deterministic DDIM lookahead estimator and utilizes linear weighting of multi-dimensional video quality metrics to calibrate rewards, significantly improving the alignment quality of text-to-video generation without updating model parameters.
InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
Yanggan Gu (Hong Kong Polytechnic University), Hongxia Yang (Hong Kong Polytechnic University)
CodeOptimizationReinforcement Learning from Human FeedbackTransformerLarge Language ModelReinforcement LearningText
π― What it does: This paper proposes the InfiFPO method, which achieves implicit model fusion during the preference alignment phase by directly integrating the sequence probabilities of multiple source models into the reference model, thereby enhancing the performance of the Pivot model.
Infinite Neural Operators: Gaussian processes on functions
Daniel Augusto de Souza (University College London), Marc Peter Deisenroth (University College London)
CodeTime SeriesPhysics Related
π― What it does: The study investigates infinite-width neural operators (NO), proving that their limit is a function-valued Gaussian process and providing the corresponding covariance function.
Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs
Mana Sakai (University of Tokyo), Masaaki Imaizumi (University of Tokyo)
CodeTransformer
π― What it does: This paper derives the infinite-width limit distribution of a single-layer multi-head attention under the standard scaling of 1/βn, proving that it exhibits a hierarchical non-Gaussian distribution.
π― What it does: A unified spatiotemporal autoregressive framework called InfinityStar is proposed, capable of generating high-resolution images and dynamic videos, supporting various tasks such as text-to-image, text-to-video, image-to-video, and video extrapolation.
CodeGenerationRetrievalTransformerLarge Language ModelTextRetrieval-Augmented Generation
π― What it does: This paper proposes a context selection method for retrieval-augmented generation (RAG) based on Context Influence Value (CI Value), aimed at filtering noisy retrieval results and improving generation quality.
π― What it does: The InfMasking method is proposed, which uses infinite masks to randomly obscure multimodal features and align unmasked representations to enhance collaborative information, forming an unsupervised multimodal representation learning framework.
Information Retrieval Induced Safety Degradation in AI Agents
Cheng Yu (Technical University of Munich), Orestis Papakyriakopoulos (Princeton University)
CodeRetrievalSafty and PrivacyTransformerLarge Language ModelTextRetrieval-Augmented Generation
π― What it does: The study investigates the phenomenon of security degradation in retrieval-augmented AI agents, assessing the impact of retrieval on rejection rates, bias, and harmfulness.
Moongyu Jeon (Yonsei University), Albert No (Yonsei University)
CodeDiffusion modelTextSequential
π― What it does: This paper proposes an information-theoretic framework for discrete diffusion models and proves two types of information-minimization relationships (I-MDSE and I-MDCE), thereby providing an exact likelihood interpretation of DSE and DCE losses.
π― What it does: This paper proposes Diff-ABFlow, a dual-modal optical flow estimation framework that utilizes event-frame dual modalities, employing a diffusion model and integrating appearance-boundary complementary information from frames and events.
π― What it does: Proposes InstanceAssemble, a method for layout-aware image generation that achieves this through instance assembly attention, allowing precise control over bounding box positions and supporting multimodal content.
InstructFlow: Adaptive Symbolic Constraint-Guided Code Generation for Long-Horizon Planning
Haotian Chi (Jilin University), Haiyan Yin (Agency for Science, Technology and Research)
CodeRobotic IntelligenceAI Code AssistantTransformerLarge Language ModelText
π― What it does: This paper studies a multi-agent, symbolic, feedback-driven robot code generation framework called InstructFlow, which can decompose natural language instructions into a hierarchical instruction graph, generate executable code, and perform local corrections by abstracting symbolic constraints when execution fails.
InstructRestore: Region-Customized Image Restoration with Human Instructions
Shuaizheng Liu (Hong Kong Polytechnic University), Lei Zhang (Hong Kong Polytechnic University)
CodeRestorationDiffusion modelImage
π― What it does: A region-customized image restoration framework called InstructRestore is proposed, which can achieve local restoration and bokeh control based on user-specified areas and enhancement levels.
π― What it does: This paper studies the discretization problem of solving high-dimensional partial differential equations (PDEs) using the backward stochastic differential equation (BSDE) method. It theoretically and experimentally verifies the irreducible bias generated by the Euler-Maruyama (EM) integration scheme in first-order self-consistency loss, and subsequently proposes converting forward-backward SDEs to Stratonovich form and using stochastic Heun integration, successfully eliminating this bias.
Interaction-Centric Knowledge Infusion and Transfer for Open Vocabulary Scene Graph Generation
Lin Li (Hong Kong University of Science and Technology), Long Chen (Tencent)
CodeObject DetectionKnowledge DistillationTransformerPrompt EngineeringVision Language ModelImage
π― What it does: A framework for open vocabulary scene graph generation based on interactive perception (ACC) is proposed, achieving more accurate object and relationship detection through interaction-driven knowledge injection and transfer.
Interactive Cross-modal Learning for Text-3D Scene Retrieval
Yanglin Feng (Sichuan University), Peng Hu (Sichuan University)
CodeRetrievalDomain AdaptationTransformerLarge Language ModelContrastive LearningTextPoint Cloud
π― What it does: An interactive text-3D scene retrieval method called IDeal is proposed, which enhances the alignment between text and 3D scenes through continuous question answering.
Interpretable and Parameter Efficient Graph Neural Additive Models with Random Fourier Features
Thummaluru Siddartha Reddy (Fujitsu Research of India), Mahesh Chandran (Fujitsu Research of India)
CodeExplainability and InterpretabilityComputational EfficiencyGraph Neural NetworkGraph
π― What it does: A lightweight interpretable graph neural additive model G-NAMRFF is designed, which integrates graph structure and node features using random Fourier features and learnable FIR filters, achieving independent contributions of each feature and interpretability;
Interpretable Next-token Prediction via the Generalized Induction Head
Eunji Kim (Microsoft Research), Jianfeng Gao (Microsoft Research)
CodeGenerationExplainability and InterpretabilityKnowledge DistillationTransformerLarge Language ModelTextBiomedical DataMagnetic Resonance ImagingRetrieval-Augmented Generation
π― What it does: This paper proposes an interpretable next-word prediction model GIM, which makes the induction head mechanism in large language models explicit as a retrieval-based method, and validates its effectiveness in text generation and fMRI prediction tasks.
Interpreting Emergent Features in Deep Learning-based Side-channel Analysis
Sengim Karayalcin, Stjepan Picek (Radboud University)
CodeExplainability and InterpretabilityConvolutional Neural NetworkTime SeriesBenchmark
π― What it does: The researchers conducted a post-hoc analysis of successful deep learning side-channel attack models using mechanistic interpretability methods, reverse-engineering and reconstructing the masked values in the cryptographic implementation, revealing which parts of the physical leakage the model utilized for predictions.
Interpreting vision transformers via residual replacement model
Jinyeong Kim (Yonsei University), Seong Jae Hwang (Yonsei University)
CodeExplainability and InterpretabilityTransformerAuto EncoderImage
π― What it does: This paper studies the internal features and mechanisms of the Vision Transformer (ViT) and proposes a residual substitute model based on residual flow to explain the decision-making process of ViT and debug model biases.
Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
Jiaye Qian (Sun Yat-sen University), Sibei Yang (Sun Yat-sen University)
CodeTransformerVision Language ModelImageText
π― What it does: A unified multi-path intervention framework called AllPath is proposed, which combines image-text and text-text attention paths to reduce hallucinations in large visual language models.
Hao Wang (Xiaohongshu Inc), Zhouchen Lin (Peking University)
CodeOptimizationTabular
π― What it does: A dual-layer optimization-based missing value imputation method called Kernel Point Imputation (KPI) is proposed, which adaptively selects the model form for each feature in RKHS and uses complete features (oracle features) as supervisory signals for imputation.
π― What it does: This paper proposes an Inverse Optimization Latent Variable Model (IO-LVM) that learns the cost latent variables of Constraint Optimization Problems (COP) through a variational autoencoder and generates constraint-satisfying paths or cycles during the decoding phase using a black-box solver.
Investigating and Mitigating Catastrophic Forgetting in Medical Knowledge Injection through Internal Knowledge Augmentation Learning
Yuxuan Zhou (Tsinghua University), Ji Wu (Tsinghua University)
CodeTransformerLarge Language ModelSupervised Fine-TuningTextBiomedical Data
π― What it does: This paper studies the phenomenon of catastrophic forgetting in large language models during the process of medical knowledge injection and proposes an InternAL method that enhances injection using the model's internal knowledge.
InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems
Noam Elata (Technion), Michael Elad
CodeRestorationGenerationDiffusion modelImage
π― What it does: InvFusion is proposed, a diffusion model framework that simultaneously considers both training-based and zero-shot methods in inverse problems;
InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy
Vishnu Vinod (Cognitive and Robotics Artificial Intelligence Institute, Indian Institute of Technology Madras), Abhradeep Guha Thakurta
CodeGenerationSafty and PrivacyLarge Language ModelText
π― What it does: A framework named InvisibleInk is proposed for long text generation of large models while maintaining strict differential privacy (DP) guarantees.
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningTabularBenchmark
π― What it does: In an offline environment, an implicit policy g(s'|s) is learned using only expert state trajectories and state-action data containing suboptimal behaviors, achieving observational learning by matching the joint distribution of state transitions.