Conference on Neural Information Processing Systems · 1376 papers
Laplacian Canonization: A Minimalist Approach to Sign and Basis Invariant Spectral Embedding
George Ma, Yisen Wang (Peking University)
CodeGraph Neural NetworkGraphBiomedical Data
🎯 What it does: A normalization method for the eigenvectors of the Laplacian operator (Laplacian Canonization) is proposed to eliminate sign and basis ambiguities in spectral embedding;
Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
Xinyi Wang (University of California), William Yang Wang (University of California)
CodeClassificationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: A latent variable model for LLM is proposed from a Bayesian perspective, and a two-step demonstration selection algorithm is designed: first, a small LLM learns the task concept words, and then examples that best infer the concept are selected for in-context learning in the large model.
CodeTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: A semi-parametric reinforcement learning framework called REMEMBERER based on LLM is proposed, which utilizes external sustainable experience memory to enhance decision-making performance.
Large Language Models are Visual Reasoning Coordinators
Liangyu Chen (Nanyang Technological University), Ziwei Liu (Nanyang Technological University)
CodeTransformerLarge Language ModelPrompt EngineeringVision Language ModelMultimodality
🎯 What it does: This paper proposes Cola, which utilizes large language models as coordinators to integrate descriptions and answers generated by multiple visual language models (such as OFA and BLIP) through natural language prompts to complete visual reasoning tasks.
Large Language Models Are Zero-Shot Time Series Forecasters
Nate Gruver (New York University), Andrew Gordon Wilson (New York University)
CodeTransformerLarge Language ModelTime Series
🎯 What it does: This paper proposes a framework called LLMTIME that transforms time series into text strings and then utilizes pre-trained large language models (LLMs) for zero-shot prediction.
Large Language Models can Implement Policy Iteration
Ethan Brooks (University of Michigan), Satinder Singh (University of Michigan)
CodeTransformerLarge Language ModelReinforcement LearningPrompt EngineeringText
🎯 What it does: Implementing policy iteration using large language models, leveraging context learning to infer world models and policies, completing Q-value estimation and greedy improvement without the need for gradients or expert demonstrations.
Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
Noah Hollmann (University of Freiburg), Frank Hutter (University of Freiburg)
CodeExplainability and InterpretabilityData-Centric LearningTransformerLarge Language ModelPrompt EngineeringTabularChain-of-Thought
🎯 What it does: Using large language models to achieve context-aware automatic feature engineering (CAAFE), generating interpretable Python feature engineering code to enhance the predictive performance of tabular data.
Large Language Models of Code Fail at Completing Code with Potential Bugs
Tuan Dinh (University of Wisconsin-Madison), George Karypis (Amazon Web Services)
CodeAI Code AssistantTransformerLarge Language ModelText
🎯 What it does: This study investigates the ability of large language models to complete code in the presence of potentially defective code contexts, and proposes two new datasets and three post-correction methods.
🎯 What it does: This paper proposes a Transformer-based 3D motion transfer framework called LART, which can transfer motion from a driving sequence to unseen static 3D meshes while achieving high-fidelity motion generation and preserving the geometric details of the meshes.
🎯 What it does: This paper proposes an exploration method called Lattice, which injects time-related and actuator-related noise into the last layer of the policy network's latent state to enhance learning efficiency in high-dimensional continuous control tasks.
Latent Field Discovery in Interacting Dynamical Systems with Neural Fields
Miltiadis Kofinas (University of Amsterdam), Efstratios Gavves (University of Amsterdam)
CodeGraph Neural NetworkAuto EncoderTime SeriesSequentialPhysics Related
🎯 What it does: The Aether framework is proposed, which utilizes neural fields and equivariant graph networks to unsupervisedly discover global field effects in the system from observed trajectories and applies them to trajectory prediction.
Sebastian Zeng (University of Salzburg), Roland Kwitt (University of Salzburg)
CodeTime SeriesStochastic Differential Equation
🎯 What it does: This paper proposes a variational Bayesian method for learning latent stochastic differential equations (latent SDEs) on a homeomorphic space (using the unit sphere as an example), constructing solvable and structure-preserving SDEs using the action of Lie groups;
🎯 What it does: This paper proposes a method to directly estimate and apply a transformation T to translate between the latent spaces of pre-trained models, achieving zero-shot model stitching across architectures, modalities, and tasks without the need to retrain the decoder.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
Weixi Feng (University of California), William Yang Wang (University of California)
CodeGenerationData SynthesisTransformerLarge Language ModelPrompt EngineeringImageMultimodality
🎯 What it does: Utilizing large language models (LLM) to generate visual layouts of 2D images and 3D indoor scenes through in-context learning and CSS-style prompts, achieving high-quality visual content generation in collaboration with existing layout-to-image generation models.
LD2: Scalable Heterophilous Graph Neural Network with Decoupled Embeddings
Ningyi Liao (Nanyang Technological University), Jieming Shi (Hong Kong Polytechnic University)
CodeGraph Neural NetworkGraph
🎯 What it does: This paper proposes a scalable heterogeneous graph neural network LD2, which utilizes precomputed low-dimensional adjacency and feature embeddings, completely decoupling graph structure propagation during training, and supports lightweight mini-batch training for large-scale heterogeneous graphs.
LEACE: Perfect linear concept erasure in closed form
Nora Belrose (EleutherAI), Stella Biderman (Booz Allen Hamilton)
CodeTransformerLarge Language ModelText
🎯 What it does: The LEACE (Least-Squares Concept Erasure) method is proposed, which can completely eliminate specified concepts (such as gender or part of speech) from internal representations through linear transformations without changing model parameters.
Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs
Dmitry Chistikov (University of Warwick), Ranko Lazic (University of Warwick)
CodeTabular
🎯 What it does: This paper analyzes the training dynamics of a single hidden layer ReLU network learning a single neuron, starting from small initialization and using gradient flow, proving convergence to zero loss and an implicit bias towards minimal rank.
Learning Adaptive Tensorial Density Fields for Clean Cryo-ET Reconstruction
YUANHAO WANG, Wolfgang Heidrich (King Abdullah University of Science and Technology)
CodeRestorationData SynthesisImage
🎯 What it does: The paper proposes a deep learning framework based on adaptive quadtree and tensor decomposition for the rapid reconstruction and denoising of 3D density fields from tilted series Cryo-ET data.
Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning
Guozheng Ma (Tsinghua University), Dacheng Tao (University of Sydney)
CodeAutonomous DrivingReinforcement LearningImage
🎯 What it does: This paper systematically analyzes the key attributes of data augmentation (DA) in visual reinforcement learning through experiments, and based on this, proposes a new random PadResize (Rand PR) transformation and a periodic fusion augmentation scheme (CycAug) to enhance sample efficiency.
Panagiotis Misiakos (ETH Zurich), Markus Püschel (ETH Zurich)
CodeOptimizationGraph
🎯 What it does: A continuous optimization method named SparseRC is proposed for learning the sparse causal linear structural equation model (SEM) that generates directed acyclic graphs (DAGs);
Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation
Zihao Yue (Renmin University of China), Qin Jin (Renmin University of China)
CodeGenerationRetrievalTransformerSupervised Fine-TuningVision Language ModelImageVideoText
🎯 What it does: A new training objective called SMILE (Semipermeable Maximum Likelihood Estimation) is proposed, which normalizes probabilities only for the current set of label words at each time step, suppressing the 'concise optimization' caused by MLE and encouraging the generation of more detailed image descriptions; further training and evaluation are conducted on the BLIP model.
Learning Domain-Aware Detection Head with Prompt Tuning
Haochen Li (Intelligent Software Research Center, Institute of Software, Chinese Academy of Sciences), Yunji Chen (State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences)
CodeObject DetectionDomain AdaptationTransformerPrompt EngineeringVision Language ModelImage
🎯 What it does: This paper proposes a method that utilizes a Visual-Language Model (VLM) as a robust detection backbone and generates detection heads for each domain through learnable domain adaptation prompts to achieve adaptive learning for cross-domain object detection.
🎯 What it does: A self-supervised learning method based on maximum manifold capacity (MMCR) is proposed, which simplifies the manifold capacity theory into a nuclear norm loss to directly optimize the linear separability and compressibility of representations.
Learning Efficient Surrogate Dynamic Models with Graph Spline Networks
Chuanbo Hua (Korea Advanced Institute of Science and Technology), Jinkyoo Park (Korea Advanced Institute of Science and Technology)
CodeGraph Neural NetworkGraphPhysics Related
🎯 What it does: This paper proposes GRAPHSPLINENETS, which quickly predicts the dynamics of continuous physical systems through graph neural networks and orthogonal spline interpolation (OSC).
🎯 What it does: A long-term MCMC amortization method based on diffusion models (DAMC) is proposed for learning latent space energy-based models (LEBM), addressing the issue of low sampling quality in traditional short-run MCMC.
🎯 What it does: This paper proposes a two-stage LSTK framework, which first uses distance supervision and a text entailment model to extract soft triples from text corpora, and then uses these soft triples along with the original structured triples to learn text enhancement rules (TE-rules) for inductive reasoning in knowledge graphs.
🎯 What it does: By leveraging the rich semantic information and coarse location (whole image box) from additional image classification data to enhance the training of long-tail object detectors, the detection model achieves better recognition capabilities for low-sample categories.
Learning Interpretable Low-dimensional Representation via Physical Symmetry
Xuanjie Liu (Mohamed bin Zayed University of Artificial Intelligence), Gus Xia (Mohamed bin Zayed University of Artificial Intelligence)
CodeExplainability and InterpretabilityRepresentation LearningAuto EncoderVideoTime SeriesPhysics RelatedAudio
🎯 What it does: The paper proposes using physical symmetry as a prior constraint for self-supervised learning to learn interpretable low-dimensional temporal representations.
🎯 What it does: This paper addresses the issue of GNN generalization decline caused by changes in graph structure and proposes the Cluster Information Transfer (CIT) mechanism to learn node representations that are robust to structural drift.
🎯 What it does: This paper proposes a non-parametric domain-invariant representation learning based on the Nadaraya-Watson (NW) head, which manipulates the support set to encode causal assumptions during training and inference, encouraging the model to learn representations that rely solely on environment-independent features.
Learning Large Graph Property Prediction via Graph Segment Training
Kaidi Cao (Stanford University), Bryan Perozzi (Google)
CodeGraph Neural NetworkGraph
🎯 What it does: This paper proposes a framework based on Graph Segment Training (GST), which utilizes pre-partitioned graph segments and a historical embedding table to achieve constant memory training for large graph attribute prediction, and mitigates the issue of outdated embeddings through prediction head fine-tuning and old embedding dropout techniques.
🎯 What it does: A framework is proposed that utilizes gradients and Bayesian model selection to automatically learn hierarchical equivariance, capable of discovering symmetry constraints at each layer from training data.
🎯 What it does: The Modulated Transformation Module (MTM) is proposed, which achieves explicit modeling of geometric deformations in generated images and videos by introducing learnable offsets modulated by latent vectors in convolution.
Learning Motion Refinement for Unsupervised Face Animation
Jiale Tao (University of Electronic Science and Technology of China), Lixin Duan (University of Electronic Science and Technology of China)
CodeGenerationData SynthesisVideo
🎯 What it does: This paper proposes an unsupervised facial animation framework that utilizes a prior coarse motion model (local affine/thin-plate spline) to generate coarse motion flows, and designs a non-prior refinement module based on structural correlation volumes to iteratively refine the motion to capture micro-expressions and detailed movements.
Learning non-Markovian Decision-Making from State-only Sequences
Aoyang Qin (Tsinghua University), Sirui Xie (University of California, Los Angeles)
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackSequential
🎯 What it does: A generative model called LanMDP based on non-Markov decision processes (nMDP) is proposed to learn implicit actions and make decisions from demonstration sequences that contain only states.
🎯 What it does: A probabilistic symmetrization framework is proposed, which aligns any base model (such as MLP or Transformer) to be an equivariant function of a given group by learning the distribution of group transformations under input conditions.
🎯 What it does: The study achieves robustness in inverse problems by adding Gaussian noise (jittering) during training, proving that under the linear model of subspace denoising, jittering can obtain an optimal robust estimator equivalent to adversarial training, and its effectiveness is validated in tasks such as image denoising, deconvolution, and compressed sensing.
Learning Rate Free Sampling in Constrained Domains
Louis Sharrock (Lancaster University), Christopher Nemeth (Lancaster University)
CodeOptimizationTabular
🎯 What it does: A series of completely learning rate-free particle sampling algorithms are proposed for sampling within constrained domains, along with their theoretical framework and implementation.
🎯 What it does: This paper introduces the concept of reproducibility from measurement theory into deep embedding learning, evaluates the reproducibility of embeddings using ICC, and further designs an ICC regularizer as a supplement to contrastive loss, promoting the model to learn more reproducible embeddings.
Learning Robust Statistics for Simulation-based Inference under Model Misspecification
Daolang Huang (Aalto University), Samuel Kaski (Aalto University)
CodeRecurrent Neural NetworkAuto EncoderTime Series
🎯 What it does: A robust statistical learning method based on regularization loss is proposed to address model misspecification in simulation-based inference (SBI);
🎯 What it does: A single-source edge-level graph neural network (REST) is proposed for inducing subgraph representations, addressing the issue of decreased inference performance caused by the mixing of irrelevant rules within subgraphs.
Learning Shared Safety Constraints from Multi-task Demonstrations
Konwoo Kim (Carnegie Mellon University), Steven Wu
CodeSafty and PrivacyRobotic IntelligenceReinforcement LearningSequential
🎯 What it does: This paper proposes an inverse constraint learning method based on multi-task demonstrations to learn shared safety constraints from expert demonstrations.
Learning Time-Invariant Representations for Individual Neurons from Population Dynamics
Lu Mi (Allen Institute for Brain Science), Uygar Sümbül (Allen Institute for Brain Science)
CodeClassificationRepresentation LearningTransformerSupervised Fine-TuningMultimodalityBiomedical Data
🎯 What it does: Using a self-supervised neural dynamic model, we learn time-invariant single-cell representations and use these representations to predict the transcriptomic classification of cells.
Learning to Augment Distributions for Out-of-distribution Detection
Qizhou Wang (Hong Kong Baptist University), Bo Han (Hong Kong Baptist University)
CodeAnomaly DetectionOptimizationImage
🎯 What it does: A distribution-augmented OOD learning framework (DAL) is proposed, which reduces the distribution difference between auxiliary OOV and true OOV by searching for the worst OOV distribution within the Wasserstein ball, thereby improving open-world OOD detection performance.
🎯 What it does: This study investigates how to accelerate mixed-integer linear programming (MILP) solvers by learning to select and configure separators within the Branch-and-Cut framework, proposing the 'Separator Configuration' task and providing an end-to-end learning method.
🎯 What it does: This paper proposes a method for auxiliary dataset grouping called MolGroup, based on a routing mechanism and dual-layer optimization, aimed at small molecule property prediction tasks. It can automatically identify and select auxiliary datasets that have a high affinity with the target dataset, thereby enhancing model performance.
🎯 What it does: This paper studies how to efficiently fine-tune pre-trained models in reinforcement learning and proposes the L2M method to avoid catastrophic forgetting.
🎯 What it does: This paper presents NeuOpt, a learning-search (L2S) solver that can flexibly execute any k-opt, and introduces Guided Infeasible Region Exploration (GIRE) in CVRP to overcome feasibility mask limitations.
🎯 What it does: This paper proposes an end-to-end learning framework for generative retrieval of document identifiers (docid) called GENRET, replacing traditional rule-based or clustering-based segmentation methods.
Dan Friedman (Princeton University), Danqi Chen (Princeton University)
CodeTransformerText
🎯 What it does: A Transformer model that can be directly trained and subsequently converted into a readable program—Transformer Programs—is proposed, and its effectiveness is validated across various tasks.
🎯 What it does: Combining the pre-trained CLIP model with a learned ranking method to enhance ordinal classification performance using language priors.
🎯 What it does: Designed an unsupervised learning framework based on GFlowNet to solve NP-hard combinatorial optimization problems in graph theory (maximum independent set, maximum clique, minimum dominating set, maximum cut).
Leveraging Locality and Robustness to Achieve Massively Scalable Gaussian Process Regression
Robert F Allison, Edward Pyzer-Knapp (IBM Research)
CodeGaussian SplattingTabular
🎯 What it does: An efficient Gaussian process regression algorithm that decouples parameter estimation from prediction is proposed, utilizing nearest neighbor prediction to achieve low computational cost and good prediction accuracy.
🎯 What it does: By utilizing sparse and shared feature activation in multi-task supervised learning, we learn disentangled representations to enhance generalization capabilities on out-of-distribution (OOD) data.
🎯 What it does: This paper proposes the VCD framework, which includes the Visual Center Expert VCD-E and the Camera-Only Apprentice VCD-A. It utilizes LiDAR depth as a prior, long-term sequence fusion, trajectory-based distillation, and occupancy reconstruction to enhance the 3D detection performance of cameras.
LICO: Explainable Models with Language-Image COnsistency
Yiming Lei (Fudan University), Hongming Shan (Fudan University)
CodeClassificationExplainability and InterpretabilityPrompt EngineeringVision Language ModelContrastive LearningImage
🎯 What it does: The LICO framework is proposed, which enhances the interpretability and classification performance of visual models by aligning image feature space with CLIP text features.
Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory
Xin Cheng (Peking University), Rui Yan (Remin University of China)
CodeGenerationRetrievalTransformerLarge Language ModelTextRetrieval-Augmented Generation
🎯 What it does: Proposes the Selfmem framework, which utilizes text generated by the model itself as infinite memory to iteratively enhance the quality of retrieval-augmented text generation.
CodeGenerationTransformerLarge Language ModelDiffusion modelText
🎯 What it does: A likelihood-based diffusion language model called Plaid has been researched and implemented, resulting in Plaid1B, which outperforms the small autoregressive model GPT-2 124M in zero-shot log likelihood;
🎯 What it does: The LinGCN framework is designed to reduce the multiplication depth and latency of graph convolutional network inference under CKKS homomorphic encryption by utilizing differentiable structured linearization and learnable polynomial substitutes.
🎯 What it does: By introducing syntactic information during the inference process of the diffusion model and utilizing the loss function of the cross-attention map, the attention maps of modifiers and entity nouns overlap while separating from other words, thereby improving the attribute-object binding errors in text-to-image generation.
LinkerNet: Fragment Poses and Linker Co-Design with 3D Equivariant Diffusion
Jiaqi Guan (University of Illinois Urbana-Champaign), Jianzhu Ma (Tsinghua University)
CodeDrug DiscoveryDiffusion modelGraph
🎯 What it does: This paper proposes the LinkerNet model, which jointly generates fragment poses and 3D linkers without knowing the relative positions of the fragments, achieving the co-design of PROTACs and general fragment linkers.
LLM-Pruner: On the Structural Pruning of Large Language Models
Xinyin Ma (National University of Singapore), Xinchao Wang (National University of Singapore)
CodeCompressionTransformerLarge Language ModelText
🎯 What it does: A task-agnostic structured pruning framework called LLM-Pruner has been designed and implemented for compressing large language models (LLaMA, Vicuna, ChatGLM), retaining only a small amount of public data and completing compression and recovery within three hours.
LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
Yujie Lu (University of California), William Yang Wang (University of California)
CodeGenerationData SynthesisExplainability and InterpretabilityTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: A text and image alignment evaluation framework called LLMScore based on large language models is proposed, which can generate multi-granularity (whole image and object-level) visual descriptions and provide scores and reasons.
Locality Sensitive Hashing in Fourier Frequency Domain For Soft Set Containment Search
Indradyumna Roy (Indian Institute of Technology Bombay), Abir De (Indian Institute of Technology Gandhinagar)
CodeRetrievalOptimizationTabular
🎯 What it does: Designed and implemented an asymmetric local sensitive hash based on the Fourier frequency domain (FOURIERHASHNET) for soft set inclusion retrieval.
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Jae Sung Park (University of Washington), Yejin Choi (University of Washington)
CodeGenerationKnowledge DistillationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringVision Language ModelImageTextMultimodality
🎯 What it does: This paper generates visual commonsense question answering by allowing a large language model (ChatGPT) to produce diverse textual descriptions of images and local regions. It then uses a supervised critique model to filter high-quality samples, constructing a corpus of 1 million local visual commonsense data, and fine-tunes visual-language models such as BLIP-2 on this corpus, enabling them to answer questions in a 'pointing' manner under zero-shot conditions.
Lockdown: Backdoor Defense for Federated Learning with Isolated Subspace Training
Tiansheng Huang (Georgia Institute of Technology), Ling Liu (Georgia Institute of Technology)
CodeFederated LearningImage
🎯 What it does: In the federated learning environment, Lockdown is proposed, utilizing isolated subspace training and dynamic subspace search to counteract backdoor attacks, addressing the issue of toxic coupling in traditional sparsification defenses within federated learning.
LogSpecT: Feasible Graph Learning Model from Stationary Signals with Recovery Guarantees
Shangyuan Liu (Chinese University of Hong Kong), Anthony Man-Cho So (Chinese University of Hong Kong)
CodeGraph Neural NetworkGraph
🎯 What it does: This paper proposes LogSpecT and its robust version rLogSpecT for learning graph structures from stationary graph signals, addressing the infeasibility issues of traditional SpecT and rSpecT.
🎯 What it does: A dynamical model TDM based on time-reversible symmetry is proposed, and an efficient offline reinforcement learning algorithm TSRL is designed based on this model.
🎯 What it does: Proposes the Lookaround optimizer, which alternates between multi-data augmentation training and weight averaging to continuously explore the interior of loss basins and achieve flatter minima.
🎯 What it does: An end-to-end lossy image compression framework is proposed, combining traditional transform coding with conditional diffusion models, achieving reverse diffusion reconstruction controlled by content latent variables.
🎯 What it does: An unsupervised graph-level representation learning framework based on the Lovász number (Lovász principle) is proposed, enhancing its expressive power through subgraph Lovász numbers.
🎯 What it does: A self-supervised visual model LVM-Med based on second-order graph matching was trained, undergoing large-scale pre-training on approximately 1.3M medical images and fine-tuning on 15 downstream tasks (segmentation, classification, detection, etc.);
Macro Placement by Wire-Mask-Guided Black-Box Optimization
Yunqi Shi (Nanjing University), Chao Qian (Nanjing University)
CodeOptimizationTabularBenchmark
🎯 What it does: A new macro block placement framework called WireMask-BBO is proposed, which achieves high-quality non-overlapping placement by directly treating macro block positions as gene expressions and using wire mask-based greedy improvement evaluation.
🎯 What it does: This paper proposes a graph neural network model named MAG-GNN, which enhances the expressive power of GNNs by using reinforcement learning to select a small number of the most discriminative subgraphs from all possible subgraphs, while significantly reducing the computational overhead of subgraph enumeration.
Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
Baohao Liao (University of Amsterdam), Christof Monz (University of Amsterdam)
CodeTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: This paper proposes three reversible parameter and memory-efficient fine-tuning methods (MEFT) that transform pre-trained language models (PLMs) into reversible networks without additional pre-training, thereby significantly reducing activation memory usage by only caching the final output during fine-tuning.
Sang Keun Choe (Carnegie Mellon University), Eric Xing (Carnegie Mellon University)
CodeMeta LearningSupervised Fine-TuningImageText
🎯 What it does: A scalable meta-learning framework named SAMA is proposed, enabling efficient training of meta-learning in large-scale models and multi-GPU environments.
🎯 What it does: A black-box model extraction attack method based on public data, MARICH, is proposed, which can construct a replica model that is distributionally equivalent to the target model with a very small number of queries.
MarioGPT: Open-Ended Text2Level Generation through Large Language Models
Shyam Sudhakaran, Sebastian Risi
CodeGenerationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
🎯 What it does: This paper presents MarioGPT, a large language model capable of generating playable Super Mario Bros levels based on natural language prompts.
🎯 What it does: This paper proposes the Markovian sliced Wasserstein (MSW) distance, which improves the redundancy projection problem of traditional Sliced Wasserstein by introducing a first-order Markov structure in the projection direction.
🎯 What it does: An efficient video semantic segmentation framework based on mask propagation, MPVSS, is proposed. It uses a powerful query-based image segmenter to generate binary masks and class predictions on key frames, and then employs a query-based flow estimation module to generate segment-level flows for each mask. Finally, the masks are projected onto non-key frames through bilinear projection along the flow direction, achieving fast and accurate video segmentation.
🎯 What it does: A self-supervised framework called Masked Image Residual Learning (MIRL) is proposed to address the performance degradation issue of deep Vision Transformers (ViT) during pre-training with Masked Image Modeling (MIM) and to enable efficient training of deeper ViTs.
🎯 What it does: A Masked Space-Time Hash encoding (MSTH) method is proposed, which decomposes dynamic scenes into a weighted combination of 3D hash encoding and 4D hash encoding, and achieves automatic separation of static and dynamic areas through a learnable mask, significantly reducing hash collisions and training costs.
Mass-Producing Failures of Multimodal Systems with Language Models
Shengbang Tong (University of California Berkeley), Jacob Steinhardt (University of California Berkeley)
CodeGenerationAnomaly DetectionTransformerLarge Language ModelPrompt EngineeringTextMultimodality
🎯 What it does: MULTIMON has been developed, an automated evaluation framework based on error consistency and large language models, capable of mining single errors from text corpora, generalizing them into systematic failures, and automatically generating new failure instances;
🎯 What it does: A general framework called MathNAS is proposed to transform NAS into Integer Linear Programming (ILP), utilizing modular search space to split into block performance evaluation and network performance prediction, thereby reducing the network search complexity from exponential to polynomial level.
Matrix Compression via Randomized Low Rank and Low Precision Factorization
Rajarshi Saha (Stanford University), Mert Pilanci (Stanford University)
CodeCompressionImageText
🎯 What it does: A matrix compression algorithm LPLR is proposed, which simultaneously achieves low-rank decomposition and low-precision quantization, and provides a theoretical error upper bound.
🎯 What it does: A self-supervised audio-video learning framework called MAViL has been developed, utilizing three objectives: masked reconstruction, cross-modal and intra-modal contrastive learning, and self-training to learn unified audio-video representations.
Davoud Ataee Tarzanagh (University of Pennsylvania), Samet Oymak (University of Michigan)
CodeOptimizationTransformerImage
🎯 What it does: This paper studies the optimization dynamics of the Transformer attention mechanism, proving that gradient descent converges to the maximum margin token selection on attention weights, and explores the joint convergence of attention heads and prediction heads.
Maximization of Average Precision for Deep Learning with Adversarial Ranking Robustness
Gang Li (Texas A&M University), Tianbao Yang (Texas A&M University)
CodeOptimizationAdversarial AttackImage
🎯 What it does: A framework for adversarial robustness optimization targeting average precision (AP) is proposed, along with a stochastic optimization algorithm that can be efficiently solved without the need for large batch training.
🎯 What it does: A unified RL framework called MEX (Maximize to Explore) is proposed, which automatically balances exploration and exploitation by simultaneously maximizing the estimated error penalty and the hypothesis-based expected return within a single unconstrained objective.
🎯 What it does: Deploying visual Transformers on microcontrollers with extremely low memory, the MCUFormer framework is proposed to achieve co-optimization of hardware and algorithms.
🎯 What it does: A novel zero-cost proxy called MeCo (and its optimized version MeCo opt) is proposed, which evaluates network architecture by calculating the minimum eigenvalue of the feature map from a single random input;
🎯 What it does: This paper proposes a cross-lingual medical vision-language pre-training framework Med-UniC, which combines chest X-ray images and radiology reports in English and Spanish to achieve cross-lingual visual-language alignment and self-supervised visual alignment, and unifies text representation through cross-lingual text alignment regularization (CTR).
CodeGenerationData-Centric LearningAI Code AssistantTransformerLarge Language ModelSupervised Fine-TuningText
🎯 What it does: Proposes the Meet-in-the-Middle (MIM) pre-training paradigm, which simultaneously trains forward (left-to-right) and backward (right-to-left) language models within the same shared decoder, and encourages consistency between the two models' word distributions at each position through consistency regularization.
🎯 What it does: A compressed low-precision optimizer that compresses the state of optimizers like Adam/AdamW to 4-bit integers, balancing accuracy and memory efficiency.
🎯 What it does: Converts regular expression rules into byte-level recurrent neural networks, and then transforms them into a pooling soft random forest that can be implemented on switches through semi-supervised knowledge distillation, achieving high zero-shot accuracy, better performance after training, and significantly improved throughput after deployment.
Metropolis Sampling for Constrained Diffusion Models
Nic Fishman (University of Oxford), Valentin De Bortoli (École Normale Supérieure Ulm)
CodeGenerationData SynthesisOptimizationDiffusion modelScore-based ModelTabularTime Series
🎯 What it does: A Metropolis-based discretization method for reflected Brownian motion is proposed for training diffusion models in constrained Riemannian multi-dimensional spaces.