🎯 What it does: This paper studies the implementation and theory of Gradient Regularization (GR) in deep learning, proposing the use of the finite difference method (forward/few-difference) to compute the gradient regularization term, significantly reducing computational costs and improving generalization performance.
Understanding Int4 Quantization for Language Models: Latency Speedup, Composability, and Failure Cases
Xiaoxia Wu (Microsoft), Yuxiong He (Microsoft)
CodeComputational EfficiencyKnowledge DistillationTransformerLarge Language ModelText
🎯 What it does: A feasibility study on INT4 weight and activation quantization for Transformer language models is conducted, and an end-to-end INT4 inference pipeline is constructed.
🎯 What it does: This paper analyzes and utilizes effective resistance to measure the over-compression problem of GNNs, and proposes a reconnection algorithm based on total effective resistance to alleviate over-compression.
Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits
Xuejie Liu (Peking University), Yitao Liang (Beijing Institute for General Artificial Intelligence)
CodeGenerationKnowledge DistillationImage
🎯 What it does: By distilling the latent variable information from deep generative models (such as VQ-VAE) into tractable probabilistic circuits (PC), the PC can achieve higher likelihood performance on image modeling tasks;
🎯 What it does: Under the premise of unsupervised learning, a Lift-Map-Detect (LMD) framework is designed utilizing the denoising mapping capability of diffusion models. This involves first performing mask lifting (Lift) on the input image, then using the diffusion model for filling (Map), and finally calculating the perceptual distance between the original and reconstructed images (Detect) to determine whether the sample is OOD.
🎯 What it does: The paper studies the geometric structure of latent space in generative models (such as GANs and VAEs) and proposes that the latent space should be a 'simplicial cluster' to minimize the occurrence of generated samples falling outside the support of the target distribution, and verifies the relationship between this structure and model performance.
🎯 What it does: This study investigates the implicit absolute positional information in padding within convolutional neural networks and proposes a new evaluation method called PPP, systematically analyzing its formation, evolution, and impact on model performance.
🎯 What it does: A general export algorithm called UPSCALE is proposed, which can perform channel pruning without the constraints of traditional methods, and eliminates memory copies caused by inconsistent pruning in multi-branch networks through channel reordering, thus balancing high accuracy and low latency.
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Gati Aher (Olin College of Engineering), Adam Tauman Kalai (Microsoft Research)
CodeGenerationTransformerLarge Language ModelPrompt EngineeringText
🎯 What it does: A zero-shot method named Turing Experiment (TE) is proposed and implemented, utilizing large language models (such as GPT) to simulate multiple 'human' participants, replicating four classic human experiments (Ultimatum Game, garden-path sentences, Milgram obedience experiment, Wisdom of Crowds) and evaluating the model's simulation accuracy.
🎯 What it does: A mixed hypernetwork model VaMoH based on variational autoencoders is proposed to learn distributions in function space, supporting data generation and inference for continuous coordinates.
🎯 What it does: A retrieval-augmented question answering framework based on variational inference (VOD) is proposed and implemented, capable of end-to-end training of the retriever and reader models, and estimating task likelihood through self-normalized importance sampling.
🎯 What it does: This paper proposes a new graph neural network, VonMisesNet, for directly generating 3D molecular conformations that satisfy the Boltzmann distribution from molecular structures.
Wasserstein Barycenter Matching for Graph Size Generalization of Message Passing Neural Networks
Xu Chu (Tsinghua University), Hong Mei (Peking University)
CodeGraph Neural NetworkGraph
🎯 What it does: A Wasserstein barycenter matching (WBM) layer is proposed, embedded into a message passing neural network (MPNN) to enhance the graph size generalization capability;
Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributes
Zhaowei Zhu (University of California, Santa Cruz), Yang Liu (ByteDance Research)
CodeTabular
🎯 What it does: A fairness assessment method based on weak proxies is proposed, which can accurately estimate group fairness in the absence of sensitive attributes and provide theoretical guarantees;
🎯 What it does: This paper conducts a systematic analysis of the representations in the first layer of convolutional neural networks (CNNs) and proposes the energy profile metric to quantify the sensitivity of filters to different spatial frequencies. Experiments demonstrate that CNNs with different initializations, architectures, datasets, and even training with random labels exhibit a high consistency in the energy profile of the first layer. Subsequently, an analytical formula for the energy profile is derived in a simplified linear CNN, proving that this energy profile approaches whitening during gradient descent training, and the formula is fitted to the energy profile of real nonlinear CNNs, validating the consistency between theory and practice.
🎯 What it does: This study investigates the generalization problem of offline goal-conditioned reinforcement learning (offline GCRL) on unseen targets, proposing theoretical analysis and a practical algorithm (GOAT) while constructing a new evaluation benchmark.
What Makes Entities Similar? A Similarity Flooding Perspective for Multi-sourced Knowledge Graph Embeddings
Zequn Sun (Nanjing University), Wei Hu (Nanjing University)
CodeGraph Neural NetworkGraph
🎯 What it does: This paper interprets the working principles of translation-based and aggregation-based entity alignment models by viewing the correction of entity pair similarity as a fixed point of similarity flooding, and based on this, proposes two improvement methods: similarity flooding based on entity combinations and autoregressive neighborhood aggregation.
🎯 What it does: This paper proposes X-Paste, a scalable Copy-Paste enhancement framework that utilizes CLIP and StableDiffusion to automatically collect and filter large-scale instances, generating high-quality masks that are then collaged onto background images for training.