Conference on Neural Information Processing Systems Β· 1376 papers
Fine-Grained Visual Prompting
Lingfeng Yang (Nanjing University of Science and Technology), Jian Yang (Beijing Academy of Artificial Intelligence)
CodeClassificationObject DetectionSegmentationTransformerPrompt EngineeringVision Language ModelImage
π― What it does: This paper proposes Fine-Grained Visual Prompts (FGVP), which obtain semantic masks through the Segment Anything model and apply Blur Reverse Mask visual prompts on images to achieve zero-shot instance-level tasks (such as pointing expression understanding and part detection).
CodeOptimizationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringText
π― What it does: A memory-efficient zero-order optimizer MeZO is proposed, which can fine-tune large-scale language models using only forward passes.
π― What it does: This paper proposes Sharpness-Aware Posterior (SA-Posterior), which incorporates the consideration of model flatness in the posterior inference of Bayesian neural networks, resulting in models sampled from the posterior that are broader and more robust.
π― What it does: This paper proposes FlatMatch, a semi-supervised learning method that bridges labeled and unlabeled data through cross-sharpness regularization, enhancing the model's generalization performance.
Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning
Zih-Yun Chiu (University of California), Michael C. Yip (University of California)
CodeReinforcement LearningSequential
π― What it does: The KGRL framework and KIAN model are proposed, enabling RL to utilize and integrate any external policies to achieve knowledge accessibility, sample efficiency, transferability, composability, and incremental learning.
π― What it does: This paper proposes Flow Factorized Representation Learning, which utilizes the gradient potential field learned in the latent space (dynamically optimal transport evolving over time) to generate decomposable transformation paths, achieving factorized modeling of input transformations.
Flow Matching for Scalable Simulation-Based Inference
Jonas Bernhard Wildberger, Bernhard SchΓΆlkopf (Max Planck Institute for Intelligent Systems)
CodeFlow-based ModelTime SeriesSequentialPhysics Related
π― What it does: A Bayesian posterior estimation method using continuous normalizing flows and flow matching (FMPE) is proposed, which can achieve scalable and directly assessable posterior distributions in simulation inference.
π― What it does: Proposes the FASTEN framework, which utilizes facial optical flow networks, flow attention, and spatiotemporal aggregation to achieve 3D mask defense;
π― What it does: A vehicle-infrastructure collaborative 3D detection framework FFNet based on feature flow is proposed, which utilizes feature flow to predict future features to compensate for temporal asynchrony and achieve low communication overhead.
Kunjal Panchal (University of Massachusetts), Hui Guan (University of Massachusetts)
CodeFederated LearningImageText
π― What it does: This paper proposes a method called Flow that simultaneously achieves personalized models for each client and each instance in federated learning, utilizing dynamic routing to determine whether to use local parameters or global parameters during inference.
π― What it does: Developed and implemented a constraint-based reinforcement learning method called FlowPG, which directly generates actions that satisfy constraints during the reinforcement learning process, avoiding traditional projection solutions.
π― What it does: A feature layer self-supervised learning method based on ViT, called FLSL, is designed, utilizing a dual-layer clustering framework that combines mean shift clustering and k-means, allowing image features to achieve semantic clustering at both local and global levels, significantly improving the performance of dense prediction tasks.
π― What it does: The FLuID framework is proposed, which dynamically identifies and discards 'invariant' neurons that contribute little to the global model through Invariant Dropout, thereby generating sub-models for slow devices (stragglers) to alleviate their computational and communication load;
FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space
Shengzhong Liu (Shanghai Jiao Tong University), Tarek Abdelzaher (University of Illinois at Urbana-Champaign)
CodeAnomaly DetectionRepresentation LearningTransformerContrastive LearningMultimodalityTime Series
π― What it does: This paper proposes a self-supervised contrastive learning framework named FOCAL, specifically designed to extract shared and private features from multimodal time series perception signals and achieve semantic representation through contrastive learning.
π― What it does: A query-centered few-shot semantic segmentation model AMFormer is proposed, which achieves precise segmentation of query images using adversarial mining Transformer with only rough support information.
π― What it does: For the few-shot image classification task, this paper designs a Position Prompt to guide the pre-trained visual Transformer to focus on the key entities in the image that are most relevant to the current category during the fine-tuning process, thereby improving classification performance.
Focused Transformer: Contrastive Training for Context Scaling
Szymon Tworkowski (IDEAS NCBR), Piotr MiΕoΕ (IDEAS NCBR)
CodeTransformerContrastive LearningText
π― What it does: A Focused Transformer (FOT) is proposed, which extends the effective context length of the model by training the attention layer through contrastive learning.
π― What it does: This paper proposes the SALE (State-Action Representation Learning) method, which enhances representation learning for low-dimensional continuous control tasks by learning state-action embeddings, and combines it with TD3 to form a new TD7 algorithm.
Samuel Dooley (Abacus.AI), Colin White (California Institute of Technology)
CodeTransformerTime Series
π― What it does: This paper proposes ForecastPFN, a zero-shot time series forecasting model pre-trained with synthetic data, capable of making predictions without further training.
ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning
Junguang Jiang (Tsinghua University), Mingsheng Long (Tsinghua University)
CodeDomain AdaptationOptimizationImage
π― What it does: The ForkMerge method is proposed to dynamically find the optimal auxiliary task weights through periodic forking and merging models, thereby alleviating the negative transfer problem in auxiliary task learning.
Foundation Model is Efficient Multimodal Multitask Model Selector
Fanqing Meng (Shanghai AI Laboratory), Ping Luo (University of Hong Kong)
CodeClassificationOptimizationComputational EfficiencyTransformerLarge Language ModelVision Language ModelImageTextMultimodality
π― What it does: This paper studies a model selection method called EMMS, which can quickly predict the performance of pre-trained models in multi-modal multi-task scenarios.
FouriDown: Factoring Down-Sampling into Shuffling and Superposing
Qi Zhu (University of Science and Technology of China), Feng Zhao (University of Science and Technology of China)
CodeRestorationConvolutional Neural NetworkImage
π― What it does: This paper proposes a general framework for downsampling in the Fourier domainβFouriDown, which can dynamically learn frequency weighting and overlapping mechanisms based on image context, addressing the bias and aliasing issues caused by traditional static weighting.
FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective
Kun Yi (Beijing Institute of Technology), Zhendong Niu (HeFei University of Technology)
CodeGraph Neural NetworkTime Series
π― What it does: Redefines the multivariate time series forecasting problem as a pure graph neural network task on a hyper-variable graph and designs FourierGNN for prediction.
π― What it does: A four-dimensional continuous representation of the hand based on Fourier query flow is proposed, utilizing RGB video to learn the spatiotemporal continuous reconstruction of hand shapes.
π― What it does: By performing contrastive learning and predictive pre-training between molecular graphs and fragment graphs, the representation ability of GNN for molecular structures is enhanced.
Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
Hanzhuo Huang (ShanghaiTech University), Sibei Yang (ShanghaiTech University)
CodeGenerationTransformerLarge Language ModelDiffusion modelVideoText
π― What it does: A zero-shot, no-training text-to-video generation process is proposed, utilizing a large language model (LLM) to generate temporal sequence prompts, and then employing a pre-trained latent diffusion model (LDM) to generate semantically coherent, temporally consistent, and high-quality videos using techniques such as joint noise sampling, step-aware attention transfer, and dual-path interpolation.
π― What it does: This paper utilizes generative models to synthesize a large number of high-quality synthetic images under semantic mask conditions, and significantly improves fully supervised semantic segmentation performance when combined with real data or pre-trained models.
π― What it does: Utilize frequency domain transformations (such as DCT) to parameterize data, select a small number of high-variance dimensions in the frequency domain, and optimize synthetic data, thereby achieving dataset distillation under a limited budget.
Frequency-Enhanced Data Augmentation for Vision-and-Language Navigation
Keji He (Chinese Academy of Sciences), Xinchao Wang (National University of Singapore)
CodeData SynthesisRetrievalVision Language ModelImageText
π― What it does: This paper proposes a frequency domain-based data augmentation method (FDA) that enhances the visual-text matching and navigation performance of visual-language navigation (VLN) models by mixing high-frequency components in the Fourier domain.
From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Models to Pre-trained Machine Reader
Weiwen Xu (Chinese University of Hong Kong), Lidong Bing (Alibaba Group)
CodeClassificationRetrievalTransformerLarge Language ModelSupervised Fine-TuningText
π― What it does: This paper proposes a model called PMR (Pre-trained Machine Reader), which transforms a pre-trained masked language model (MLM) into a machine reading comprehension (MRC) model. It significantly improves the performance of span extraction tasks through continuous pre-training using large-scale MRC-style training data automatically generated from Wikipedia hyperlinks.
π― What it does: This paper studies a high-fidelity audio decoder based on multi-band diffusion, which can restore audio symbols generated by low-bitrate discrete encoders into high-quality audio waveforms.
π― What it does: Trained an agent that uses only pixel screenshot input and a general mouse and keyboard action space, capable of performing tasks in GUI instruction-following tasks like MiniWob++ and surpassing human performance.
From ViT Features to Training-free Video Object Segmentation via Streaming-data Mixture Models
Roy Uziel (Ben Gurion University of the Negev), Oren Freifeld (Ben Gurion University of the Negev)
CodeObject DetectionSegmentationTransformerVideo
π― What it does: A novel unsupervised, non-training, low memory consumption semi-supervised video object segmentation method is proposed, utilizing pre-trained ViT features and a multi-scale vMF mixture model that can be updated on streaming data for object modeling, refined through pixel-level assignment and pixel-adaptive CRF.
Fully Dynamic $k$-Clustering in $\tilde O(k)$ Update Time
Sayan Bhattacharya (University of Warwick), Nikos Parotsidis (Google Research)
CodeOptimizationTabular
π― What it does: Designed and implemented an O(1) approximate fully dynamic k-median/k-means algorithm, with an amortized update time of Λ(O(k)) and a query time of Λ(O(k^2)).
π― What it does: A graph mixup data augmentation method based on the fused Gromov-Wasserstein (FGW) distance, called FGWMixup, is proposed. It aims to obtain the optimal node matching between two graphs through the optimal transport (OT) problem, thereby generating more representative mixed graphs for graph-level classification.
Future-Dependent Value-Based Off-Policy Evaluation in POMDPs
Masatoshi Uehara (Genentech), Wen Sun (Cornell University)
CodeReinforcement LearningSequential
π― What it does: This paper proposes a future-dependent value function to address the offline policy evaluation (OPE) problem in partially observable Markov decision processes (POMDPs), and provides the corresponding Bellman equation and minimax learning algorithm.
π― What it does: A variational autoencoder based on GΓ‘csβKΓΆrner common information (GKβVAE) is proposed, which can separate and quantify common and unique information from multi-view data.
π― What it does: Designed and implemented GAUCHE, a Gaussian process library specifically for chemistry, supporting various representations of molecules, reactions, and proteins, as well as their strings, fingerprints, and graph kernels, and integrated with GPyTorch/BoTorch;
Tobias Leemann (University of TΓΌbingen), Gjergji Kasneci (Technical University of Munich)
CodeSafty and PrivacyImageTabular
π― What it does: A new privacy concept called f-Membership Inference Privacy (f-MIP) is proposed, and its implementation is achieved through hypothesis testing analysis of the SGD training process.
π― What it does: A new Gaussian Mixture Solver (GMS) is proposed, which relaxes the assumption of Gaussian transition kernels in the reverse sampling of diffusion models and uses a Gaussian mixture model to approximate the true reverse transition distribution.
π― What it does: This paper proposes a detection framework based on Gaussian processes (Gaussian Process Probes, GPP), which measures the model's representation of concepts and its uncertainty by constructing the classifier distribution corresponding to the activation vectors of the model output;
π― What it does: A lightweight zero-cost proxy CRoZe is proposed to evaluate the robustness against various perturbations in the network initialization state and to quickly find general robust models in NAS search.
Generalized test utilities for long-tail performance in extreme multi-label classification
Erik Schultheis (Aalto University), Krzysztof Dembczynski (Yahoo! Research)
CodeClassificationOptimizationText
π― What it does: This paper addresses the long-tail label problem in extreme multi-label classification (XMLC) and proposes an inference method based on the Expected Test Utility (ETU) framework, which can directly optimize macro-average metrics (such as macro F1, macro recall, coverage, etc.) under a given budget k, thereby enhancing the recall ability for rare labels.
π― What it does: This paper proposes a general method for achieving path consistency (PC) in continuous decision-making environments with immediate rewards (such as Atari games), called GW-PCZero;
π― What it does: A general importance weighting method (GIW) is proposed and implemented, capable of addressing all four types of distribution shift problems with support transitions, and utilizes the validation set to achieve support partitioning during training.
π― What it does: Re-defines the sequential recommendation task as a learning task to generate an 'ideal item' (oracle item), and directly generates this ideal item based on the user's historical interactions using a guided diffusion model, then provides a recommendation list through nearest neighbor retrieval.
CodeGenerationRetrievalTransformerLarge Language ModelDiffusion modelImageTextMultimodalityRetrieval-Augmented Generation
π― What it does: In this paper, the authors propose the GILL (Generating Images with Large Language Models) framework, which combines a frozen text large language model (LLM) with a pre-trained image encoder/decoder through embedding mapping. This allows for the processing of any interleaved image and text inputs, and simultaneously generates text, retrieves images, and creates new images within the same model.
π― What it does: A generation model based on the semi-dual form of unbalanced optimal transport (UOT) called UOTM is proposed to replace traditional OT generation models.
π― What it does: A strategy network based on discrete regularization flow is designed, and IAR-A2C is proposed to achieve safe policy learning in constrained multi-dimensional discrete action spaces through invalid action rejection.
π― What it does: A generalizable neural surface reconstruction model GenS has been developed, capable of directly reconstructing 3D surfaces from multi-view images without the need for single-scene optimization.
π― What it does: Proposes Geodesic Multi-Modal Mixup (mΒ²-Mix) to fine-tune CLIP, generating hard negative samples and enhancing the uniformity and alignment of cross-modal representations, thereby achieving more robust multi-modal transfer.
Johann Brehmer (Qualcomm AI Research), Taco Cohen (Qualcomm AI Research)
CodeTransformerPoint CloudPhysics Related
π― What it does: This paper presents the Geometric Algebra Transformer (GATr), a general geometric data network that utilizes projective geometric algebra (G_{3,0,1}) for data representation and implements E(3) equivariance through a Transformer.
Geometric Transformer with Interatomic Positional Encoding
Yusong Wang (Xi'an Jiaotong University), Tie-Yan Liu (Microsoft Research)
CodeTransformerGraph
π― What it does: Proposes Geoformerβa geometric molecular modeling framework based on Transformer, which utilizes Interatomic Position Encoding (IPE) to capture the three-dimensional structure of molecules and predict molecular properties.
π― What it does: A fully differentiable variational Bayesian system, GeoPhy, is proposed, which can jointly infer the evolutionary tree topology and branch lengths without pre-selecting tree topologies.
π― What it does: A nonlinear influence function approximation method named GEX is proposed, utilizing geometric ensemble to eliminate the self-influence distribution bias caused by traditional bilinear approximation, thereby allowing for a more accurate assessment of the impact of training samples on model predictions during the post-training phase.
GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning
Haiteng Zhao (Peking University), Qi Liu (Peking University)
CodeDrug DiscoveryGraph Neural NetworkTransformerSupervised Fine-TuningTextGraphBiomedical Data
π― What it does: A unified graph-text Transformer model GIMLET is proposed, which utilizes natural language instructions to perform molecular property prediction tasks under zero-shot conditions.
π― What it does: A two-stage video question-answering model called Glance-Focus is proposed, which first generates event memories from long videos through unsupervised or supervised methods, and then uses these memories as prompts to quickly locate key segments related to the questions and infer answers.
Zeren Tan (Tsinghua University), Jian Li (Tsinghua University)
CodeExplainability and InterpretabilityConvolutional Neural NetworkTransformerImageText
π― What it does: This paper studies an improved explanation framework called GLIME, aimed at enhancing the interpretability of black-box model predictions.
π― What it does: A low-light image enhancement framework based on diffusion models is proposed, utilizing global structure-aware rank regularization and uncertainty-guided pixel-level regularization to reduce the curvature of ODE trajectories and enhance details and contrast.
Global-correlated 3D-decoupling Transformer for Clothed Avatar Reconstruction
Zechuan Zhang (Zhejiang University), Yi Yang (University of Technology Sydney)
CodeRestorationGenerationTransformerImage
π― What it does: A global context 3D decoupling network based on Transformer (GTA) is proposed, capable of reconstructing high-quality 3D models of clothed humans from a single image.
π― What it does: A novel non-autoregressive video generation framework called GLOBER is proposed, which first generates global features as global guidance and then synthesizes video frames through a non-autoregressive decoder, achieving global coherence and local realism.
GlyphControl: Glyph Conditional Control for Visual Text Generation
Yukang Yang (Princeton University), Kai Chen (Microsoft Research Asia)
CodeGenerationDiffusion modelImageText
π― What it does: Proposes the GlyphControl method, which implements controllable visual text generation by adding a glyph image-based ControlNet on Stable Diffusion.
π― What it does: A generalizable neural semantic field (GNeSF) is proposed, which integrates multi-view 2D semantics through a soft voting mechanism and utilizes disparity information to predict 3D semantics.
Goal Driven Discovery of Distributional Differences via Language Descriptions
Ruiqi Zhong (University of California), Jacob Steinhardt (University of California)
CodeTransformerLarge Language ModelSupervised Fine-TuningTextBenchmark
π― What it does: This paper proposes a system for automatically discovering and generating differences between two large corpora. The goal-driven D5 task aligns user-specified exploration objectives with text and outputs natural language predicate descriptions of the differences.
π― What it does: This paper proposes and validates the concept of Hierarchical Linear Feature Connectivity (LLFC), which states that along the linear interpolation path of two model parameters, the feature mappings of almost all layers maintain a linear relationship. It further explores the mechanism behind LLFC and its relationship with existing Permutation methods.
π― What it does: Through theoretical and experimental analysis, the expressive power of persistent homology (PH) under vertex color and edge color filtering in graph structure recognition is clarified, and a more powerful topological feature extraction method called RePHINE is proposed;
π― What it does: This paper presents Gold-YOLO, an improved real-time object detector based on the YOLO series, with the core improvements being the Gather-and-Distribute (GD) mechanism and MAE pre-training;
GPEX, A Framework For Interpreting Artificial Neural Networks
Amir Akbarnejad (University of Alberta), Nilanjan Ray (University of Alberta)
CodeExplainability and InterpretabilityKnowledge DistillationImage
π― What it does: This paper proposes the GPEX framework, which explains any feedforward artificial neural network (ANN) using Gaussian processes (GP), derives a new ELBO that globally matches the GP posterior with the ANN output, and implements knowledge distillation.
GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks
Zhonghang Li (South China University of Technology), Chao Huang (University of Hong Kong)
CodeGraph Neural NetworkAuto EncoderGraphTime Series
π― What it does: This paper proposes GPT-ST, a pre-training framework based on masked autoencoders to enhance the predictive performance of spatiotemporal graph neural networks.
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Rui Yang (Tsinghua University), Ying Shan (Tencent)
CodeImage TranslationGenerationRetrievalTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringImageTextMultimodalityRetrieval-Augmented Generation
π― What it does: This paper proposes GPT4Tools, which utilizes GPT-3.5 as a teacher to generate tool instruction data, and fine-tunes open-source LLMs (such as Vicuna, LLaMA, OPT) with LoRA to actively invoke multimodal tools in visual tasks.
π― What it does: This paper proposes a gradient flossing method that dynamically controls the Jacobian matrix of RNN gradients by regularizing the Lyapunov exponent, enhancing the stability of gradient propagation and training effectiveness.
Sanghyun Son (University of Maryland), Ming Lin (University of Maryland)
CodeOptimizationReinforcement Learning
π― What it does: This paper proposes a reinforcement learning method that combines the analytical gradient of differentiable environments with Proximal Policy Optimization (PPO) β GI-PPO. The core idea is to bridge the RP gradient and PPO updates through the Ξ±-policy and adaptively adjust Ξ± to control the gradient variance and bias.
Grammar Prompting for Domain-Specific Language Generation with Large Language Models
Bailin Wang (Massachusetts Institute of Technology), Yoon Kim (Massachusetts Institute of Technology)
CodeGenerationAI Code AssistantTransformerLarge Language ModelPrompt EngineeringTextChain-of-Thought
π― What it does: Proposes a grammar prompting method that allows LLMs to first predict a minimized BNF grammar subset with a small number of examples, and then generate corresponding DSL programs according to that grammar.
GRAND-SLAMINβ Interpretable Additive Modeling with Structural Constraints
Shibal Ibrahim (Massachusetts Institute of Technology), Rahul Mazumder (Massachusetts Institute of Technology)
CodeOptimizationExplainability and InterpretabilityTabular
π― What it does: The GRAND-SLAMIN framework is proposed, which can learn interpretable additive models (GAM) with sparse interactions and hierarchical constraints in an end-to-end manner.
Graph Convolutional Kernel Machine versus Graph Convolutional Networks
Zhihao Wu (Shenzhen Research Institute of Big Data), Jicong Fan (Chinese University of Hong Kong)
CodeClassificationOptimizationExplainability and InterpretabilityGraph Neural NetworkGraph
π― What it does: This paper proposes Graph Convolutional Kernel Machines (GCKM), which combines graph convolution with kernel methods for graph learning tasks.
Graph Denoising Diffusion for Inverse Protein Folding
Kai Yi (University of New South Wales), Yu Guang Wang (Shanghai Jiao Tong University)
CodeGenerationProtein Structure PredictionGraph Neural NetworkDiffusion modelGraphBiomedical Data
π― What it does: A graph denoising diffusion model (GRADE-IF) is proposed, which can generate diverse and foldable amino acid sequences under the condition of a given protein backbone.
GraphPatcher: Mitigating Degree Bias for Graph Neural Networks via Test-time Augmentation
Mingxuan Ju (University of Notre Dame), Yanfang Ye (University of Notre Dame)
CodeGraph Neural NetworkGraph
π― What it does: A testing-time data augmentation framework called GRAPHPATCHER is proposed, which alleviates the degree bias of graph neural networks by dynamically generating virtual neighbors around low-degree nodes to fill in their sparse neighborhoods.
Grounding Neural Inference with Satisfiability Modulo Theories
Zifan Wang (Center for AI Safety), Matt Fredrikson (Carnegie Mellon University)
CodeOptimizationTransformerImage
π― What it does: This paper proposes an SMTLayer, which embeds an SMT solver into a deep network as a pluggable layer, enabling forward solving of symbolic constraints and utilizing unsatisfied core or MaxSMT information for gradient updates during backpropagation, allowing direct use of symbolic knowledge during both training and inference phases.
Haris Aziz (University of New South Wales Sydney), Nisarg Shah (University of Toronto)
CodeTabularReview/Survey Paper
π― What it does: Proposes the introduction of the concept of 'core' in peer review to construct a review assignment scheme that satisfies fairness for all subgroups.
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningVision Language ModelTextMultimodality
π― What it does: Proposes the Adaptive Return-conditioned Policy (ARP), which calculates the visual and textual similarity as immediate rewards through a pre-trained multimodal encoder, and uses this reward to train a return-conditioned behavior cloning strategy to enhance generalization in unseen environments and unseen instructions.
Guiding Large Language Models via Directional Stimulus Prompting
Zekun Li (University of California Santa Barbara), Xifeng Yan (University of California Santa Barbara)
CodeReinforcement Learning from Human FeedbackTransformerLarge Language ModelSupervised Fine-TuningReinforcement LearningPrompt EngineeringText
π― What it does: A framework named Directional Stimulus Prompting (DSP) is proposed, which utilizes a small adjustable strategy model to generate instantiated 'directional stimulus' prompts, achieving fine-grained, instance-specific output guidance in black-box large language models (LLMs) without parameter tuning.
Guiding The Last Layer in Federated Learning with Pre-Trained Models
Gwen Legate (Concordia University), Eugene Belilovsky (Concordia University)
CodeFederated LearningImageText
π― What it does: In federated learning, an efficient recent class mean head initialization for FedNCM is proposed by utilizing a pre-trained model to fine-tune only the head layer, and further constructing a two-stage FedNCM+FT process.
π― What it does: A GUST model is designed to achieve unsupervised visual object grouping using neuronal coherence, and it can achieve compositional generalization in scenes with different numbers of objects.
π― What it does: Proposes H2RBox-v2, a weakly supervised detection framework that utilizes horizontal box annotations to learn rotation boxes through symmetry self-supervised learning.
π― What it does: This study investigates the model heterogeneity in visual recognition within federated learning, systematically evaluating the performance of 19 state-of-the-art visual architectures across 4 federated datasets.
Hardware Resilience Properties of Text-Guided Image Classifiers
Syed Talal Wasim (Mohamed bin Zayed University of AI), Gu-Yeon Wei (Harvard University)
CodeClassificationRecognitionTransformerLarge Language ModelPrompt EngineeringVision Language ModelImageTextMultimodality
π― What it does: By initializing the classification layer with rich text descriptions generated by GPT-3 during training and the output of the CLIP text encoder, the robustness of the image classification model to instantaneous bit flips in hardware during deployment is significantly improved.
π― What it does: A new Mixup training objective called Decoupled Mixup is proposed, which can mine discriminative features from hard mixed samples while maintaining smoothness.
Harnessing the power of choices in decision tree learning
Guy Blanc (Stanford University), Mo Tiwari (Stanford University)
CodeClassificationExplainability and InterpretabilityComputational EfficiencyTabular
π― What it does: This paper proposes a method to extend traditional greedy decision tree algorithms (such as ID3, C4.5, CART) to Topk, where at each node, instead of considering only the single best feature, it considers the k best features, thereby improving learning effectiveness while maintaining interpretability.
Muxi Chen (Chinese University of Hong Kong), Qiang Xu (Chinese University of Hong Kong)
CodeExplainability and InterpretabilityData-Centric LearningLarge Language ModelVision Language ModelDiffusion modelImage
π― What it does: An automated and interpretable model debugging framework called HiBug has been developed, which utilizes large pre-trained models to automatically generate task-related visual attributes, label images, and identify low-performance data slices to discover systematic errors in the model and conduct root cause analysis.
π― What it does: This paper proposes a Hierarchical Adaptive Value Estimation framework (HAVE) for multimodal visual reinforcement learning, which can dynamically estimate and allocate the contributions of different sensors such as RGB, event cameras, and depth, achieving collaborative decision-making with multimodal information.
π― What it does: This paper proposes HiDe-Prompt, which explicitly optimizes prompt parameters using hierarchical decomposition (within-task prediction, task-identity inference, task-adaptive prediction) to enhance continuous learning performance under self-supervised pre-training.
π― What it does: Hierarchically fuse the prior features generated by the latent diffusion model with the Transformer regression model for image deblurring.