🎯 What it does: A joint energy-based model (EBM) is proposed as a prior for multi-layer generative models, capturing intra-layer contextual relationships and inter-layer structures by jointly modeling the energies of all latent variables.
Learning Multi-Modal Class-Specific Tokens for Weakly Supervised Dense Object Localization
Lian Xu (University of Western Australia), Dan Xu (Hong Kong University of Science and Technology)
CodeObject DetectionTransformerVision Language ModelContrastive LearningImageMultimodality
🎯 What it does: This paper proposes a multimodal category-specific tagging framework based on Transformer, which jointly learns CLIP text and visual features to accomplish weakly supervised dense object localization tasks.
🎯 What it does: Learn video representations that can simultaneously encode action steps and their temporal order from a large number of tutorial videos and their voiceovers through unsupervised learning;
Learning Situation Hyper-Graphs for Video Question Answering
Aisha Urooj, Mubarak Shah (University of Central Florida)
CodeTransformerVideo
🎯 What it does: Using the Transformer architecture to learn contextual hypergraphs (including actions and entity relationships) in videos, and reasoning for video question answering tasks through this hypergraph.
🎯 What it does: A joint estimation of disparity and uncertainty deep stereo matching network called SEDNet is proposed, which enhances the accuracy of disparity and the reliability of uncertainty estimation by constraining the distributions of both using KL divergence.
🎯 What it does: This paper proposes a hierarchical prosody modeling network for movie dubbing tasks, which achieves high-quality, emotionally synchronized dubbing synthesis by mapping three types of visual information—lip movements, facial emotional expressions (emotional values and arousal in the emotional dimension), and scene atmosphere—onto the duration, energy, pitch, and emotion of speech.
Learning To Fuse Monocular and Multi-View Cues for Multi-Frame Depth Estimation in Dynamic Scenes
Rui Li (Northwestern Polytechnical University), Yanning Zhang (Northwestern Polytechnical University)
CodeDepth EstimationDomain AdaptationAutonomous DrivingConvolutional Neural NetworkSimultaneous Localization and MappingImage
🎯 What it does: A depth estimation framework that integrates monocular and multi-frame visual information is proposed, utilizing a cross-thread fusion module to achieve more accurate depth predictions in dynamic scenes.
Learning To Generate Image Embeddings With User-Level Differential Privacy
Zheng Xu (Google Research), H. Brendan McMahan (Google Research)
CodeFederated LearningSafty and PrivacyRepresentation LearningConvolutional Neural NetworkImage
🎯 What it does: This paper proposes an algorithm called DP-FedEmb, which utilizes user-level differential privacy (DP) to train image embedding models, focusing on addressing the balance between privacy and efficiency under large class spaces and large model parameter scales.
🎯 What it does: A Text-Grounded Contrastive Learning (TCL) framework is proposed, which utilizes data containing only image-text pairs and directly incorporates a text localization step into contrastive learning to learn region-text alignment, thereby achieving open-world semantic segmentation.
🎯 What it does: This paper proposes an unsupervised domain adaptation method that reduces geometric shifts in object detection by learning a set of homographies, thereby improving cross-domain performance.
Learning Visual Representations via Language-Guided Sampling
Mohamed El Banani (University of Michigan), Justin Johnson (University of Michigan)
CodeRepresentation LearningTransformerLarge Language ModelContrastive LearningImageText
🎯 What it does: Utilizing pre-trained language models to calculate the semantic similarity of image captions, selecting semantically similar image pairs to replace traditional image augmentation or visual nearest neighbors for contrastive learning training of visual representations.
🎯 What it does: Proposes the Semantic-Aware Virtual Contrastive (SAVC) framework, which introduces virtual classes for supervised contrastive learning during the base training phase of Few-Shot Class-Incremental Learning (FSCIL) to enhance inter-class separation while retaining sufficient space for subsequent new classes.
🎯 What it does: A self-supervised adversarial noise masking method (SANM) is proposed, which regularizes the network by adaptively generating masking regions based on label quality to prevent overfitting on data with noisy labels, and utilizes a self-supervised reconstruction task to provide noise-independent supervisory signals.
🎯 What it does: This paper proposes a low-parameter, low-label semi-supervised 3D LiDAR point cloud semantic segmentation framework called LiM3D, which can maintain or even improve segmentation accuracy in scenarios with scarce annotations.
🎯 What it does: This study proposes a method called LiDAR2Map for online construction of high-precision semantic maps using LiDAR, and enhances the semantic expression capability of LiDAR features through online camera information distillation.
🎯 What it does: A LiDAR-based gait recognition framework called LidarGait is proposed, and the first large-scale multimodal LiDAR gait benchmark dataset SUSTech1K is constructed.
🎯 What it does: By embedding the pre-trained 2D StyleGAN2 into a 3D NeRF, high-resolution synthetic training samples with precise 3D bounding boxes are generated.
🎯 What it does: A two-step video depth estimation framework is proposed, which first normalizes camera poses using optical flow and monocular depth estimation, and then obtains absolute depth through scale alignment and residual depth learning.
🎯 What it does: A LinK module based on a linear kernel generator is proposed for efficient computation of large convolution kernels in LiDAR 3D perception.
🎯 What it does: Lite DETR is proposed, a pluggable and efficient Transformer encoder that updates high and low-level feature interactions, significantly reducing the computational load of multi-scale features.
🎯 What it does: A lightweight CNN-Transformer hybrid architecture called Lite-Mono is proposed for self-supervised monocular depth estimation. The model is small in size and low in parameter count, yet achieves competitive accuracy.
🎯 What it does: This paper constructs a new facial image dataset, FH37K, containing 17 fine-grained facial beard attributes, and trains attribute classifiers on this dataset.
🎯 What it does: A multi-person long-term artistic swimming action quality assessment dataset LOGO has been constructed, and a GOAT module based on group graph convolution and attention fusion has been proposed.
🎯 What it does: This paper proposes a local-to-global cross-modal fusion network named LoGoNet, designed to fuse LiDAR point clouds and multi-camera images to enhance 3D object detection accuracy.
Long-Tailed Visual Recognition via Self-Heterogeneous Integration With Knowledge Excavation
Yan Jin (Xiamen University), Hanzi Wang (Xiamen University)
CodeClassificationRecognitionKnowledge DistillationMixture of ExpertsImage
🎯 What it does: This paper proposes a long-tail visual recognition framework called SHIKE based on mixed experts, which enhances the performance of tail classes by adaptively fusing features of different depths and mining knowledge.
Lookahead Diffusion Probabilistic Models for Refining Mean Estimation
Guoqiang Zhang (University of Technology Sydney), W. Bastiaan Kleijn (Victoria University of Wellington)
CodeGenerationData SynthesisDiffusion modelImage
🎯 What it does: A Lookahead Diffusion Probabilistic Model (LA-DPM) is proposed, which improves mean estimation by extrapolating the estimates of x for two consecutive steps during the reverse sampling process, thereby enhancing sampling quality.
🎯 What it does: A low-light image enhancement framework is proposed that simultaneously performs structural modeling and appearance enhancement, achieving clearer and more realistic enhancement effects through a structure-guided appearance enhancement module.
LP-DIF: Learning Local Pattern-Specific Deep Implicit Function for 3D Objects and Scenes
Meng Wang (Tsinghua University), Zhizhong Han (Wayne State University)
CodeGenerationData SynthesisPoint CloudMesh
🎯 What it does: This paper proposes LP-DIF, which reconstructs detail-rich geometries by partitioning 3D shapes into local regions and learning dedicated decoders for each pattern cluster.
🎯 What it does: This paper proposes a Long-Short Term Feature Enhancement Network (LSTFE-Net) that improves the performance of small object detection in videos by aligning short-term frames, selecting the most informative long-term frames, and aggregating multi-scale features.
🎯 What it does: The MAGE framework is proposed, unifying image generation and self-supervised representation learning, using variable mask ratios for pre-training semantic tokens generated by VQGAN;
🎯 What it does: This paper proposes MagicNet, a semi-supervised multi-organ CT segmentation method based on a teacher-student framework. The method enhances the segmentation quality of small organs by partitioning 3D CT voxels into N³ small cubes (magic-cubes), performing partition-recovery data augmentation across and within images, and utilizing cube-level local representations to fuse pseudo-labels within the same image branch.
Sungwoong Kim (Korea University), Jongmin Kim (Kakao Brain)
CodeGenerationData SynthesisTransformerVision Language ModelGenerative Adversarial NetworkImageTextMultimodality
🎯 What it does: This paper proposes a unified generative visual-language Transformer (MAGVLT) that can generate text from images, generate images from text, and even generate images and text simultaneously within a single model; it employs a non-autoregressive masking prediction mechanism to achieve parallel decoding and iterative optimization.
MAP: Multimodal Uncertainty-Aware Vision-Language Pre-Training Model
Yatai Ji (Tsinghua University), Yujiu Yang (Tencent)
CodeRecognitionRetrievalTransformerVision Language ModelImageTextMultimodality
🎯 What it does: A multi-modal uncertainty-aware pre-training framework MAP is proposed, which models visual and linguistic representations as multivariate Gaussian distributions using a Probability Distribution Encoder (PDE), and designs three distribution-based pre-training tasks.
Muhammad Uzair Khattak (Mohamed bin Zayed University of AI), Fahad Shahbaz Khan (Linköping University)
CodeClassificationRecognitionDomain AdaptationTransformerPrompt EngineeringVision Language ModelImageMultimodality
🎯 What it does: Proposed and implemented a method for simultaneous prompt learning for both the visual and language branches of CLIP (MaPLe), achieving efficient fine-tuning for downstream visual recognition tasks.
🎯 What it does: A weakly supervised infrared small target detection framework called LESPS is proposed, which can gradually evolve single-point labels into pixel-level target masks through intermediate predictions of the network.
🎯 What it does: Proposes the MarginMatch method, which combines consistency regularization and pseudo-labeling, and utilizes the dynamic changes of pseudo-margins during training to filter high-quality pseudo-labels to enhance semi-supervised learning effectiveness.
🎯 What it does: Self-supervised facial video representation learning is based on reconstructing densely occluded facial regions to learn general facial features;
🎯 What it does: In the task of multi-scan 3D point cloud semantic segmentation, a pluggable MarS3D module is proposed, which can add multi-scan perception capabilities to existing single-scan models and achieve joint prediction of semantic categories and motion states.
Mask DINO: Towards a Unified Transformer-Based Framework for Object Detection and Segmentation
Feng Li (Hong Kong University of Science and Technology), Heung-Yeung Shum (Hong Kong University of Science and Technology)
CodeObject DetectionSegmentationTransformerImage
🎯 What it does: Mask DINO achieves a unified framework for object detection and image segmentation (instance, panoptic, semantic) by adding a mask prediction branch to the DINO detection framework.
Xiaoyi Dong (University of Science and Technology of China), Nenghai Yu (Xiamen University)
CodeObject DetectionSegmentationRetrievalKnowledge DistillationRepresentation LearningTransformerVision Language ModelContrastive LearningImageTextMultimodality
🎯 What it does: This paper proposes MaskCLIP, which combines mask self-distillation and visual-language contrastive learning for pre-training, enhancing the transferability of visual models.
🎯 What it does: A sample-driven image translation framework named MATEBIT is proposed, which utilizes Masked and Adaptive Transformer to learn cross-domain correspondences and achieves local and global style control through contrastive style learning and a U-Net decoder.
🎯 What it does: This paper proposes the DMAE framework, which utilizes a teacher model pre-trained with MAE to achieve efficient knowledge distillation during the self-supervised phase by aligning intermediate features.
Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers
Bin Ren (University of Trento), Wei Wang (Beijing Jiaotong University)
CodeClassificationSafty and PrivacyTransformerImage
🎯 What it does: This paper studies the positional embeddings of visual Transformers, demonstrating their explicit learning of two-dimensional spatial relationships and leading to privacy leakage, and proposes the Masked Jigsaw Puzzle (MJP) positional embedding scheme.
🎯 What it does: This paper proposes a self-supervised video representation learning method that utilizes Masked Motion Encoding (MME) to learn video features by recovering motion trajectories.
🎯 What it does: This paper proposes an unsupervised 3D representation learning framework MSC based on scene-level point clouds, combining contrastive learning and masked point modeling for pre-training.
🎯 What it does: A Masked Video Distillation (MVD) framework is proposed, which first uses MAE/VideoMAE pre-trained image or video models as teachers, and then uses the high-level features output by these teachers as the target for mask feature prediction to train a student Vision Transformer to learn more semantically meaningful spatiotemporal representations.
🎯 What it does: This paper proposes a scheme for sparsifying and compressing grid-based neural fields using multi-level wavelet transforms and learnable masks, balancing rendering quality and storage efficiency.
🎯 What it does: A mutual correction framework MCF is proposed for semi-supervised medical image segmentation, utilizing two different structured sub-networks to correct deviations from each other.
🎯 What it does: A unified Meta architecture called PointMeta is proposed, which abstracts the building blocks of existing point cloud networks into four meta-functions: neighbor update, aggregation, point update, and position embedding, and based on this, an efficient PointMetaBase module is introduced.
Meta-Learning With a Geometry-Adaptive Preconditioner
Suhyun Kang (Seoul National University), Wonjong Rhee (Seoul National University)
CodeOptimizationMeta LearningImage
🎯 What it does: This paper studies preconditioned gradient descent under the MAML framework, proposing the Geometry-Adaptive Preconditioner (GAP) and its low-computation approximation, Approximate GAP. These methods can adaptively utilize task-specific and path-dependent preconditioner matrices in the inner loop while satisfying Riemannian metrics, thereby enhancing meta-learning performance.
🎯 What it does: A framework for joint learning of infrared-visible image fusion and object detection, called MetaFusion, is proposed, which achieves mutual promotion of the two tasks through meta-feature embedding.
🎯 What it does: A methane plume detection network called MethaneMapper based on hyperspectral Transformer has been designed and implemented, capable of end-to-end detection and quantification of methane emissions, and a large methane hotspot dataset MHS has been released.
🎯 What it does: This paper proposes a Masked Image Consistency (MIC) module that enhances the learning of contextual relationships in the target domain by randomly masking patches in target domain images and ensuring the network remains consistent with the pseudo-labels of the complete images, thereby improving unsupervised domain adaptation performance.
Mind the Label Shift of Augmentation-Based Graph OOD Generalization
Junchi Yu (Chinese Academy of Sciences), Ran He (Chinese Academy of Sciences)
CodeGraph Neural NetworkGraph
🎯 What it does: This study investigates the label shift problem in out-of-distribution generalization using graph neural networks (GNNs) and proposes a label-invariant subgraph augmentation method called LiSA based on a variable subgraph generator, which generates diverse environments to learn invariant GNNs.
🎯 What it does: This paper proposes a Flat Trajectory Distillation (FTD) method aimed at reducing the cumulative trajectory error during the dataset distillation process, thereby improving the performance of the synthetic dataset in actual training.
MISC210K: A Large-Scale Dataset for Multi-Instance Semantic Correspondence
Yixuan Sun (Fudan University), Wenqiang Zhang (Fudan University)
CodeObject DetectionSegmentationTransformerImage
🎯 What it does: This paper proposes a multi-instance semantic correspondence task and constructs the MISC210K dataset, which consists of 218K image pairs and 34 categories. A dual-path collaborative learning framework (instance-level co-segmentation and keypoint matching) is designed to address this task.
CodeTransformerMixture of ExpertsVision Language ModelMultimodality
🎯 What it does: This paper proposes MixPHM, a redundancy-aware parameter-efficient fine-tuning method for optimizing large-scale vision-language models (VLM) in low-resource visual question answering (VQA) tasks, achieving better results than full fine-tuning.
🎯 What it does: The MMANet framework is proposed, which jointly achieves robust learning for missing modalities through a teacher network, a deployment network, and a regularization network, and enhances model performance through Marginal-Aware Distillation (MAD) and Modality-Aware Regularization (MAR).
🎯 What it does: This study proposes the problem of multimodal generalization (MMG) for first-person action recognition and constructs the MMG-Ego4D dataset and benchmark, which includes video, audio, and IMU three modalities.
Model Barrier: A Compact Un-Transferable Isolation Domain for Model Intellectual Property Protection
Lianyu Wang (Institute of High Performance Computing), Huazhu Fu (Nanjing University of Aeronautics and Astronautics)
CodeDomain AdaptationSafty and PrivacyConvolutional Neural NetworkGenerative Adversarial NetworkImage
🎯 What it does: This paper proposes the Compact Un-Transferable Isolation Domain (CUTI-domain), which enhances private style features within the authorized domain and constructs similar isolation domains to limit the model's transfer and recognition capabilities in unauthorized domains, thereby achieving model IP protection.
🎯 What it does: Proposed the ESP framework to achieve end-to-end visual information extraction, unifying entity localization, annotation, and linking, and released the challenging SIBR dataset;
🎯 What it does: This paper proposes a single-stage learning framework designed specifically for the task of Novel Class Discovery, based on symmetric KL divergence with cross-class and internal consistency constraints;
🎯 What it does: A modular short-term video memorability prediction model M3-S has been constructed, utilizing four modules: low-level visual features, scene segmentation, action recognition, and contextual similarity to obtain hierarchical memory features, which are then fused for regression prediction.
🎯 What it does: This paper proposes a two-stage MOSO framework, which first uses MOSO-VQVAE to decompose videos into three types of discrete tokens: motion, scene, and object, and then uses MOSO-Transformer to predict future videos at the token level.
🎯 What it does: This paper proposes MotionTrack, an online multi-object tracking framework that uses an Interaction Module to handle target interactions in dense crowds and a Refind Module to re-identify long-term occluded targets through historical trajectories.
🎯 What it does: The paper presents MOTRv2, an end-to-end multi-object tracking method that combines a pre-trained YOLOX detector with the MOTR framework.
MP-Former: Mask-Piloted Transformer for Image Segmentation
Hao Zhang (Hong Kong University of Science and Technology), Lei Zhang (Hong Kong University of Science and Technology)
CodeSegmentationTransformerImage
🎯 What it does: A Mask-Piloted Transformer (MP-Former) is proposed to improve the mask attention of Mask2Former, achieving more consistent multi-layer mask predictions and enhancing instance, semantic, and panoptic segmentation performance.
🎯 What it does: Proposes an efficient 3D object detection framework that generates proposals only in the current frame and propagates proposals along the time axis to sample point clouds.
🎯 What it does: A dual-contrast mechanism-based neural architecture search method is proposed for the object re-identification task, automatically designing a lightweight network called MSINet.
🎯 What it does: A multi-scale LiDAR-Camera fusion framework called MSMDFusion is proposed for 3D object detection, utilizing virtual points to project image semantics into 3D space.
Jasdeep Singh (Indian Institute of Technology Ropar), G. Sankara Raju Kosuru (Indian Institute of Technology Ropar)
CodeVideo
🎯 What it does: This paper proposes a multi-domain lightweight network that achieves video motion magnification through a combination of frequency domain phase-amplitude transformation and spatial domain multi-scale texture correction.
🎯 What it does: This paper proposes a multi-level aligned logit distillation method, which achieves knowledge distillation by aligning predictions at the instance, batch, and class levels, as well as enhancing predictions through temperature calibration, using only the logit outputs from the teacher model.
🎯 What it does: This paper studies the problem of missing modalities in multimodal learning and proposes a shared-specific feature modeling framework called ShaSpec, which can simultaneously address various combinations of missing modalities during both training and testing phases.
🎯 What it does: This paper proposes a method for reconstructing 3D shapes using only calibrated multi-view azimuth maps—Multi-View Azimuth Stereo (MVAS).
Multimodal Prompting With Missing Modalities for Visual Recognition
Yi-Lun Lee (National Yang Ming Chiao Tung University), Chen-Yu Lee (Google)
CodeClassificationRecognitionTransformerPrompt EngineeringVision Language ModelImageTextMultimodality
🎯 What it does: This study addresses the issues of missing modalities and the cost of fine-tuning large models in multimodal learning, proposing a missing-aware prompts approach that enhances performance under various missing conditions by training less than 1% of the parameters on the ViLT pre-trained multimodal Transformer.
🎯 What it does: This paper proposes a Neural Radiance Feature Field (NRFF) that achieves high-quality view synthesis through multi-scale tensor decomposition and rendering equation feature encoding.
🎯 What it does: A self-supervised pre-training method based on LiDAR point clouds, MV-JAR, is proposed, and a new data-efficient evaluation benchmark is established on the Waymo dataset.
N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution
Haram Choi (Sogang University), Jihoon Yang (Sogang University)
CodeRestorationSuper ResolutionTransformerImage
🎯 What it does: Proposes an N-Gram context mechanism and constructs lightweight image super-resolution networks such as NGswin and SwinIR-NG based on this, addressing the issues of limited receptive fields and high computational costs of window self-attention.
🎯 What it does: A Transformer-based neural network architecture representation model called NAR-Former is proposed, which can encode network topology and operation information into sequences and generate a unified vector representation for predicting attributes such as model accuracy and latency.
🎯 What it does: Using natural language semantics to assist sign language recognition, we propose language-aware label smoothing and cross-modal mixup techniques, and design a video-keypoint network to enhance sign language classification performance.
🎯 What it does: The NeAT framework is proposed, utilizing multi-view images and neural implicit functions (SDF + validity branch) to achieve arbitrary topology surface reconstruction, and enabling end-to-end training through differentiable volumetric rendering.
🎯 What it does: Proposes Neighborhood Attention (NA) and the Neighborhood Attention Transformer (NAT) based on NA, and implements an efficient CUDA/C++ extension called NATTEN.
🎯 What it does: This paper proposes NeRF-DS, a neural radiance field model for dynamic reflective objects, capable of reconstructing and rendering high-quality novel view images from monocular RGB videos.
🎯 What it does: By first extracting a sparse subnetwork from a dense network and gradually expanding the width or depth during the training process, dynamic network expansion is achieved, significantly accelerating training.
CodePose EstimationCompressionTransformerSupervised Fine-TuningSimultaneous Localization and MappingImage
🎯 What it does: An end-to-end neural coordinate mapping (NeuMap) method is designed, utilizing learnable spatial codes and a Transformer self-decoder to perform cross-attention on image features, thereby regressing 3D scene coordinates and achieving camera localization within a minimal storage space.
🎯 What it does: This study investigates the phenomenon of neural dependence that occurs in large-scale image classification models, where the logits of certain categories can be obtained by a linear combination of a few other categories.
🎯 What it does: A neural Fourier filter bank has been designed and implemented, capable of performing spatial partitioning and frequency encoding simultaneously on a multi-scale grid, thereby achieving efficient and high-quality 2D image fitting, 3D shape reconstruction, and NeRF view synthesis.
🎯 What it does: A neural rate estimator based on variational autoencoders is proposed, utilizing a low-complexity bottleneck layer for unsupervised training in the Split-DNN model, achieving efficient image feature compression and distributed inference.
🎯 What it does: In example texture synthesis, the CNN-MRF model is improved by proposing the Guided Correspondence distance and loss, achieving higher quality texture generation.
Neural Transformation Fields for Arbitrary-Styled Font Generation
Bin Fu (Shenzhen Institute of Advanced Technology), Yu Qiao (Shenzhen Institute of Advanced Technology)
CodeGenerationGenerative Adversarial NetworkImage
🎯 What it does: A few-shot font generation model based on Neural Transformation Fields (NTF) is proposed, viewing font generation as a continuous transformation process of pixel creation and dissipation.
Jiahao Li (Microsoft Research), Yan Lu (Microsoft Research)
CodeCompressionOptical FlowVideo
🎯 What it does: A new neural video encoder (DCVC-DC) has been designed within the deep contextual video compression framework to enhance coding efficiency through hierarchical quality structure, cross-group offset diversification, and quadtree partition entropy model.
🎯 What it does: A 3D scene generation framework based on a hierarchical latent diffusion model is designed, capable of automatically generating high-quality open-world 3D scenes from multi-view images and depth data, and supports various post-processing such as conditional generation, scene editing, and style transfer.