π― What it does: This paper proposes a framework for audio-driven co-speech gesture generation based on diffusion models, called DiffGesture, which can generate highly relevant and temporally coherent full-body pose sequences without using text or speaker identity.
Jun Nagata (DENSO IT LAB INC), Yusuke Sekikawa (DENSO IT LAB INC)
CodeAutonomous DrivingOptical FlowImageVideo
π― What it does: This paper proposes an incremental full optical flow estimation method TEGBP based on event cameras, which infers complete optical flow from sparse normal flow.
CodeGenerationData SynthesisVision Language ModelGenerative Adversarial NetworkPoint CloudMesh
π― What it does: A 3D texture shape generation model named TAPS3D was trained, capable of generating high-quality, controllable three-dimensional objects in one go based on given text prompts.
TarViS: A Unified Approach for Target-Based Video Segmentation
Ali Athar (RWTH Aachen University), Bastian Leibe (RWTH Aachen University)
CodeObject TrackingSegmentationTransformerVideo
π― What it does: This paper proposes TarViS, a unified Transformer-based network that can perform video instance segmentation, video panoptic segmentation, video object segmentation, and point tracking within the same model.
Task Difficulty Aware Parameter Allocation & Regularization for Lifelong Learning
Wenjin Wang (Zhejiang University), Yin Zhang (Zhejiang University)
CodeKnowledge DistillationNeural Architecture SearchMixture of ExpertsImage
π― What it does: A lifelong learning framework called PAR is proposed, which can dynamically select parameter allocation or parameter regularization strategies based on the learning difficulty of tasks.
Tao Yu (National University of Singapore), Xinchao Wang (National University of Singapore)
CodeClassificationDomain AdaptationTransformerVision Language ModelContrastive LearningImageMultimodality
π― What it does: Proposes Task Residual Tuning (TaskRes), which adds learnable residual parameters to the CLIP pre-trained text classifier while keeping it unchanged to achieve efficient transfer learning.
Task-Specific Fine-Tuning via Variational Information Bottleneck for Weakly-Supervised Pathology Whole Slide Image Classification
Honglin Li (Zhejiang University), Lin Yang (Westlake University)
CodeClassificationDomain AdaptationComputational EfficiencyTransformerContrastive LearningImageBiomedical Data
π― What it does: A task-specific fine-tuning framework based on information bottleneck is proposed, which significantly improves weakly supervised whole slide image classification performance by utilizing multi-instance learning and SSL pre-trained features.
π― What it does: This paper proposes a vision-based joint perception and prediction framework called TBP-Former, which utilizes a PoseSync BEV encoder and a spatial-temporal pyramid Transformer to detect, segment, and predict future trajectories of targets such as vehicles and pedestrians in synchronized BEV space.
Teaching Structured Vision & Language Concepts to Vision & Language Models
Sivan Doveh (IBM Research), Leonid Karlinsky (MIT-IBM Watson AI Lab)
CodeClassificationData-Centric LearningTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelContrastive LearningTextMultimodality
π― What it does: By processing the existing visual-text pair data, structured visual-language concept (SVLC) positive and negative samples are generated using methods such as rules, LLM, and analogy, supplemented by additional loss training to enhance the VL model's understanding of object attributes, relationships, and states of SVLC.
π― What it does: This paper proposes the Text-Guided Video Completion (TVC) task, which can generate complete video sequences based on given starting frames, ending frames, or frames from both ends, along with natural language descriptions.
π― What it does: This paper proposes a method for self-supervised representation learning that utilizes vehicle motion to obtain LiDAR views of the same object at different times, training the network to learn time-consistent and robust point cloud features against object dynamics.
π― What it does: A self-learning method called TeSLA is proposed, which utilizes automatic adversarial augmentation for online adaptation of pre-trained models on unlabeled streaming test data.
Text With Knowledge Graph Augmented Transformer for Video Captioning
Xin Gu (University of Chinese Academy of Sciences), Longyin Wen (ByteDance Inc.)
CodeGenerationTransformerVideoText
π― What it does: This paper proposes TextKG, a dual-stream Transformer model enhanced by knowledge graphs, aimed at generating more accurate video subtitles.
Texts as Images in Prompt Tuning for Multi-Label Image Recognition
Zixian Guo (Harbin Institute of Technology), Wangmeng Zuo (Harbin Institute of Technology)
CodeClassificationRecognitionPrompt EngineeringVision Language ModelImageText
π― What it does: This paper proposes treating text descriptions as images for prompt tuning (TaI prompting) and introduces dual-granularity prompt tuning (TaI-DPT) in multi-label image recognition to simultaneously leverage global and local features.
π― What it does: A new framework A5 (Adversarial Augmentation to Defend Against Adversarial Attacks) is proposed, which is the first certified preventive defense method against adversarial attacks, aimed at ensuring that any attack will fail by constructing defensive perturbations.
π― What it does: This paper studies and implements an attack method called EfficFrog that can implant an 'efficiency backdoor' in dynamic neural networks. The attacker can cause the victim model to consume more computational resources when triggered by specific inputs using a small amount of training data, leading to a decrease in system availability.
π― What it does: This paper proposes a Weakly Supervised Instance Segmentation (WSSIS) framework that utilizes point labels with just one pixel as a weak supervision source and refines coarse masks using MaskRefineNet.
The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training
Gi-Cheon Kang (Seoul National University), Byoung-Tak Zhang (Seoul National University)
CodeGenerationData SynthesisRetrievalTransformerVision Language ModelGenerative Adversarial NetworkImageMultimodality
π― What it does: Proposes Generative Self-Training (GST) to expand the VisDial training set by generating multi-turn visual dialogue data on unlabeled images from the web.
Thermal Spread Functions (TSF): Physics-Guided Material Classification
Aniket Dashpute (Rice University), Oliver Cossairt (University of Arizona)
CodeClassificationRecognitionImagePhysics Related
π― What it does: Using low-power laser heating and recording the thermal diffusion process of objects with a thermal camera, the thermal diffusion function (TSF) is extracted. By solving the inverse heat conduction equation, the thermal diffusivity and emissivity are obtained, which are then used as features for material classification, ultimately achieving an accuracy of about 86% in multi-class recognition.
π― What it does: This paper proposes the ThinkTwice end-to-end autonomous driving framework, which implements a scalable decision network through a multi-layer coarse-to-fine prediction decoder (Look, Predict, Refine).
π― What it does: Proposes the Tree-structured Implicit Neural Compression (TINC) framework, which first divides large-scale data into blocks and uses MLP for local INR representation, then shares parameters in a tree hierarchical structure to enhance compression quality.
π― What it does: Knowledge is transferred from a large-scale Masked Image Modeling (MIM) pre-trained model to a small Vision Transformer (ViT) through knowledge distillation, thereby enhancing the performance of the small model.
π― What it does: During the inference phase, online adaptation for the pre-trained model is performed, proposing an unsupervised input transformation invariance regularization objective to prevent model collapse under small batch sizes.
π― What it does: By designing two contrastive modules, Patch Token Contrast (PTC) and Class Token Contrast (CTC), in the Vision Transformer, the issue of over-smoothing produced by ViT is addressed, thereby improving the weakly supervised semantic segmentation performance using only image-level labels.
π― What it does: Proposes the Token Turing Machine (TTM), an autoregressive Transformer with external memory, designed for efficiently handling long-sequence visual understanding tasks;
TokenHPE: Learning Orientation Tokens for Efficient Head Pose Estimation via Transformers
Cheng Zhang (Central China Normal University), Youfu Li (City University of Hong Kong)
CodePose EstimationTransformerImage
π― What it does: A TokenHPE method based on Transformer is proposed, which achieves head pose estimation by learning the relationships of key facial parts, showing excellent performance especially in extreme poses and occlusion scenarios.
CodeRestorationSuper ResolutionOptimizationExplainability and InterpretabilityComputational EfficiencyAuto EncoderImage
π― What it does: A lightweight hyperspectral image super-resolution framework based on coordinated optimization is proposed, utilizing explicit degradation estimation and a sparse mixture prior autoencoder to recover high spatial resolution HSI.
π― What it does: This paper conducts a systematic investigation and summarizes the current state of human evaluation in the text-to-image generation field, finding a lack of standardization and poor reproducibility. Subsequently, a unified evaluation protocol based on absolute scoring is designed and validated, along with the public release of implementations, templates, and evaluation data.
π― What it does: This paper presents a large-scale aesthetic assessment dataset for artistic images, BAID (60,337 artworks, 360,000+ votes), and designs the Style-specific Art Assessment Network (SAAN) model for automatic evaluation of the aesthetic quality of artistic images.
π― What it does: This paper proposes a source-agnostic semantic segmentation model adaptation method called DT-ST, which implements online self-training through dynamic teacher updates and training consistency resampling.
π― What it does: An improved Joint Energy-based Model (JEM) called SADA-JEM is proposed, which utilizes a single network to simultaneously perform image classification and generation.
Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration
Kemal Oksuz (Five AI Ltd), Puneet K. Dokania (Five AI Ltd)
CodeObject DetectionDomain AdaptationImage
π― What it does: This paper proposes the Self-Aware Object Detection (SAOD) task, which requires detectors to have reliable uncertainty estimation, good calibration, and robustness to domain transfer in safety-critical scenarios.
π― What it does: This study designs and implements an adversarial texture mesh (AT3D) that can be 3D printed in real environments for physical black-box attacks on facial recognition systems while circumventing multimodal anti-spoofing mechanisms.
π― What it does: A partial label learning framework named PaPi is proposed, eliminating the contrastive learning module and using prototype alignment to guide the linear classifier's self-teaching, significantly improving representation learning and label disambiguation performance.
π― What it does: A framework named Iterative Multi-Scale Feature Aggregation (IMFA) is proposed to improve the efficiency and effectiveness of Transformer-based object detectors in utilizing multi-scale features.
π― What it does: This paper studies a modality-agnostic person re-identification framework called UNIReID, which can simultaneously handle text, sketches, and joint queries of both, addressing the issue of modality uncertainty.
Hyojun Go (Riiid AI Research), Seungtaek Choi (Riiid AI Research)
CodeSegmentationGenerationKnowledge DistillationMixture of ExpertsDiffusion modelImage
π― What it does: The PPAP framework is proposed, utilizing multiple experts, parameter-efficient fine-tuning, and unsupervised knowledge transfer, allowing publicly available offline models to achieve conditional guidance in diffusion models in a plug-and-play manner.
π― What it does: An Adaptive Consistency Regularization (ACR) framework is proposed, utilizing a dual-branch network and dynamic logit adjustment to address the issue of unknown class distribution of unlabeled data in long-tail semi-supervised learning.
π― What it does: A document image tampering text detection framework based on visual and frequency domain dual modalities is proposedβDocument Tampering Detector (DTD). It introduces a technical solution utilizing Frequency Perception Head (FPH), Multi-view Iterative Decoder (MID), and Adaptive Compression Curriculum Learning (CLTD) to enhance detection accuracy and robustness. Additionally, a new tampered text dataset, DocTamper, consisting of 170,000 images, has been constructed.
π― What it does: A method for converting unbiased perspective SDF to voxel rendering is proposed, combined with MVS point cloud supervision to achieve mask-free neural implicit surface reconstruction.
Towards Unified Scene Text Spotting Based on Sequence Generation
Taeho Kil (Naver Cloud), Daehee Kim (Naver Cloud)
CodeRecognitionObject DetectionTransformerMixture of ExpertsImageText
π― What it does: A unified scene text detection and recognition model UNITS based on sequence generation is proposed, capable of handling four detection formats: center points, bounding boxes, quadrilaterals, and polygons within a single model, and achieving ultra-long sequence inference through starting point prompts.
Towards Universal Fake Image Detectors That Generalize Across Generative Models
Utkarsh Ojha (University of Wisconsin Madison), Yong Jae Lee (University of Wisconsin Madison)
CodeClassificationData SynthesisAnomaly DetectionTransformerVision Language ModelGenerative Adversarial NetworkContrastive LearningImage
π― What it does: This paper studies the problem of detecting fake images generated by generative models and finds that traditional binary classifiers have poor generalization ability across models.
π― What it does: We propose TRACE, a single-stage network based on 5D representation that can simultaneously regress the 3D poses, shapes, and global trajectories of multiple people under dynamic cameras.
π― What it does: A method called TAR3 is proposed, which can adaptively pre-train models by retrieving external unlabeled samples during both training and inference.
π― What it does: A trainable projection gradient method (TPGM) is proposed, which automatically learns the distance constraints between each layer and the pre-trained model during the fine-tuning process, thereby improving OOD robustness while maintaining ID performance.
Erik GΓ€rtner (Google Research), Cristian Sminchisescu (Google Research)
CodePose EstimationOptimizationTransformerTabularTime Series
π― What it does: This paper proposes a Transformer-based learning optimizer called Optimus, which utilizes neural networks to learn each step's updates and precondition matrices, achieving more efficient iterative optimization.
π― What it does: Constructed the LipRF framework, utilizing the appearance representation of the pre-trained NeRF and the mapping of Lipschitz MLP to achieve cross-view consistency and lighting realism in 3D scene stylization.
Trap Attention: Monocular Depth Estimation With Manual Traps
Chao Ning (Northwestern Polytechnical University), Hongping Gan (Northwestern Polytechnical University)
CodeDepth EstimationTransformerImage
π― What it does: This paper proposes a new Trap Attention mechanism that combines depthwise separable convolutions and manually defined trap functions, enabling global feature interaction while maintaining linear complexity, thus achieving monocular depth estimation.
π― What it does: This paper proposes a method for crown instance segmentation and counting based on RGB sequences collected by UAVs. It first performs over-segmentation on the sequences and constructs contour maps, then uses Graph Convolutional Networks (GCN) to learn contour features and complete crown merging, resulting in accurate instance masks and tree counts.
π― What it does: This paper proposes a Three-View (TPV) representation and the TPVFormer transformer model, which predicts 3D semantic occupancy volumes using only camera input combined with sparse LiDAR annotations, achieving complete spatial semantic occupancy prediction.
π― What it does: This paper proposes a one-stage temporal action detection framework called TriDet, which combines the Trident-head and SGP layer to model the relative probability distribution of action boundaries and achieve scalable granularity perception in the feature pyramid.
π― What it does: The paper proposes a Trojan attack method for diffusion models called TrojDiff, which can inject triggers during training, allowing the model to generate preset attack targets when receiving trigger noise.
π― What it does: A tunable convolution layer is proposed that can adjust the behavior of neural networks through interactive parameters during inference, combined with a parameterized multi-objective loss to achieve multi-objective control.
Turning Strengths Into Weaknesses: A Certified Robustness Inspired Attack Framework Against Graph Neural Networks
Binghui Wang (Illinois Institute of Technology), Yun Dong (Nanchang University)
CodeAdversarial AttackGraph Neural NetworkGraph
π― What it does: A robust attack framework based on authentication has been designed, utilizing the authentication perturbation size of nodes to guide graph structure attacks, thereby enhancing the effectiveness of existing attack methods.
π― What it does: Proposes the Twin Contrastive Learning (TCL) model, which combines contrastive learning with Gaussian Mixture Models (GMM) to learn robust representations from data with noisy labels and automatically correct the labels.
Kun Yan (Peking University), Yan Lu (Microsoft Research Asia)
CodeObject DetectionSegmentationVideo
π― What it does: This paper proposes a two-frame video object segmentation (Two-Shot VOS) training framework that uses annotations from only two frames of each video, leveraging pseudo-labels and semi-supervised learning to enhance model performance.
π― What it does: This paper proposes the ULIP framework, which aligns 3D point cloud features using the pre-trained CLIP image-text feature space to construct a unified representation of language, images, and point clouds in three modalities.
π― What it does: The TEMPURA framework is proposed for unbiased dynamic scene graph generation, addressing issues such as long-tail distribution, noise, and temporal inconsistency.
π― What it does: This paper addresses the Semantically Coherent Out-of-Distribution (SCOOD) detection task and proposes an energy-based uncertainty-aware optimal transport (ET) scheme. By assigning semantically consistent labels to unlabeled data and combining a cross-set expansion strategy (Lrep) to enhance semantic discrimination, it achieves more accurate OOD detection.
π― What it does: This study investigates the intrinsic attribute decoupling capability of Stable Diffusion and proposes that optimizing only the mixed weights of two text embeddings can achieve various image editing tasks without fine-tuning the model.
π― What it does: A unified GC-VRNN framework is proposed, capable of simultaneously performing missing value imputation for multi-agent trajectories and future trajectory prediction.
Uncurated Image-Text Datasets: Shedding Light on Demographic Bias
Noa Garcia (Osaka University), Yuta Nakashima (Osaka University)
CodeObject DetectionGenerationRetrievalTransformerVision Language ModelDiffusion modelImageTextMultimodality
π― What it does: This study constructs PHASE annotations containing six perceptual attributes: age, gender, skin color, race, emotion, and activity on the large-scale unfiltered image-text dataset GCC, and evaluates the social biases and amplification phenomena in image description, CLIP embedding, and text-to-image generation tasks based on this.
π― What it does: This paper studies the geometric meaning of the feature functions learned in deep functional maps and proposes two simple improvement methods to enhance the accuracy of shape correspondence.
Understanding and Improving Visual Prompting: A Label-Mapping Perspective
Aochuan Chen (Michigan State University), Sijia Liu (MIT-IBM Watson AI Lab)
CodeClassificationOptimizationExplainability and InterpretabilityPrompt EngineeringContrastive LearningImage
π― What it does: A visual prompting framework based on label mapping (LM) called ILM-VP is proposed, which can improve the accuracy of the target task and enhance interpretability through co-learning of LM and visual prompts via iterative optimization, without fine-tuning the source model.
π― What it does: This paper studies the geometric structure of the last layer feature centers and classifiers in semantic segmentation. It finds that, unlike the neural collapse in image classification, semantic segmentation loses isometric symmetry in feature centers and classifiers due to contextual relevance and class imbalance, leading to poor performance in minority classes.
Unicode Analogies: An Anti-Objectivist Visual Reasoning Challenge
Steven Spratley (University of Melbourne), Tim Miller (University of Melbourne)
CodeConvolutional Neural NetworkImage
π― What it does: Designed and released an advanced matrix problem (Unicode Analogies) dataset based on Unicode characters to evaluate the analogy reasoning and fluid conceptualization capabilities of visual systems;
π― What it does: This paper proposes a completely label-free video object segmentation method that automatically generates pseudo-masks through spatiotemporal pixel clustering on unlabeled videos, and jointly learns mask embeddings and cross-frame correspondences. The trained network can directly perform mask-based continuous segmentation.
π― What it does: This paper presents UDOP, a unified document AI foundation model capable of simultaneously processing visual, textual, and layout information, and consolidating various document tasks into a sequence generation framework, supporting document understanding, question answering, information extraction, classification, as well as document generation and editing.
π― What it does: This paper presents UniHCP, a unified Vision Transformer model capable of simultaneously handling five types of human perception tasks: pose estimation, human segmentation, pedestrian detection, person re-identification (ReID), and attribute recognition.
π― What it does: A unified instance-aware model called UNINEXT is proposed, which can complete 10 types of instance-aware sub-tasks, including object detection, instance segmentation, video tracking, and semantic video segmentation, within the same framework through prompts (category names, language expressions, reference boxes/masks).
π― What it does: The UnSniffer framework is proposed to simultaneously detect unknown objects and known objects under known categories in object detection.
π― What it does: A framework for unpaired image translation based on the shortest path assumption is proposed, constructing a continuous path from the source domain to the target domain, and constraining the generative model by regularizing the path length.
π― What it does: An unsupervised deep probabilistic point cloud registration framework, UDPReg, is proposed, capable of handling partially overlapping point clouds.
π― What it does: An end-to-end network is proposed that does not require signed distance supervision, priors, or normal information, capable of directly inferring the Signed Distance Function (SDF) from a single sparse point cloud.
π― What it does: This paper proposes an unsupervised intrinsic image decomposition method using LiDAR intensity (IID-LI), which decomposes color images into albedo and shade components.
π― What it does: By first identifying the image background and utilizing background seeds extracted through self-supervised Transformer attention, high-quality foreground masks are obtained through self-supervised training with a minimal 1Γ1 convolutional layer on frozen DINO features, thus achieving unsupervised object localization.
π― What it does: This paper proposes an unsupervised 3D U-Net network that utilizes a volume of optical flow over time for multi-motion segmentation, and is trained using a spatiotemporal parameter motion model and temporal consistency regularization.
π― What it does: A framework for unsupervised visible-infrared person re-identification is proposed, which utilizes progressive graph matching to mine cross-modal correspondences and reduces modal discrepancies through alternating cross-modal contrastive learning.
π― What it does: This paper proposes a source unsupervised general domain adaptation method based on global and local clustering (GLC), which can achieve model reuse for domain shift and category shift under the premise of using only a closed-set model pre-trained on the source domain.
V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting
Haibao Yu (Tsinghua University), Zaiqing Nie (Baidu Inc.)
CodeObject TrackingAutonomous DrivingTransformerSimultaneous Localization and MappingTime SeriesSequential
π― What it does: This paper presents the V2X-Seq large-scale continuous V2X dataset, which includes synchronized perception and trajectory prediction data for vehicles and infrastructure.
π― What it does: In the absence of image-text paired annotations, this paper proposes to use the CLIP pre-trained model to estimate the text embeddings corresponding to images and to train a text-to-image generation model by maximizing the likelihood in an unsupervised manner through variational inference.
π― What it does: A perspective-dependent normalization-based NeRF training framework (VDN-NeRF) is proposed, which extracts viewpoint-invariant deep features through self-distillation to suppress shape-radiance ambiguity and improve the quality of geometric reconstruction under non-Lambertian surfaces and dynamic lighting conditions.
VectorFloorSeg: Two-Stream Graph Attention Network for Vectorized Roughcast Floorplan Segmentation
Bingchen Yang (University of Chinese Academy of Sciences), Jun Xiao (University of Chinese Academy of Sciences)
CodeSegmentationGraph Neural NetworkImageGraph
π― What it does: This paper addresses the semantic segmentation of rooms in roughly drawn vector floor plans (house floor plans), specifically predicting the spatial area and category of each room.
Wei Lin (Graz University of Technology), Horst Bischof (Graz University of Technology)
CodeDomain AdaptationVideo
π― What it does: This paper proposes an online video test-time adaptation method called ViTTA, aimed at improving action recognition performance on test videos with distribution shifts.
π― What it does: A Transformer-based multi-view 3D object detection framework VEDet is proposed, which utilizes viewpoint equivariance to learn the consistency of objects across different camera perspectives, thereby improving 3D localization accuracy.
π― What it does: A novel sparse convolution (VirConv) is proposed to integrate virtual points obtained through depth completion and LiDAR points, achieving efficient multi-modal 3D object detection.
VisFusion: Visibility-Aware Online 3D Scene Reconstruction From Videos
Huiyu Gao (Australian National University), Miaomiao Liu (Australian National University)
CodeConvolutional Neural NetworkRecurrent Neural NetworkSimultaneous Localization and MappingVideo
π― What it does: This paper proposes an online, visually perceptive 3D scene reconstruction method called VisFusion, which can generate detailed and dense 3D models in real-time from calibrated monocular video.
π― What it does: This paper proposes a visual constraint broadband spectral design method to optimize the mixed lighting of visible and near-infrared LEDs, aiming to enhance RGB reconstruction effects in completely dark environments.
π― What it does: A two-stage framework and Mask Auto-Labeler are proposed to automatically generate high-quality masks under box supervision for instance segmentation training.
π― What it does: This paper proposes a visually dependent Transformer (DependencyViT) that achieves unsupervised construction of a hierarchical dependency tree of image patches through reverse attention, and designs a lightweight version (DependencyViT-Lite) to implement dynamic visual pooling.
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Ming Y. Lu (Massachusetts Institute of Technology), Faisal Mahmood (Harvard University)
CodeClassificationTransformerPrompt EngineeringContrastive LearningImageTextBiomedical Data
π― What it does: Proposes the MI-Zero framework, which utilizes contrastive vision-language pre-trained models to achieve zero-shot transfer for large-sized whole slide images (WSI) through multi-instance learning, completing cancer subtype classification.