π― What it does: A hierarchical contrastive learning framework is proposed for blind detection of damaged regions in images and non-destructive image restoration.
π― What it does: This paper proposes a hierarchical point-based active learning framework for semi-supervised 3D point cloud semantic segmentation, which can significantly improve segmentation performance with very few labeled points.
Hierarchical Prior Mining for Non-local Multi-View Stereo
Chunlin Ren (Northwestern Polytechnical University), Jiaqi Yang (Northwestern Polytechnical University)
CodeRestorationDepth EstimationSimultaneous Localization and MappingImageBenchmark
π― What it does: A multi-view stereo reconstruction method based on hierarchical prior mining, HPM-MVS, is proposed, which integrates non-local expandable sampling patterns (NESP), KNN-based plane prior construction, and a multi-scale hierarchical prior mining framework.
π― What it does: A hierarchical split graph convolutional network (HD-GCN) is proposed for skeletal action recognition, constructing a hierarchical split graph (HD-Graph) and combining it with an attention aggregation module (A-HA), followed by a six-stream non-motion flow ensemble to enhance performance.
π― What it does: A heterogeneous multimodal vehicle-to-vehicle collaborative perception framework named HM-ViT is proposed, which can share and fuse information among vehicles with different numbers and types of sensors, enhancing 3D object detection performance.
π― What it does: In the structural reconstruction task, a frequency domain feature learning strategy called F-Learn is introduced, which enhances topological reasoning accuracy by performing frequency domain convolution to fuse geometric fragments on low-level features.
Homeomorphism Alignment for Unsupervised Domain Adaptation
Lihua Zhou (University of Electronic Science and Technology of China), Ferrante Neri (University of Surrey)
CodeDomain AdaptationFlow-based ModelImage
π― What it does: A homomorphic mapping (HMA) based on Invertible Neural Networks (INN) is proposed, establishing feature spaces on both the source and target domains, maintaining topological structure through homomorphic mapping and achieving distribution alignment, ultimately training the model simultaneously in both spaces.
Homography Guided Temporal Fusion for Road Line and Marking Segmentation
Shan Wang (Data61 CSIRO), Hongdong Li (Australian National University)
CodeSegmentationAutonomous DrivingImageVideo
π― What it does: This paper proposes a homotopy transformation-based temporal fusion module called HomoFusion, which utilizes information from adjacent frames to recover occluded lane lines and markings, achieving lightweight lane marking segmentation.
HSE: Hybrid Species Embedding for Deep Metric Learning
Bailin Yang (Zhejiang Gongshang University), Chao Song (Zhejiang Gongshang University)
CodeRetrievalContrastive LearningImage
π― What it does: In deep metric learning, this paper proposes the Hybrid Species Embedding (HSE) method, which generates unlabeled mixed samples (Hybrid species) through mixed sample data augmentation, serving as additional training signals to enhance the generalization ability of the embedding space.
I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision
Sophia Gu (Allen Institute for Artificial Intelligence), Aniruddha Kembhavi (Allen Institute for Artificial Intelligence)
CodeDomain AdaptationRepresentation LearningTransformerLarge Language ModelSupervised Fine-TuningContrastive LearningImageTextMultimodality
π― What it does: This paper proposes a cross-modal zero-shot transfer method (CLOSE), which trains a model using text data in a visual-text joint embedding space obtained through contrastive learning, and then replaces text embeddings with image embeddings to complete visual tasks. It demonstrates that training solely on text can achieve performance close to that of image-trained models in tasks such as image captioning, visual entailment, VQA, and visual news.
CodeClassificationExplainability and InterpretabilityConvolutional Neural NetworkContrastive LearningImage
π― What it does: An interpretable category incremental continual learning method called ICICLE is proposed, which gradually expands knowledge through prototype component learning.
π― What it does: We propose ClipVID, an end-to-end video object detection model that enhances object representation by utilizing identity-consistent temporal context aggregation and achieves clip-level parallel prediction.
π― What it does: An unsupervised, unlabelled identity-seeking self-supervised representation learning (ISR) method is proposed, which learns a human re-identification model that can efficiently perform in unseen domains using large-scale video data.
π― What it does: This paper proposes a method for generating synthetic facial data based on an identity-conditioned diffusion model called IDiff-Face, which can provide high-quality training samples with identity distinguishability and intra-class diversity for facial recognition models.
π― What it does: This paper proposes an Iterative Hierarchical Structure-based Scene Flow Estimation Network (IHNet), which improves motion estimation between point clouds by guiding the current level with the high-resolution estimation results from the previous iteration.
π― What it does: A neural feature activation model IIEU based on a multi-criteria decision-making (MCDM) perspective is proposed to address the feature scoring mismatch problem and enhance the performance of ReLU and SOTA activation functions.
π― What it does: Injecting a classifier for new categories into a pre-trained classification model to achieve zero-shot classification without any image data.
π― What it does: This paper proposes an Implicit AutoEncoder (IAE) that uses implicit surface representation to address the sampling variation problem in self-supervised representation learning of point clouds by changing the decoder output to an implicit function.
π― What it does: A framework for low-light image enhancement using implicit neural representation (NeRCo) is proposed, capable of recovering visually pleasing high-brightness images under unsupervised conditions.
π― What it does: An Implicit Learnable Alignment (ILA) method is proposed, which uses an implicit learnable mask for coarse alignment of adjacent frames, replacing traditional spatiotemporal attention for temporal modeling in video recognition.
π― What it does: This paper proposes a natural language supervision method based on a fixed text encoder (TeS), which regularizes the classifier of the visual model by introducing a text reference distribution, alleviating the conflict between pre-trained model bias and catastrophic forgetting.
π― What it does: Using two pre-trained 2D diffusion models in the vertical direction to solve 3D inverse problems, constructing generation and reconstruction of 3D volumes.
π― What it does: This paper studies the robustness differences of the visual BERT pre-training method (especially MAE) under adversarial attacks and proposes a scheme to enhance the robustness of MAE during the testing phase through frequency domain visual prompts.
π― What it does: On existing adversarial training models, only the least important modules are fine-tuned, and interpolation is used to enhance generalization ability.
Improving Lens Flare Removal with General-Purpose Pipeline and Multiple Light Sources Recovery
Yuyan Zhou (Nanjing University of Aeronautics and Astronautics), Chongyi Li (Nankai University)
CodeRestorationData SynthesisTransformerImage
π― What it does: This paper proposes a new pixel-level convex combination synthesis process based on ISP to generate more realistic halo pollution images, and designs a multi-source recovery strategy that does not require hard thresholds. It also contributes a real halo test dataset across multiple devices; the U-Former/UNet model trained based on this scheme performs better on various benchmarks.
π― What it does: This study investigates the low-frequency feature bias of pixel-level MIM and proposes a multi-layer feature fusion method to enhance the representation capability of MAE and PixMIM.
Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints
Chenjie Cao (Fudan University), Yanwei Fu (Fudan University)
CodePose EstimationRetrievalTransformerImage
π― What it does: A cascade matching model based on Transformer, CasMTR, is proposed, achieving denser and more accurate image matching, supporting high-resolution images.
π― What it does: A hierarchical perception framework (HAFrame) is proposed, which fixes this framework in the linear classifier of deep networks in advance, and uses cosine similarity auxiliary loss to cluster features onto the corresponding classifier vectors, thereby reducing the severity of errors.
π― What it does: In the scenario of single-image target domain adaptation, a framework named Informative Data Mining (IDM) is proposed, which can quickly adapt a pre-trained source domain semantic segmentation model using only one unlabeled target image and a limited number of iterations.
Man Yao (Xi'an Jiaotong University), Guoqi Li (Chinese Academy of Sciences)
CodeSpiking Neural NetworkVideo
π― What it does: This paper conducts a systematic analysis of the redundancy problem in Spiking Neural Networks (SNN) and proposes a high-level spatial attention (ASA) module that optimizes membrane potential distribution to reduce redundant spikes and enhance performance.
Inspecting the Geographical Representativeness of Images from Text-to-Image Models
Abhipsa Basu (Indian Institute of Science), Danish Pruthi (Indian Institute of Science)
CodeGenerationDiffusion modelImage
π― What it does: By comparing the geographic representativeness of images generated by DALLΒ·E 2 and Stable Diffusion through human evaluation across different countries, this study analyzes their representativeness, authenticity, and the feasibility of automatic evaluation methods.
Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection
Feng Liu (University of Chinese Academy of Sciences), Qixiang Ye (University of Chinese Academy of Sciences)
CodeObject DetectionTransformerImage
π― What it does: Proposes to completely transfer the pre-trained Vision Transformer (ViT) encoder-decoder to the object detector, removing the traditional FPN and using only the decoder as the detection head, while adding a Multi-Scale Feature Modulator (MFM) to construct a 'fully pre-trained' feature extraction path.
π― What it does: A new video question answering task, IntentQA, is proposed, focusing on video intent reasoning, and a corresponding large-scale dataset is constructed.
You Huang (Xiamen University), Rongrong Ji (Contemporary Amperex Technology Co. Limited)
CodeSegmentationTransformerImage
π― What it does: This paper proposes InterFormer, which separates image preprocessing from interaction, using large-scale Vision Transformer for offline encoding, and employs a lightweight I-MSA module for real-time segmentation on CPU during interaction.
π― What it does: This paper proposes a self-supervised invariant feature regularization (INV-REG) method that iteratively learns data partitioning and feature regularization to eliminate biases caused by representations such as race and gender, thereby enhancing fairness in facial recognition.
π― What it does: The INVJOINT invariant training framework is proposed, which significantly improves few-shot point cloud classification performance by selecting and training joint hard samples between 2D and 3D models.
Inverse Problem Regularization with Hierarchical Variational Autoencoders
Jean Prost (University of Bordeaux), Nicolas Papadakis (University of Bordeaux)
CodeRestorationSuper ResolutionAuto EncoderImage
π― What it does: A hierarchical variational autoencoder (HVAE) is proposed as a prior to solve linear inverse problems (deblurring, super-resolution, inpainting, etc.) by alternately optimizing images and latent variables.
π― What it does: This paper proposes an open semi-supervised learning framework named IOMatch, which can jointly utilize both internal and external samples from unlabeled data for training.
π― What it does: We propose DualMind, a general decision-making agent that uses dual-stage training to directly execute tasks across multiple domains, scenarios, and different executors based on prompts.
π― What it does: A dual-stream Joint-Relation Transformer is proposed for multi-person motion prediction, capable of simultaneously modeling skeletal joint information and inter-joint relationships, incorporating relationship-aware attention and relationship supervision into the Transformer.
π― What it does: This study focuses on 3D human mesh recovery from a single image in occluded scenes, proposing the JOTR framework that integrates 2D global features and 3D local features through a Transformer for coarse-to-fine alignment.
π― What it does: A simple attention-based pooling method called SimPool is proposed to replace the default pooling in convolutional and Transformer networks.
π― What it does: A self-supervised monocular depth estimation model is proposed, trained using a large-scale SlowTV video dataset extracted from YouTube, achieving zero-shot cross-domain generalization.
Knowing Where to Focus: Event-aware Transformer for Video Grounding
Jinhyun Jang (Yonsei University), Kwanghoon Sohn (Korea Institute of Science and Technology)
CodeRecognitionSegmentationTransformerVideo
π― What it does: This paper proposes an event-aware video alignment framework EaTR, which utilizes event reasoning and moment reasoning to dynamically generate moment queries, achieving end-to-end video moment localization.
π― What it does: A federated active learning framework KAFAL is designed for non-IID data, which efficiently learns a global model with a limited labeling budget by utilizing knowledge-specialized active sampling and knowledge compensation federated updates while preserving data privacy.
Label-Efficient Online Continual Object Detection in Streaming Video
Jay Zhangjie Wu (Show Lab), Mike Zheng Shou (Show Lab)
CodeObject DetectionVideo
π― What it does: A more practical label-efficient online continuous object detection problem (LEOCOD) is proposed, along with a pluggable Efficient-CLS module to address it;
π― What it does: Proposes the Label-Guided Knowledge Distillation (LGKD) method, which utilizes the labels of the current step to guide the transfer of background probabilities from the old model, thereby addressing the novel-background conflict issue in continuous semantic segmentation, and validates its effectiveness on both 2D and 3D data.
π― What it does: An end-to-end HDR video reconstruction framework called LAN-HDR is proposed, which aligns and completes details of low dynamic range (LDR) video frames using brightness information, and generates flicker-free HDR videos through temporal consistency loss.
π― What it does: This paper studies the position embedding problem in visual Transformers and proposes a Layer Adaptive Position Embedding (LaPE) scheme, which uses two independent layer normalizations to process token embeddings and position embeddings separately at each layer, and progressively integrates position embeddings layer by layer.
π― What it does: A lightweight Large Selective Kernel Network (LSKNet) is proposed, which enhances remote sensing object detection by dynamically adjusting large convolution kernels.
π― What it does: A three-stage framework for recognizing occluded facial expressions is proposed: first, use ViT-SVDD to detect occluded patches; then, restore the complete face through a hybrid reconstruction network that integrates ViT and CNN; finally, extract expression-related ViT latent vectors from the reconstruction process and perform expression classification together with CNN features.
π― What it does: This paper proposes LATR, an end-to-end 3D lane detection framework based on Transformer, which directly performs 3D lane localization on front-view images, eliminating the need for traditional intermediate 3D representations such as BEV or projections.
π― What it does: This paper proposes a text-driven image segmentation method based on latent diffusion models, ZNet and LD-ZNet, which utilizes the compressed latent space z of LDM and internal visual-language features to achieve more robust semantic segmentation.
π― What it does: This paper proposes a neural field that combines feature rendering loss and occupancy-SDF mixed representation to recover high-detail 3D geometry of indoor scenes from multi-view images.
π― What it does: A self-supervised contrastive learning loss GroCo based on differentiable sorting networks is proposed, treating positive and negative samples as groups and optimizing their sorting constraints.
π― What it does: This paper proposes an unsupervised learning method for degradation correction filtering, utilizing a spatially variable degradation adaptive regression module (DARM) to transform the complex degradation of low-resolution images into known degradation, allowing existing super-resolution networks to operate in blind scenarios.
π― What it does: This paper proposes a multi-scale vector quantization degradation model that utilizes the intrinsic features of animated videos to generate more realistic low-resolution training data, thereby enhancing the super-resolution effect of animated videos.
π― What it does: Learning fine-grained features for pixel-level video correspondence, combining synthetic and real video self-supervised training, and employing soft labels, adversarial domain adaptation, and a coarse-to-fine matching framework to improve correspondence accuracy and efficiency.
Learning from Noisy Data for Semi-Supervised 3D Object Detection
Zehui Chen (University of Science and Technology of China), Feng Zhao (University of Science and Technology of China)
CodeObject DetectionAutonomous DrivingPoint Cloud
π― What it does: This paper proposes a pseudo-labeling based semi-supervised 3D object detection framework called NoiseDet, which effectively reduces the negative impact of pseudo-label noise on the model by softening classification targets, probability regression, and pixel-level feature consistency constraints.
π― What it does: This study investigates how to use energy-based prior with a joint latent space in multi-layer generative models to learn hierarchical features and proposes a joint training framework.
π― What it does: This paper proposes a multi-label classification task with the coexistence of long-tail distribution and partial labels (PLT-MLC), and designs an end-to-end COMIC framework for learning.
π― What it does: A navigation visual representation learning method based on contrastive learning, called Ego-Map, is proposed. It aligns the ego-centric images with the semantic maps generated from their trajectories, thereby enhancing the ability to capture the semantic and spatial information required for navigation.
Learning Optical Flow from Event Camera with Rendered Dataset
Xinglong Luo (University of Electronic Science and Technology of China), Shuaicheng Liu (Hong Kong University of Science and Technology)
CodeData SynthesisOptical FlowImageVideo
π― What it does: A new event optical flow dataset MDR based on graphic rendering is proposed, along with the design of an Adaptive Density Module (ADM) to enhance the robustness of event optical flow estimation.
π― What it does: A cross-domain semantic segmentation framework RTea based on pseudo-relation learning is proposed, which utilizes a pixel relationship matrix generated by pseudo-labels to guide model learning.
π― What it does: This paper proposes an EM-like framework that dynamically estimates the number of categories and prototypes using a semi-supervised Gaussian mixture model, and completes clustering of unlabeled data through prototype contrastive learning, achieving universal category discovery without prior knowledge of the number of categories.
π― What it does: An end-to-end feature learning framework is proposed, which constructs visual feature branches and spatial context branches in parallel on a CNN backbone. It captures local descriptor types and their spatial distribution through online token learning and random walk-based distance encoding, and fuses them with visual features using cross-attention to generate spatial context-aware global features for instance image retrieval.
π― What it does: A video-based deep state recognition framework called Deep State Identifier is proposed to automatically identify the most critical states for final rewards in the reinforcement learning process without action information.
π― What it does: A lightweight and efficient dynamic upsampler called DySample is proposed, which utilizes point sampling to achieve feature upsampling, eliminating the need for traditional dynamic convolution, additional high-resolution guided features, and custom CUDA code.
Learning Vision-and-Language Navigation from YouTube Videos
Kunyang Lin (South China University of Technology), Chuang Gan (UMass Amherst)
CodeRecognitionRetrievalRobotic IntelligenceTransformerVision Language ModelVideoMultimodality
π― What it does: This paper constructs a large-scale VLN dataset (YouTube-VLN) using YouTube house tour videos and pre-trains the 'Lily' agent on this dataset to enable it to learn visual language navigation from natural videos.
Lecture Presentations Multimodal Dataset: Towards Understanding Multimodality in Educational Videos
Dong Won Lee (Carnegie Mellon University), Louis-Philippe Morency (Carnegie Mellon University)
CodeGenerationRetrievalTransformerLarge Language ModelVision Language ModelImageVideoTextMultimodality
π― What it does: A large-scale Lecture Presentations Multimodal (LPM) dataset is proposed, along with three retrieval/generation tasks designed to evaluate visual language models' cross-modal understanding of educational videos.
π― What it does: A new object detection model called Focus-DETR is proposed, aimed at improving computational efficiency and model accuracy by focusing on more informative tokens.
π― What it does: A new skeleton action recognition framework called STC-Net is proposed, which enhances the model's spatiotemporal perception ability using the Spatiotemporal Curve (STC) module and the Dilated Graph Convolution (DK-GC) module.
π― What it does: A multi-modal LiDAR-camera fusion network LCPS is proposed for 3D panoramic segmentation, integrating four modules: ACPA, SARA, PVP, and FOG;
π― What it does: This paper proposes an unsupervised domain adaptation framework named LiDAR-UDA, which utilizes self-supervised temporal consistency to improve the transfer performance of LiDAR semantic segmentation models.
π― What it does: A lightweight single-image super-resolution network SPIN is proposed, utilizing superpixel partitioning and Transformer, which integrates superpixel aggregation, cross-superpixel attention, and internal superpixel attention;
LIMITR: Leveraging Local Information for Medical Image-Text Representation
Gefen Dawidowicz (Technion Israel Institute of Technology), Ayellet Tal (Technion Israel Institute of Technology)
CodeRetrievalRepresentation LearningTransformerContrastive LearningImageTextBiomedical Data
π― What it does: This paper proposes a model named LIMITR, which is used to learn the joint representation of chest X-ray images and their corresponding radiology reports, and applies it to tasks such as text-image retrieval, phrase localization, and category retrieval.
π― What it does: A framework called Lip2Vec is proposed to reconstruct visual speech recognition into audio speech recognition, achieving text output using only video by learning the mapping from lip features to audio latent space.
π― What it does: This paper proposes the LIST network, which utilizes both local and global features and achieves implicit reconstruction of 3D objects from a single view through spatial transformation, avoiding the need for camera estimation and pixel alignment.
LISTER: Neighbor Decoding for Length-Insensitive Scene Text Recognition
Changxu Cheng (DAMO Academy), Cong Yao (DAMO Academy)
CodeRecognitionTransformerImageText
π― What it does: A length-insensitive scene text recognition framework called LISTER is proposed, which maintains high accuracy on both short and long texts.
π― What it does: This paper presents the LivelySpeaker system, which splits the generation of speech-accompanied gestures into two stages: semantic-aware generation and rhythm-aware generation. These stages are implemented using CLIP text embeddings and an MLP-based diffusion model, ultimately achieving gesture animations that are both semantically and rhythmically aligned.
π― What it does: The paper proposes a 3D reconstruction framework capable of online processing of dynamic camera poses, addressing the issue of geometric inconsistency caused by traditional methods neglecting pose updates.
π― What it does: A large-scale multi-reference super-resolution dataset LMR was constructed, and the MRefSR method was proposed to achieve joint super-resolution of multi-reference images.
π― What it does: A locally context-aware active domain adaptation framework LADA is proposed, which selects target samples based on local inconsistencies predicted by the model and enhances model adaptation through an incremental anchor point set.
Localizing Moments in Long Video Via Multimodal Guidance
Wayner Barrios (Dartmouth), Bernard Ghanem (King Abdullah University of Science and Technology)
CodeObject DetectionRetrievalTransformerVision Language ModelVideoTextMultimodalityAudio
π― What it does: A two-stage multimodal guided video temporal localization framework is proposed, which first uses a guiding model to filter describable windows and then inputs them into a baseline localization model to enhance the grounding effect of long videos.
Logic-induced Diagnostic Reasoning for Semi-supervised Semantic Segmentation
Chen Liang (Zhejiang University), Yi Yang (Zhejiang University)
CodeSegmentationImage
π― What it does: Designed and implemented LOGICDIAG, a framework that integrates symbolic logic knowledge with sub-symbolic semi-supervised semantic segmentation, using logical diagnosis to correct pseudo-labels and mitigate confirmation bias.
π― What it does: LOGICSEG is proposed, a visual semantic parsing framework that combines hierarchical semantic structures with first-order logic reasoning, capable of utilizing both sub-symbolic learning and symbolic reasoning in semantic segmentation tasks.
π― What it does: A unified model compression framework is proposed in a post-training environment, capable of performing both lossless compression (entropy coding) and lossy compression (pruning, quantization) simultaneously.
Luminance-aware Color Transform for Multiple Exposure Correction
Jong-Hyeon Baek (Chungnam National University), Yeong Jun Koh (Chungnam National University)
CodeRestorationConvolutional Neural NetworkImage
π― What it does: This paper proposes a brightness-aware color transformation (LACT) algorithm to correct overexposure and underexposure in multi-exposure images.
π― What it does: A multi-modal autoencoder framework called MAAL is proposed for learning the manipulability of 3D articulated objects, which includes an MME encoder, action memory, and a decoder, utilizing only positive samples in an end-to-end training process.
Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations
Jianren Wang (Carnegie Mellon University), Abhinav Gupta (Carnegie Mellon University)
CodeRobotic IntelligenceReinforcement Learning from Human FeedbackReinforcement LearningContrastive LearningVideo
π― What it does: Using a pre-trained visual representation learning distance function and a dynamics model, four types of manipulation tasks (pushing, picking and placing, opening doors, and turning knobs) are trained based on low-cost human video data and implemented on robots, with direct action planning through shooting methods;
MAP: Towards Balanced Generalization of IID and OOD through Model-Agnostic Adapters
Min Zhang (Zhejiang University), Kun Kuang (Zhejiang University)
CodeDomain AdaptationOptimizationImage
π― What it does: A model-agnostic method based on auxiliary adapters (MAP) is proposed, achieving a balance between IID and OOD generalization capabilities through bilevel optimization.
π― What it does: MAPConNet is proposed, a self-supervised 3D pose transfer framework that can be trained without corresponding labels or target outputs.
π― What it does: This paper proposes the use of pre-change semantic maps in remote sensing dual-phase change detection to achieve conditional change detection and cross-modal change detection;
March in Chat: Interactive Prompting for Remote Embodied Referring Expression
Yanyuan Qiao (Australian Institute for Machine Learning), Qi Wu (Australian Institute for Machine Learning)
CodeOptimizationRobotic IntelligenceTransformerLarge Language ModelPrompt EngineeringVision Language ModelText
π― What it does: An interactive planning framework named March-in-Chat (MiC) is proposed, enabling VLN agents to engage in real-time dialogue with large language models (LLMs) to generate fine-grained navigation plans based on high-level instructions and current visual perceptions, achieving remote entity localization.
π― What it does: This paper proposes a model-agnostic bias object removal framework called MARS, which addresses the misjudgment issues in weakly supervised semantic segmentation caused by background or related objects (e.g., misclassifying a railway as a train).