π― What it does: This paper proposes a frequency domain augmentation method based on Fourier basis functions (Auxiliary Fourier-basis Augmentation, AFA), which generates adversarial samples by adding plane waves with adjustable amplitudes to the image spectrum, and adapts to distribution shifts with the help of an auxiliary branch.
π― What it does: We propose Free3D, a single-view novel view synthesis method based on a pre-trained 2D diffusion model without 3D representation, capable of generating pose-accurate and consistent images within a 360Β° view.
π― What it does: The FreeU method is proposed in the U-Net structure of diffusion models, enhancing generation quality by adjustable scaling of features from the backbone network and skip connections during the inference phase.
Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture
Fei Wang (Hefei University of Technology), Meng Wang (Hefei University of Technology)
CodeTransformerContrastive LearningVideo
π― What it does: A multi-layer equivalent Transformer architecture based on frequency decoupling is proposed to amplify and enhance subtle movements in videos.
π― What it does: This paper proposes Frequency Adaptive Dilated Convolution (FADC), which enhances the performance of semantic segmentation and object detection by dynamically adjusting the dilation rate based on local spectral characteristics, decomposing convolutional kernel weights, and balancing frequency on features.
Tao Li (Shanghai Jiao Tong University), Xiaolin Huang (Shanghai Jiao Tong University)
CodeOptimizationConvolutional Neural NetworkImage
π― What it does: This study investigates the core mechanism of Sharpness-Aware Minimization (SAM), decomposing the batch gradient into full gradient components and noise components. It finds that the noise component is key to improving generalization and proposes the Friendly-SAM (F-SAM) algorithm based on this insight.
From Correspondences to Pose: Non-minimal Certifiably Optimal Relative Pose without Disambiguation
Javier Tirado-GarΓn (University of Zaragoza), Javier Civera (University of Zaragoza)
CodePose EstimationOptimizationImage
π― What it does: A method is proposed to estimate the relative pose of the camera directly from matching points without the need for post-processing ambiguity resolution.
π― What it does: A weakly supervised semantic segmentation framework S2C is proposed, which directly improves the quality of CAM during the training phase using SAM and generates more accurate pseudo-labels.
CodePose EstimationOptimizationSimultaneous Localization and MappingOptical FlowImage
π― What it does: This paper analyzes the gradient variance problem of differentiable bundle adjustment layers in attitude estimation and proposes using weights obtained from internal optimization to weight flow loss in order to reduce gradient noise, thereby accelerating and stabilizing training.
π― What it does: We propose WeCLIP, a weakly supervised semantic segmentation single-stage pipeline that utilizes a frozen CLIP model as the backbone network, and we design a decoder and a CAM refinement module to achieve pixel-level segmentation.
Frozen Feature Augmentation for Few-Shot Image Classification
Andreas BΓ€r (Technische UniversitΓ€t Braunschweig), Manoj Kumar (Google DeepMind)
CodeClassificationTransformerImage
π― What it does: The study directly applies data augmentation (Frozen Feature Augmentation, FroFA) on frozen features of pre-trained vision Transformers to enhance few-shot image classification performance.
π― What it does: A shape completion model FSC is proposed for very sparse point cloud inputs, capable of reconstructing a complete 3D point cloud from dozens of points.
Fully Exploiting Every Real Sample: SuperPixel Sample Gradient Model Stealing
Yunlong Zhao (Central South University), Wei Chen (Zhejiang University)
CodeKnowledge DistillationAdversarial AttackImage
π― What it does: This paper proposes a model stealing method based on superpixel gradients, called SPSG, which utilizes gradient information from a small number of real samples to improve the replication effect on black-box models.
π― What it does: Proposes a 3D component grouping task and designs a gradient field-based autoregressive sampling framework G-FARS to automatically group from a mixed component set.
CodeGenerationData SynthesisDepth EstimationVision Language ModelDiffusion modelNeural Radiance FieldImage
π― What it does: A 3D generation framework G3DR based on single-view images is proposed, capable of generating high-quality, geometrically realistic 3D objects from large-scale single-view data such as ImageNet.
π― What it does: This paper presents the Gaussian Head Avatar, which reconstructs high-fidelity human head models through controllable dynamic 3D Gaussian point clouds, enabling expression-driven and pose control.
π― What it does: A unified Transformer framework GPM is proposed, combining BERT-style autoencoding and GPT-style autoregression, supporting pre-training of point clouds, downstream classification/segmentation, and unconditional/conditional generation.
Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction
Hao Li (Xiamen University), Yuchen Han (Shanghai Jiao Tong University)
CodeClassificationTransformerLarge Language ModelPrompt EngineeringContrastive LearningImageTextBiomedical Data
π― What it does: This paper proposes a whole slide image (WSI) classification framework called FiVE, which is based on fine-grained visual-semantic interaction. It utilizes GPT-4 to automatically extract fine-grained descriptive labels from raw pathology reports and achieves efficient training through a task-specific fine-grained semantic (TFS) module and sampling strategy.
Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models
Bin Fu (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences), Yu Qiao (Shanghai Artificial Intelligence Laboratory)
CodeGenerationDiffusion modelImage
π― What it does: A multi-stage font generation framework called MSD-Font is proposed, which achieves a three-stage generation process from structure construction to style transfer and then to detail refinement by embedding the font transfer process in a latent diffusion model.
π― What it does: This paper proposes a generative 3D part assembly network based on hyper-part and whole hierarchical information, achieving the prediction of the 6 degrees of freedom pose of parts.
Generative Multi-modal Models are Good Class Incremental Learners
Xusheng Cao (Nankai University), Ming-Ming Cheng (Nankai University)
CodeClassificationGenerationTransformerLarge Language ModelImageMultimodality
π― What it does: An incremental learning framework based on Generative Multimodal Models (GMM) is proposed, achieving class-incremental learning without a classification head by generating label text and matching it with category text.
Generative Multimodal Models are In-Context Learners
Quan Sun (Beijing Academy of Artificial Intelligence), Xinlong Wang (Beijing Academy of Artificial Intelligence)
CodeGenerationData SynthesisTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageVideoTextMultimodality
π― What it does: A 37B parameter multimodal generative model, Emu2, was proposed and trained, demonstrating strong contextual learning capabilities, and achieving chat and controllable visual generation through instruction fine-tuning.
π― What it does: This paper proposes a generative model identity forgetting framework called GUIDE, which can completely eliminate the generative capability of an identity using only a single image in a pre-trained 3D GAN (EG3D);
π― What it does: Trained a text + image conditional generation network based on a diffusion model, capable of generating corresponding action images and target state images based on input images and textual prompts of actions or final states, while maintaining scene consistency.
π― What it does: This paper proposes GeoAuxNet, which uses a voxel-guided dynamic point network to generate point-level geometric information and injects this fine-grained point information into voxel features through a hierarchical geometric pooling, thereby achieving unified representation learning of multi-sensor point clouds.
π― What it does: A generalizable neural radiance field framework called GeFu is proposed, which can synthesize high-quality new views in unseen scenes using just a few perspectives.
π― What it does: GigaPose is proposed, a 6D pose estimation method for novel objects that achieves fast and robust coarse pose estimation using limited templates and single image patch correspondences, and can be combined with subsequent refinement methods.
GLACE: Global Local Accelerated Coordinate Encoding
Fangjinhua Wang (ETH Zurich), Marc Pollefeys (Microsoft)
CodePose EstimationRetrievalSimultaneous Localization and MappingImage
π― What it does: GLACE proposes a method for achieving high-precision visual localization in large-scale scenes using a single network without the need for 3D point clouds or depth maps.
GLiDR: Topologically Regularized Graph Generative Network for Sparse LiDAR Point Clouds
Prashant Kumar (Indian Institute of Technology Delhi), Prem Kalra (Indian Institute of Technology Delhi)
CodeSegmentationGenerationAutonomous DrivingGraph Neural NetworkSimultaneous Localization and MappingPoint Cloud
π― What it does: Using graph neural networks and 0-dimensional persistent homology regularization for static point completion and dynamic/static segmentation of sparse LiDAR point clouds.
π― What it does: The P NeRF method is proposed, which utilizes the global and hierarchical geometric consistency priors of pre-trained models to improve the NeRF reconstruction quality of indoor scenes from a limited number of views.
Global and Local Prompts Cooperation via Optimal Transport for Federated Learning
Hongxia Li (ShanghaiTech University), Ye Shi (ShanghaiTech University)
CodeOptimizationFederated LearningTransformerPrompt EngineeringVision Language ModelImage
π― What it does: This paper proposes FedOTP, a method for tuning visual-language models that simultaneously learns global and local prompts within a federated learning framework, achieving collaboration between the two through unbiased optimal transport.
π― What it does: This work proposes a cost aggregation method based on geometric consistency called GoMVS, which projects neighborhood costs into the depth space of the reference pixel using the local plane assumption of adjacent pixels and surface normal vectors, followed by convolutional aggregation.
π― What it does: A general open vocabulary neural semantic field named GOV-NeSF is proposed for achieving 3D and 2D open vocabulary semantic segmentation and novel view rendering in unseen scenes through multi-view fusion, with training relying solely on 2D RGB images, without the need for depth, semantic labels, or 3D data.
π― What it does: A real-time high-resolution human novel viewpoint synthesis method called GPS-Gaussian is proposed, which directly regresses 3D Gaussian points through pixel-level Gaussian parameter mapping, enabling the generation of high-quality viewpoint images without separate optimization.
GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation
Tong Wu (Chinese University of Hong Kong), Gordon Wetzstein (Stanford University)
CodeGenerationTransformerLarge Language ModelPrompt EngineeringTextMultimodality
π― What it does: A new metric is proposed that utilizes GPT-4V to automatically generate evaluation prompts and conduct multi-dimensional assessments of text-to-3D generation models.
π― What it does: A dynamic axial graph construction (DAGC) method is proposed to build an efficient Vision GNN, and a hybrid CNN-GNN structure called GreedyViG is developed based on DAGC to enhance visual task performance while maintaining low parameters and computational power.
Grounded Question-Answering in Long Egocentric Videos
Shangzhe Di (Shanghai Jiao Tong University), Weidi Xie (Shanghai AI Lab)
CodeGenerationRetrievalTransformerLarge Language ModelVideoText
π― What it does: This paper proposes a unified model called GroundVQA, which implements spatiotemporal localization and question-answering tasks for long-term egocentric videos.
Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
Walid Bousselham (University of Bonn), Hilde Kuehne (Goethe University Frankfurt)
CodeObject DetectionSegmentationTransformerVision Language ModelContrastive LearningImageMultimodality
π― What it does: Based on a pre-trained visual-language transformer, a training-free 'Grounding Everything Module' (GEM) is proposed, which achieves object localization and segmentation of open vocabulary through self-self attention.
π― What it does: A self-supervised 3D point cloud representation learning framework called GroupContrast is proposed, which combines segment grouping and semantic-aware contrastive learning to enhance the representation consistency of semantically similar points in point clouds.
Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection
Jongha Kim (Korea University), Hyunwoo J. Kim (Korea University)
CodeRecognitionObject DetectionTransformerImage
π― What it does: This paper proposes SpeaQ, a specialized and quality-aware multi-assignment label allocation method for Transformer-based visual relation detection.
π― What it does: Using 3D Gaussian Splatting for inverse rendering to recover the geometry, materials, and lighting of a scene, generating high-quality new view renderings and relighting from multi-view images.
CodeObject DetectionSegmentationTransformerLarge Language ModelPrompt EngineeringImageMultimodality
π― What it does: A multimodal large language model framework named GSVA is proposed, capable of simultaneously handling multi-object segmentation and empty object recognition in the General Reference Expression Segmentation (GRES) task.
π― What it does: This paper proposes a Hierarchical Visual Transformer (H-ViT) for deformable image registration, which encodes the deformation field through a dual attention mechanism.
π― What it does: A high-quality, artificially designed 3D indoor scene dataset HSSD-200 was constructed, and it was used to evaluate the generalization ability of navigation agents in real environments.
CodeGenerationData-Centric LearningTransformerLarge Language ModelVision Language ModelImageMultimodality
π― What it does: This paper proposes a cross-checking framework named HalluciDoctor, which is used to automatically detect and eliminate hallucination toxicity in machine-generated visual instruction data, and enhance the robustness of multimodal large language models through adversarial visual instruction augmentation.
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
Chaoya Jiang (National Engineering Research Center for Software Engineering Peking University), Shikun Zhang (National Engineering Research Center for Software Engineering Peking University)
CodeRepresentation LearningTransformerLarge Language ModelContrastive LearningImageTextMultimodality
π― What it does: This paper proposes a new training method for multimodal large language modelsβHallucination Augmented Contrastive Learning (HACL), which enhances the alignment of visual and language representations and suppresses hallucination generation by incorporating hallucination descriptions generated by GPT-4 as hard negative samples in contrastive learning between vision and text.
HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
Tianrui Guan (University of Maryland), Tianyi Zhou (University of Maryland)
CodeRecognitionData SynthesisTransformerLarge Language ModelVision Language ModelImageMultimodalityBenchmark
π― What it does: A benchmark called HallusionBench is proposed to systematically evaluate large visual language models in terms of language hallucinations and visual illusions in image context reasoning.
Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID
Wentan Tan, Dapeng Tao (Yunnan University)
CodeRetrievalTransformerLarge Language ModelPrompt EngineeringImageTextMultimodality
π― What it does: This paper constructs a large-scale transferable human retrieval dataset by automatically generating text descriptions using a multimodal large language model and combining noise suppression techniques, significantly improving cross-domain text retrieval performance.
π― What it does: A Transformer model named HEAL-SWIN is proposed, which maps high-resolution fisheye images onto a sphere using HEALPix equal-area equidistant pixel grids, and performs tasks such as semantic segmentation, depth estimation, and classification based on this mapping.
Mason Long Wang (Stanford University), Jiajun Wu (Stanford University)
CodeDiffusion modelAudio
π― What it does: This paper presents the DIFFRIR framework, which synthesizes realistic spatial audio at any room position using a small number of RIR measurements and simple geometric models.
π― What it does: We propose HiT-ADV, a 3D point cloud adversarial attack method based on shape deformation, which can generate effective perturbations for the model without producing obvious outliers and is almost imperceptible to the human eye.
π― What it does: Proposes the HiKERβSGG method, which utilizes hierarchical knowledge graphs and layered reasoning to achieve robust scene graph generation, and constructs the VGβC (20 types of distortion) benchmark.
π― What it does: This paper proposes a method for generating 3D clothed human models from single-view color images using a parametric human model that utilizes high-frequency and low-frequency information, called HiLo.
HINTED: Hard Instance Enhanced Detector with Mixed-Density Feature Fusion for Sparsely-Supervised 3D Object Detection
Qiming Xia (Xiamen University), Cheng Wang (Xiamen University)
CodeObject DetectionAutonomous DrivingPoint Cloud
π― What it does: This paper proposes a Hard Instance Enhanced Detector (HINTED) for sparse supervised 3D object detection, which significantly improves detection performance in sparsely labeled scenarios through a Self-Boosting Teacher (SBT) and Mixed Density Student (MDS) framework.
Wenrui Cai (Beihang University), Yunhong Wang (Beihang University)
CodeObject TrackingTransformerVideo
π― What it does: This paper proposes the Historical Prompt Network and the HIPTrack tracker based on this network, which generates high-quality prompts by integrating precise foreground masks and visual features of historical targets within the traditional Siamese tracking framework, thereby improving tracking accuracy.
π― What it does: A framework for unsupervised hyperspectral image restoration based on a pre-trained diffusion model (HIR-Diff) is proposed. It decomposes hyperspectral images into low-dimensional images and coefficient matrices through low-rank decomposition. The coefficient matrix is first estimated using SVD+RRQR, and then the low-dimensional image is restored using an improved diffusion model with a total variation (TV) regularization guidance function, ultimately reconstructing a clear hyperspectral image.
π― What it does: This paper proposes an end-to-end network (HOISDF) that utilizes a global implicit Signed Distance Field (SDF) to guide 3D pose estimation of hand-object interactions.
π― What it does: This paper proposes the TeachCLIP framework, which incorporates an Attentional frame-Feature Aggregation (AFA) block into CLIP4Clip, enabling the distillation of fine-grained cross-modal knowledge from a heavy teacher model to a lightweight student model for efficient text-video retrieval.
HomoFormer: Homogenized Transformer for Image Shadow Removal
Jie Xiao (University of Science and Technology of China), Zheng-Jun Zha (University of Science and Technology of China)
CodeRestorationTransformerImage
π― What it does: This paper studies a local window Transformer (HomoFormer) that achieves spatial homogenization using random shuffling and inverse shuffling for high-resolution image shadow removal.
π― What it does: A pluggable pruning and recovery framework called Hourglass Tokenizer (HoT) is proposed to enhance the efficiency of transformer models in video-based 3D joint pose estimation.
π― What it does: A Context-based NeRF Compression (CNC) framework is proposed, which compresses the multi-resolution hash features of Instant-NGP through a context model, significantly reducing storage requirements.
Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation
Feilong Tang (Monash University), Zongyuan Ge (Monash University)
CodeSegmentationConvolutional Neural NetworkImage
π― What it does: A Context PrototypeβAware Learning (CPAL) method is designed and implemented for weakly supervised semantic segmentation, enhancing the integrity and accuracy of CAM through improved category prototype awareness.
π― What it does: This paper views the DETR series as an improvement of the RPN-refiner in Faster R-CNN, systematically analyzing the key improvement factors of DETR and proposing the Hybrid Proposal Refiner (HPR), which integrates various proposal refinement modules into a unified framework.
Yancong Lin (Delft University of Technology), Holger Caesar (Delft University of Technology)
CodeAutonomous DrivingOptimizationSimultaneous Localization and MappingOptical FlowPoint Cloud
π― What it does: This paper proposes an unsupervised LiDAR scene flow estimation method called ICP-Flow, which is completely based on ICP. It segments the point cloud into clusters using the rigid body motion assumption, registers the clusters, and recovers point-level scene flow through inter-cluster rigid transformations.
π― What it does: This paper proposes an implicit diffusion-based deblurring method called ID-Blau, which utilizes pixel-level continuous blur condition maps (direction and magnitude) to transform sharp images into controllable blurred images, thereby achieving diversified data augmentation.
ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection
Yichen Bai (Tianjin University), Changqing Zhang (Tianjin University)
CodeAnomaly DetectionPrompt EngineeringVision Language ModelContrastive LearningImage
π― What it does: This work proposes a CLIP-based ID-like Prompt learning framework for detecting out-of-distribution (OOD) samples in few-shot scenarios.
CodeClassificationExplainability and InterpretabilityComputational EfficiencyTransformerImage
π― What it does: A pixel visualization method based on game-theoretic interaction, MoXI, is proposed to efficiently identify the set of pixels that most significantly affect the confidence of image classifiers.
π― What it does: A residual pyramid network based on iterative reasoning (IIRP-Net) is proposed for unsupervised medical image deformation registration.
π― What it does: Using text-guided diffusion models to synthesize images with diverse backgrounds, textures, and materials, we constructed the ImageNet-D test set to evaluate the robustness of visual models.
Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation
Sixian Zhang (Chinese Academy of Sciences), Shuqiang Jiang (Chinese Academy of Sciences)
CodeGenerationRobotic IntelligenceTransformerLarge Language ModelReinforcement LearningMultimodality
π― What it does: A self-supervised generative mapping (SGM) framework is proposed, which utilizes local observations and the general knowledge of large language models to complete unobserved semantic map areas, thereby providing more complete environmental information for the Object Navigation (ObjectNav) task.
π― What it does: The paper proposes a two-stage task decomposition method (Task Decomposition, TaDe) for semantic segmentation from monocular images to bird's eye view (BEV). It first uses a BEV autoencoder to learn BEV structural knowledge, then maps RGB images to the BEV latent space for alignment, and finally utilizes a frozen BEV decoder to obtain segmentation results.
π― What it does: A Generalized Zero-Shot Learning (GZSL) framework DSECN is proposed, which enhances the recognition of similar and dissimilar unknown categories by introducing diverse semantics of external category names and aligning them to the visual space.
π― What it does: In the framework of Online Continual Learning, a strategy based on Collaborative Learning and Distillation Chain (CCL-DC) is proposed, aiming to significantly enhance the model's plasticity and final accuracy.
π― What it does: A single-source domain generalization method for object detection is proposed by enhancing visual corrosion on a single source domain and aligning the detection results (classification and localization) for consistency between the enhanced images and the original images.
π― What it does: A Spectral-Spatial Rectification (SSR) method is proposed, enhancing the quality of spectral snapshot reconstruction using Window Self-Attention (WSSA) and Spatial Alignment Blocks (ARB).
π― What it does: A hierarchical variational autoencoder (VAE) training framework based on reinforcement learning is proposed, aimed at eliminating posterior collapse and learning interpretable hierarchical representations.
π― What it does: The Hi-Mapper module is proposed, which constructs a visual hierarchy through a probabilistic tree and maps the hierarchy to hyperbolic space, ultimately enhancing the structured representation capability of pre-trained DNNs.
π― What it does: This study explores the possibility of transforming data to accelerate neural field training and finds that random pixel permutation can significantly improve training speed.
π― What it does: The task of 'in-context matting' is proposed, which automatically generates alpha masks for a set of target images of the same category using a single reference image and its point/brush/mask prompts, and develops the IconMatting model based on Stable Diffusion;
In-distribution Public Data Synthesis with Diffusion Models for Differentially Private Image Classification
Jinseong Park (Seoul National University), Jaewook Lee (Seoul National University)
CodeClassificationGenerationData SynthesisSafty and PrivacyDiffusion modelImage
π― What it does: This paper proposes the use of diffusion models to synthesize in-distribution (ID) public data, which is then used as warm-up data for differential privacy (DP) image classification, significantly improving the accuracy of private training.
In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging
Xin Wang (Beijing Institute of Technology), Hua Huang (Beijing Normal University)
CodeRestorationTransformerImage
π― What it does: This paper proposes In2SET, a Transformer-based DCCHI hyperspectral image reconstruction network that utilizes the internal and cross-modal similarities of panchromatic images to enhance reconstruction quality.
π― What it does: A new CNN architecture called InceptionNeXt is proposed, which utilizes a four-branch Inception depthwise separable convolution (small square convolution, horizontal/vertical strip convolution, and identity mapping) to replace traditional large-kernel depthwise separable convolution, significantly improving training/inference throughput while maintaining or enhancing accuracy.
Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation
Huyong Wang (Shenzhen University), Jing Qin (The Hong Kong Polytechnic University)
CodeSegmentationKnowledge DistillationImageBiomedical Data
π― What it does: Incremental nucleus segmentation in clinical pathological images addresses the problem of catastrophic forgetting without saving old class samples.
π― What it does: A temporal dependency classifier (TDC) is proposed, which utilizes the semantic context of historical frames to enhance classification performance in semi-supervised learning for video semantic segmentation.
π― What it does: A scale and position-sensitive loss function SLS is proposed, combined with a simplified U-Net model with multi-scale heads, MSHNet, to improve the accuracy of infrared small target detection.
π― What it does: This study investigates the adversarial robustness in transfer learning and proposes a method to enhance the robustness of downstream tasks through Adversarial Linear Initialization (RoLI).
π― What it does: The Initial Noise Optimization (INITNO) method is proposed, which enhances the performance of text-to-image diffusion models in adhering to prompts by optimizing noise in the initial latent space.
Instance Tracking in 3D Scenes from Egocentric Videos
Yunhan Zhao (University of California Irvine), Charless Fowlkes (University of California Irvine)
CodeObject TrackingSimultaneous Localization and MappingVideoMultimodalityBenchmark
π― What it does: The IT3DEgo benchmark task and dataset are proposed, focusing on tracking target instances from a first-person perspective (Egocentric) camera in real indoor 3D scenes.
Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation
Shenshen Bu (Sun Yat-sen University), Zhiming Dai (Sun Yat-sen University)
CodeGenerationKnowledge DistillationTransformerImageTextMultimodalityElectronic Health Records
π― What it does: This paper proposes a radiology report generation framework called EKAGen, which is based on instance-level expert knowledge and aggregated discriminative attention. It utilizes a unified embedding network to extract expert report knowledge, diagnose images, and retrieve corresponding instance-level knowledge, while focusing on key lesion areas through weakly supervised aggregated discriminative attention maps (ADM) and enhancing the model's global semantic consistency using a Global Information Self-Distillation (GID) strategy.
π― What it does: This study investigates an instance-level control text-to-image diffusion model that supports various positional formats and instance descriptions, generating images that conform to specified instance attributes and layouts.
π― What it does: A unified instruction-driven person re-identification (Instruct-ReID) task is proposed, where the model accepts images and multimodal instructions as input, achieving a unified deployment for various re-identification scenarios including traditional, outfit change, template outfit change, infrared-visible, text retrieval, and language instructions.
π― What it does: An adaptive focusing framework based on Spike Dispersion (SD) is proposed, along with a real-time focus measurement and adaptive pyramid search algorithm SGFS for peak cameras, which does not require image reconstruction.
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Zhe Chen (OpenGVLab), Jifeng Dai (University of Science and Technology of China)
CodeClassificationRecognitionSegmentationGenerationRetrievalTransformerLarge Language ModelVision Language ModelContrastive LearningImageVideoTextMultimodality
π― What it does: We propose InternVL, a general vision-language foundation model that extends the visual encoder to 6 billion parameters and aligns with large language models, applicable to multimodal tasks such as image and video recognition, retrieval, captioning, and dialogue.