IEEE/CVF Conference on Computer Vision and Pattern Recognition · 851 papers
Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion
Jiuhai Chen (University of Maryland), Bin Xiao (Microsoft Research)
CodeRecognitionGenerationRetrievalTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: This paper presents Florence-VL, a novel multimodal large language model that uses the Florence-2 generative visual foundation model as a visual encoder and projects the fused visual features from multiple levels and different task prompts to the LLM through Depth-Breadth Fusion (DBFusion).
🎯 What it does: This paper proposes the Forensics Adapter, which combines CLIP with a lightweight adapter tailored for specific task objectives to enhance the generalization ability of facial forgery detection.
Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models
Jin Wang (University of Hong Kong), Ping Luo (Shanghai AI Laboratory)
CodeTransformerVision Language ModelGenerative Adversarial NetworkImageVideoTextMultimodalityBenchmark
🎯 What it does: A Forensics-Bench benchmark has been constructed with a scale of 63k samples, covering 112 types of forgery detection, to evaluate the capabilities of large visual language models (LVLM) in forgery detection.
🎯 What it does: A LiDAR forest scene pose recognition method called ForestLPR is proposed, which is based on multi-layer BEV density maps and Transformers. It can adaptively focus on features at different heights, achieving rotation-invariant global descriptors.
Forming Auxiliary High-confident Instance-level Loss to Promote Learning from Label Proportions
Tianhao Ma (Jilin University), Ximing Li (Jilin University)
CodeClassificationImage
🎯 What it does: This paper studies how to effectively utilize pseudo-labels for instance-level learning in label-proportion weakly supervised learning (LLP) and proposes a high-confidence instance-level loss method to enhance classification performance.
🎯 What it does: In the multimodal crowd counting task, the authors propose an enhancement strategy that does not require additional data, parameters, or inference time consumption, improving existing models through post-pretraining cross-modal alignment (PPCA) and regional density supervision (RDS).
FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity
Jinxi Li (Hong Kong Polytechnic University), Bo Yang (Hong Kong Polytechnic University)
CodeGaussian SplattingVideoPhysics Related
🎯 What it does: We propose FreeGave, a framework that directly learns the geometry, appearance, and physical motion (velocity field) of dynamic 3D scenes from multi-view videos.
FreePCA: Integrating Consistency Information across Long-short Frames in Training-free Long Video Generation via Principal Component Analysis
Jiangtong Tan (University of Science and Technology of China), Feng Zhao (University of Science and Technology of China)
CodeGenerationData SynthesisDiffusion modelVideo
🎯 What it does: FreePCA is proposed, a training-free long video generation framework that decouples temporal feature dimensions through PCA, separating consistent appearance features from motion intensity features, and gradually fusing them to balance consistency and quality.
🎯 What it does: A frequency dynamic convolution (FDConv) module is designed and implemented to enhance the adaptability of visual tasks to different frequency information while keeping the parameter overhead low.
🎯 What it does: The FRESA method is proposed, which utilizes a small number of snapshot images to achieve real-time reconstruction and animation inference of personalized 3D skinned avatars.
From Elements to Design: A Layered Approach for Automatic Graphic Design Composition
Jiawei Lin (Xi'an Jiaotong University), Jiang Bian (Microsoft Research)
CodeGenerationTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelMultimodality
🎯 What it does: A framework for automatic graphic design generation called LaDeCo is proposed, which can plan and generate layouts and attributes layer by layer according to the semantic hierarchy of multimodal elements provided by users, ultimately synthesizing high-quality designs.
🎯 What it does: This study investigates a system that generates high-quality speech from silent talking videos, utilizing hierarchical visual encoding and flow-matching decoders for audio generation.
From Laboratory to Real World: A New Benchmark Towards Privacy-Preserved Visible-Infrared Person Re-Identification
Yan Jiang (Nanjing University of Information Science and Technology), Guoying Zhao (University of Oulu)
CodeRecognitionFederated LearningSafty and PrivacyImageBenchmark
🎯 What it does: This paper proposes the L2RW benchmark, simulating privacy-preserving visible-infrared pedestrian re-identification scenarios in the real world.
🎯 What it does: A training-free portrait re-identification framework called Pose2ID is proposed, which suppresses noise and enhances identity representation by clustering features of the same identity.
🎯 What it does: Compress the traditional bidirectional diffusion model into a four-step causal generator to achieve low-latency real-time video generation.
FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding
Rong Gao (Lappeenranta-Lahti University of Technology), Heikki Kälviäinen (Lappeenranta-Lahti University of Technology)
CodeClassificationRecognitionSegmentationTransformerLarge Language ModelSupervised Fine-TuningPrompt EngineeringVideoTextMultimodalityBenchmarkAudio
🎯 What it does: This paper presents the FSAnno dataset and the FSBench benchmark, constructing a multimodal, multi-level skating dataset that covers technical actions and artistic expressions, and designs a multi-task evaluation framework from prior knowledge to overall assessment.
🎯 What it does: This paper proposes a fully sparse 3D object detection network called FSHNet, which integrates sparse convolution with SlotFormer attention to achieve global interaction, and introduces dynamic sparse label assignment and sparse upsampling to enhance detection accuracy.
🎯 What it does: This paper proposes a cross-modal retrieval method based on fuzzy set theory, called FUME, which can adaptively estimate and utilize uncertainty information to enhance retrieval credibility during the search process.
GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion
Jiapeng Tang (Technical University of Munich), Matthias Nießner (Technical University of Munich)
CodeGenerationData SynthesisDiffusion modelVideo
🎯 What it does: Using a multi-view diffusion model and 3D Gaussian light scattering technology, we reconstruct animatable photorealistic 3D Gaussian avatars of human faces from monocular videos.
GauCho: Gaussian Distributions with Cholesky Decomposition for Oriented Object Detection
José Henrique Lima Marques (Federal University of Rio Grande do Sul), Claudio R. Jung (Federal University of Rio Grande do Sul)
CodeObject DetectionImage
🎯 What it does: A Gaussian distribution-based Cholesky decomposition regression head, GauCho, is proposed for directional object detection, directly regressing Gaussian parameters instead of traditional OBB parameters.
🎯 What it does: A two-stage framework (GaussianIP) is proposed, capable of quickly generating identity-preserving, detail-rich facial features, and finely textured clothing for 3D human avatars based on text and portrait prompts.
Gazing at Rewards: Eye Movements as a Lens into Human and AI Decision-Making in Hybrid Visual Foraging
Bo Wang (Nanyang Technological University), Mengmi Zhang (Nanyang Technological University)
CodeTransformerReinforcement LearningImage
🎯 What it does: Developed and trained a Transformer-based Visual Forager (VF) for multi-object value mixed visual foraging tasks, and validated its performance through human eye-tracking experiments.
Brayan Monroy (Universidad Industrial de Santander), Julián Tachella (CNRS)
CodeRestorationImage
🎯 What it does: This paper proposes a general self-supervised image restoration method GR2R, which can achieve unsupervised denoising under any natural exponential family noise.
🎯 What it does: A zero-shot point cloud completion framework called GenPC is proposed, which utilizes a pre-trained 3D generative model to complete missing point clouds from real scans.
Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learning
Yanbiao Ma (Xidian University), Jiayi Chen (Xidian University)
CodeDomain AdaptationFederated LearningImage
🎯 What it does: In federated learning, the GGEUR (Global Geometry-Guided Embedding Uncertainty Representation) method is proposed, which utilizes the geometric shape of the global embedding distribution to guide local data augmentation, thereby simulating the ideal global distribution on the client side.
Nikola Zubic (University of Zurich), Davide Scaramuzza (University of Zurich)
CodeAutonomous DrivingOptimizationGraph Neural NetworkOptical FlowImageGraphTime Series
🎯 What it does: A graph generation state space model (GG-SSM) is proposed, which captures long-range dependencies in high-dimensional data by dynamically constructing a minimum spanning tree graph.
🎯 What it does: The GIVEPose framework is proposed, which achieves more accurate pose prediction by gradually eliminating intra-class variations in category-level object pose estimation.
Global-Local Tree Search in VLMs for 3D Indoor Scene Generation
Wei Deng (Beijing University of Posts and Telecommunications), Huadong Ma (Beijing University of Posts and Telecommunications)
CodeGenerationData SynthesisTransformerLarge Language ModelVision Language ModelTextPoint Cloud
🎯 What it does: A global-local tree search method is proposed, utilizing a large visual language model (VLM) to generate realistic 3D indoor scene layouts through hierarchical scene representation.
Hyungyu Choi (Chung Ang University), Chanho Eom (Chung Ang University)
CodeRetrievalTransformerSupervised Fine-TuningVision Language ModelContrastive LearningImageText
🎯 What it does: A fine-tuning framework called GOAL is proposed to refine CLIP, enhancing its local and global alignment capabilities in long text and image retrieval.
GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving
Zebin Xing (University of Chinese Academy of Sciences), Wei Yin (Horizon Robotics)
CodeGenerationAutonomous DrivingTransformerRectified FlowMultimodalityTime Series
🎯 What it does: This paper proposes GoalFlow, an end-to-end automatic driving multimodal trajectory generation method that generates high-quality trajectories using precise target point constraints.
🎯 What it does: We propose GoLF-NRT, a general neural rendering Transformer that integrates global context and local geometry to address the decline in rendering quality of NeRF under a limited number of viewpoints.
🎯 What it does: In federated learning, the authors propose PEFTLeak, which maliciously designs pre-trained models and adapter modules to recover user local fine-tuning data using only lightweight adapter gradients.
Gradient-Guided Annealing for Domain Generalization
Aristotelis Ballas (Harokopio University of Athens), Christos Diou (Harokopio University of Athens)
CodeDomain AdaptationImage
🎯 What it does: This paper proposes an early parameter annealing strategy based on gradient consistency (Gradient-Guided Annealing, GGA), which enhances the model's generalization ability to unseen domains by searching for parameter points with similar gradients in the early stages of training.
🎯 What it does: This paper proposes an online 3D multi-object tracking method based on geometric relationships, GRAE-3DMOT, which aggregates detections using a relationship graph without distance thresholds and completes trajectory association.
GroupMamba: Efficient Group-Based Visual State Space Model
Abdelrahman Shaker (Mohamed Bin Zayed University of Artificial Intelligence), Fahad Shahbaz Khan (Mohamed Bin Zayed University of Artificial Intelligence)
🎯 What it does: Proposed the Modulated Group Mamba layer and a distillation-based training objective to build a stable and efficient visual SSM network.
🎯 What it does: This paper presents the GUI-Xplore dataset and the Xplore-Agent framework, aimed at enhancing the GUI agent's capabilities across applications and tasks.
🎯 What it does: For open-world task incremental learning (TIL), a hierarchical two-sample test (H2ST) method is proposed that does not require a threshold, aimed at continuously detecting and identifying out-of-distribution (OOD) samples and their task IDs, thereby enabling automatic learning of new tasks.
🎯 What it does: The FedTA method is proposed, utilizing a frozen pre-trained ViT and incorporating learnable input augmentation and Tail Anchor to address catastrophic forgetting caused by spatial-temporal heterogeneity in federated continual learning.
Samuel Rota Bulò (Meta Reality Labs), Peter Kontschieder (Meta Reality Labs)
CodeGaussian SplattingPoint CloudBenchmark
🎯 What it does: A hardware rasterization-based Ray-based 3D Gaussian splatting renderer is proposed, significantly improving the real-time rendering speed and quality of VR/MR scenes.
Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models
Zhenguang Liu (Zhejiang University), Kui Ren (Zhejiang University)
CodeGenerationData SynthesisSafty and PrivacyDiffusion modelAuto EncoderImage
🎯 What it does: CoprGuard is proposed, a watermarking framework based on spectral features, designed to detect and prevent image copyright infringement during the training and fine-tuning of diffusion models.
🎯 What it does: The GoodAC framework is proposed, which generates strong robust adversarial perturbations by utilizing global feature correlation disruption and local facial attribute distortion to prevent Latent Diffusion Models (such as Stable Diffusion) from personalized customization.
🎯 What it does: A method for hiding images in diffusion models is proposed, which achieves implicit embedding and extraction of secret images by editing the learned scoring function at specific time steps during the reverse diffusion process.
🎯 What it does: A hierarchical parameterized dataset distillation method H-PD is proposed, which improves data distillation performance at extreme compression rates by optimizing the generation of synthetic datasets layer by layer in the multi-layer feature space of GAN.
🎯 What it does: The first large-scale HDR video dataset HDRVD2K has been constructed, and a learning-based bit-depth scalable video compression network LBSVC has been proposed.
🎯 What it does: A lightweight video semantic segmentation framework SSP (Semantic Similarity Propagation) is proposed, which enhances the temporal consistency of drone videos by performing linear interpolation between the current image model prediction and the prediction aligned with the previous frame for each frame.
🎯 What it does: A controllable lighting video diffusion model LCVD is proposed, achieving high-fidelity re-lighting portrait animation from a single reference portrait under given lighting and target video.
HIIF: Hierarchical Encoding based Implicit Image Function for Continuous Super-resolution
Yuxuan Jiang (University of Bristol), David Bull (University of Bristol)
CodeRestorationSuper ResolutionImage
🎯 What it does: A new Hierarchical Implicit Image Function (HIIF) based on hierarchical encoding is proposed for continuous image super-resolution, capable of capturing details at multiple scales.
🎯 What it does: What was done: A semi-supervised learning-based LiDAR point cloud semantic segmentation framework called HiLoTs was proposed, which significantly improves segmentation performance by utilizing high and low time-sensitive flows along with the Mean Teacher mechanism.
Pei Geng (Nanjing University of Science and Technology), Shanshan Zhang (Nanjing University of Science and Technology)
CodeRecognitionObject DetectionTransformerVision Language ModelImageMultimodality
🎯 What it does: A human-object interaction (HOI) detection method based on human-object relationship priors (HORP) is proposed, which enhances the recognition capability of the 'no interaction' category by integrating 3D position priors and human gaze area priors into multimodal queries.
🎯 What it does: To optimize the matrix multiplication in backpropagation during deep learning training, a Hadamard-based Optimized Training (HOT) scheme is proposed.
🎯 What it does: The authors constructed the HOT3D dataset, collecting 833 minutes and 1.5 million frames of multi-view first-person videos from 19 subjects using the Aria and Quest 3 head-mounted devices, and provided annotations for hand and object 3D poses, point clouds, gaze, and more.
Human-centered Interactive Learning via MLLMs for Text-to-Image Person Re-identification
Yang Qin (Sichuan University), Peng Hu (Sichuan University)
CodeRecognitionRetrievalTransformerLarge Language ModelVision Language ModelImageTextMultimodalityRetrieval-Augmented Generation
🎯 What it does: This paper proposes an interactive text-image person re-identification (TIReID) framework called ICL, which utilizes a multimodal large language model (MLLM) to achieve multi-turn question answering for refining query text, and combines data augmentation RDA to enhance the model's ability to discriminate fine-grained descriptions.
🎯 What it does: A new HVI color space based on polarized HS plane and a learnable intensity collapse function is proposed, along with a corresponding dual-branch CIDNet network for low-light image enhancement.
Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation
Ting Liu (Northwestern Polytechnical University), Siyuan Li (Northwestern Polytechnical University)
CodeSegmentationContrastive LearningImage
🎯 What it does: A training-independent method for zero-shot referential image segmentation is proposed, which combines global-local feature extraction with multi-spatial guidance, utilizing SAM to generate masks and aligning features through the CLIP visual/text encoder.
Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models
Zhihang Liu (University of Science and Technology of China), Hongtao Xie (University of Science and Technology of China)
CodeRecognitionCompressionTransformerLarge Language ModelPrompt EngineeringVideoMultimodality
🎯 What it does: A dual-layer instruction injection conditional compression method called HICom is proposed for the video understanding tasks of multimodal large language models, which significantly reduces visual tokens while retaining information relevant to user instructions.
Tobia Poppi (University of Modena and Reggio Emilia), Rita Cucchiara (Istituto Italiano di Tecnologia)
CodeRetrievalSafty and PrivacyVision Language ModelContrastive LearningImageText
🎯 What it does: Construct a hierarchical relationship between safe and unsafe content in hyperbolic space to achieve safety-aware CLIP (HySAC), supporting safe retrieval and controllable traversal.
HyperSeg: Hybrid Segmentation Assistant with Fine-grained Visual Perceiver
Cong Wei (Tsinghua University), Yujiu Yang (Meituan Inc.)
CodeObject DetectionSegmentationTransformerLarge Language ModelVision Language ModelImageVideo
🎯 What it does: We propose HyperSeg, a general pixel-level image and video segmentation framework based on Visual Large Language Models (VLLM), capable of handling a variety of tasks from conventional segmentation to complex reasoning segmentation.
🎯 What it does: This paper proposes a hyperspectral super-resolution fusion method based on a zero-shot guided diffusion model and neural spatial-spectral decomposition.
IceDiff: High Resolution and High-Quality Arctic Sea Ice Forecasting with Generative Diffusion Prior
Jingyi Xu (Fudan University), Lei Bai (Fudan University)
CodeGenerationData SynthesisTransformerDiffusion modelTime Series
🎯 What it does: The IceDiff framework is proposed, which first uses a Vision Transformer to generate sea ice concentration predictions at a 25 km level, and then employs a pre-trained unconditional diffusion model and zero-shot sampling strategy to generate high-resolution sea ice concentration maps at a 6.25 km level.
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models
Junzhe Chen (Tsinghua University), Xuming Hu (Hong Kong University of Science and Technology)
CodeRecognitionObject DetectionComputational EfficiencyTransformerVision Language ModelImageTextMultimodality
🎯 What it does: This paper proposes a training-free, plug-and-play intervention method called ICT, which enhances visual attention to overall scenes and fine-grained objects by adjusting the activation values of attention heads during the forward propagation phase of LVLM, reducing object misreporting caused by the model's language bias.
🎯 What it does: To address the pseudo-correlation between tasks in multi-task learning, we first identify pseudo-related tasks by calculating the differences in correlation coefficients of task labels on the training set and the re-sampled training set by category. Then, we perform debiased adversarial training on each task, ensuring that the predictor for that task no longer utilizes information from other tasks identified as pseudo-related, thereby enhancing the model's generalization ability.
🎯 What it does: Designed the IDOL model and the HuGe100K dataset, achieving fast generation of animatable high-fidelity 3D human figures from a single image;
IDProtector: An Adversarial Noise Encoder to Protect Against ID-Preserving Image Generation
Yiren Song (National University of Singapore), Mike Zheng Shou (National University of Singapore)
CodeRecognitionGenerationSafty and PrivacyTransformerImage
🎯 What it does: A noise encoder based on Vision Transformer has been designed and trained to add small, invisible adversarial perturbations to portrait photos, thereby preventing Encoder-based identity-preserving image generation models (such as InstantID, IP-Adapter, PhotoMaker, etc.) from accurately reproducing the identity of the portraits.
🎯 What it does: A diversity evaluation metric IRS based on image retrieval is proposed, revealing the shortcomings of existing generative models in terms of diversity and introducing a diversity-aware diffusion model DiADM.
CodeSegmentationRetrievalLarge Language ModelContrastive LearningImageMultimodality
🎯 What it does: This paper proposes an image quality assessment (IQA) framework that transitions from the human visual system (HVS) to the machine visual system (MVS), and constructs a large-scale database focused on machine preferences, referred to as MPD, for the first time.
Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement
Yuchen Ren (Xi'an Jiaotong University), Chao Shen
CodeAdversarial AttackTransformerImage
🎯 What it does: By diversifying the attention maps (AMD) during the forward propagation of the Vision Transformer and applying momentum smoothing to token embeddings (MTE), a Forward Propagation Refinement (FPR) method is proposed to enhance the transfer attack capability of adversarial samples.
Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction
Teng Hu (Shanghai Jiao Tong University), Lizhuang Ma (Shanghai Jiao Tong University)
CodeGenerationTransformerLarge Language ModelImage
🎯 What it does: An improved autoregressive visual generation method IAR is proposed, which enhances the generation quality of LLM by utilizing code cluster rearrangement and clustering-guided cross-entropy loss.
🎯 What it does: A quality-aware dynamic discriminator rejection sampling (QADDRS) method is proposed, which dynamically rejects high-scoring real samples and low-scoring fake samples during the data-efficient GAN training process to alleviate discriminator overfitting and improve generation quality.
Improving Transferable Targeted Attacks with Feature Tuning Mixup
Kaisheng Liang (Hong Kong Polytechnic University), Bin Xiao (Hong Kong Polytechnic University)
CodeAdversarial AttackConvolutional Neural NetworkTransformerMixture of ExpertsImage
🎯 What it does: This paper proposes a new feature-level attack method called Feature Tuning Mixup (FTM), which enhances the transferability of targeted attacks by adding learnable perturbations in the intermediate layers and mixing them with randomly clean features.
IncEventGS: Pose-Free Gaussian Splatting from a Single Event Camera
Jian Huang (Zhejiang University), Peidong Liu (Westlake University)
CodeObject DetectionPose EstimationDepth EstimationGaussian SplattingSimultaneous Localization and MappingImagePoint Cloud
🎯 What it does: An incremental 3D Gaussian spraying (IncEventGS) method for monocular event cameras is proposed, capable of achieving 3D scene reconstruction and camera motion estimation without prior camera pose information.
🎯 What it does: This paper proposes an ANN-SNN conversion framework that can be completed solely during the inference stage, allowing for the direct conversion of pre-trained ANN models into high-performance, low-power SNNs.
Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis
Jian Han (ByteDance), Xiaobing Liu (ByteDance)
CodeGenerationData SynthesisTransformerImageText
🎯 What it does: Infinity is proposed, a visual autoregressive model based on bit quantization, utilizing an infinite vocabulary classifier and a bit self-correction mechanism to achieve high-resolution (1024×1024) text-to-image generation, leading in both speed and quality.
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
Yuhao Dong (Nanyang Technological University), Ziwei Liu (Nanyang Technological University)
CodeGenerationData SynthesisOptimizationTransformerLarge Language ModelReinforcement LearningAgentic AIVision Language ModelTextMultimodality
🎯 What it does: The Insight-V system is proposed, specifically designed to enhance the long-chain visual reasoning capabilities of multimodal large language models (MLLMs), utilizing a two-stage data generation and multi-agent training process.
CodeRecognitionObject DetectionSegmentationRetrievalComputational EfficiencyTransformerLarge Language ModelVision Language ModelMultimodalityPoint Cloud
🎯 What it does: A unified instance-aware large multimodal model, Inst3D-LMM, is proposed, capable of performing various tasks such as 3D visual localization, question answering, and dense description without the need for separate fine-tuning for each task.
🎯 What it does: An instance-level supervised layer optimization active learning framework, ISO, has been developed to dynamically decide the full annotation or weak annotation for each instance under a fixed budget.
InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption
Tiehan Fan (Nanjing University), Ying Tai (Nanjing University)
CodeGenerationData SynthesisTransformerLarge Language ModelPrompt EngineeringVideoTextMultimodalityChain-of-Thought
🎯 What it does: This paper proposes the InstanceCap framework, which achieves instance-aware structured video subtitle generation and applies it to text-to-video generation.
🎯 What it does: This paper proposes a self-supervised Instruct-CLIP method that utilizes contrastive learning to semantically align image pairs (original images and edited images) with their corresponding editing instructions, thereby improving and expanding the existing instruction-driven image editing dataset. Based on this, InstructPix2Pix is fine-tuned, significantly enhancing editing quality.
Interactive Medical Image Analysis with Concept-based Similarity Reasoning
Ta Duc Huy (Australian Institute for Machine Learning), Vu Minh Hieu Phan (Australian Institute for Machine Learning)
CodeClassificationExplainability and InterpretabilityConvolutional Neural NetworkContrastive LearningImageBiomedical Data
🎯 What it does: A Concept Similarity Reasoning Network (CSR) is proposed, achieving concept-level explanations and interactivity in medical image classification.
Jun Gao (Soochow University), Wenjie Li (Hong Kong Polytechnic University)
CodeGenerationData SynthesisTransformerPrompt EngineeringVision Language ModelMultimodalityChain-of-Thought
🎯 What it does: This paper proposes an Interleaved Chain of Thought for Multimodal Reasoning (ICoT), which allows visual language models to generate intermediate reasoning chains that simultaneously contain visual segments and textual reasoning steps during the reasoning process.
Interpretable Image Classification via Non-parametric Part Prototype Learning
Zhijie Zhu (University of New South Wales), Yang Song (University of New South Wales)
CodeClassificationExplainability and InterpretabilityTransformerContrastive LearningImage
🎯 What it does: This paper proposes a part-level interpretable image classification framework based on non-parametric prototype learning, utilizing ViT feature clustering to obtain diversified part prototypes for each category, which are directly used for classification.
Inversion Circle Interpolation: Diffusion-based Image Augmentation for Data-scarce Classification
Yanghao Wang (Hong Kong University of Science and Technology), Long Chen (Hong Kong University of Science and Technology)
CodeClassificationGenerationData SynthesisVision Language ModelDiffusion modelImage
🎯 What it does: This paper proposes an augmentation method based on diffusion models, Diff-II, which utilizes category concept learning, image inverse interpolation, and two-stage denoising to generate synthetic training samples that are both faithful and diverse, thereby improving classification performance in data-scarce scenarios.
🎯 What it does: Designed and implemented a stealthy backdoor attack in self-supervised learning (SSL) that can induce downstream classifiers to produce specified target labels while maintaining normal model performance.
Is `Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning
Ji Hyeok Jung (Sogang University), Buru Chang (Korea University)
CodeRecognitionObject DetectionTransformerLarge Language ModelSupervised Fine-TuningImageMultimodalityBenchmark
🎯 What it does: This paper proposes a method called 'egocentric instruction tuning', which refines the understanding of direction in multimodal large language models (MLLMs) to interpret the orientation of objects in images from a user-centered perspective. It also creates the EgoOrientBench benchmark to evaluate the direction recognition capabilities of MLLMs across three tasks (Choose, Verify, Freeform) and five datasets.
🎯 What it does: This paper proposes a code decoding framework based on an iterative predictor-critic, utilizing a high-quality VQGAN codebook to progressively dehaze real scene fog images.
IterIS: Iterative Inference-Solving Alignment for LoRA Merging
Hongxu Chen (University of Science and Technology of China), Long Chen (Hong Kong University of Science and Technology)
CodeOptimizationFederated LearningSafty and PrivacyTransformerLarge Language ModelVision Language ModelDiffusion modelImageText
🎯 What it does: A LoRA merging algorithm called IterIS based on an iterative reasoning-solving framework is designed to merge multi-task LoRA into a unified adapter while ensuring data privacy.
🎯 What it does: A black-box attack method for non-transferable learning models, called JailNTL, is proposed, which uses data camouflage during testing to bypass non-transferability barriers.
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Chengyue Wu (DeepSeek-AI), Ping Luo (University of Hong Kong)
CodeGenerationData SynthesisTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: Designed and trained Janus, a unified multimodal model capable of simultaneous image understanding and image generation. The key point is the separation of visual encoding into two independent channels (understanding encoder and generation encoder), using the same autoregressive Transformer to uniformly process text and visual information.
JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
Yiyang Ma (Tsinghua University), Chong Ruan (Peking University)
CodeGenerationData SynthesisTransformerLarge Language ModelSupervised Fine-TuningRectified FlowImageTextMultimodality
🎯 What it does: We propose JanusFlow, a unified multimodal framework that combines autoregressive language models with rectified flow for image understanding and generation.
🎯 What it does: This paper proposes a method to jointly remove multimodal social biases in images and texts within the CLIP model, aiming to eliminate unfair inferences caused by sensitive attributes such as gender, age, and race.