IEEE/CVF Conference on Computer Vision and Pattern Recognition · 892 papers
Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology
Andrew H. Song (Mass General Brigham), Faisal Mahmood (Mass General Brigham)
CodeSegmentationRepresentation LearningBiomedical Data
🎯 What it does: A Gaussian Mixture Model-based unsupervised whole slide representation learning framework, PANTHER, is proposed, which constructs whole slide vectors using a small number of morphological prototypes.
🎯 What it does: This paper proposes a new 3D shape representation method called Mosaic-SDF, and based on it, trains a Flow Matching model to achieve high-quality 3D generation under class conditions and text conditions.
MoST: Motion Style Transformer Between Diverse Action Contents
Boeun Kim (Korea Electronics Technology Institute), Jin Young Choi (University of Birmingham)
CodeGenerationData SynthesisTransformerVideo
🎯 What it does: This paper proposes a motion style transfer framework named MoST, which can seamlessly transfer the style of source motion to target motion even when the source and target motion contents are inconsistent, without requiring any manually annotated style labels or post-processing steps.
🎯 What it does: A dual-view (fuzzy image + rolling shutter image) motion blur decomposition framework is proposed to address the temporal uncertainty in single-image blurry video reconstruction.
🎯 What it does: A two-stage language-guided human action generation framework is proposed, utilizing 3D scene affordance as an intermediate representation to achieve more accurate scene localization and action synthesis.
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Qinghao Ye (Alibaba Group), Fei Huang (Alibaba Group)
CodeTransformerLarge Language ModelVision Language ModelImageTextMultimodality
🎯 What it does: mPLUG-Owl2 is proposed, a multimodal large language model that integrates modality cooperation and modality adaptation modules, capable of unified performance in text and multimodal tasks;
🎯 What it does: MRC-Net is proposed, a single-stage, RGB image-based 6-DoF pose estimation network that achieves high-accuracy estimation through a sequential structure of classification followed by regression.
🎯 What it does: The MRFP (Multi-Resolution Feature Perturbation) module is proposed, which applies multi-resolution perturbations to the feature space during semantic segmentation training, significantly enhancing the generalization performance from simulation to real domains.
🎯 What it does: This paper proposes a coupled learning framework MRFS to simultaneously enhance the performance of infrared-visible image fusion and semantic segmentation.
🎯 What it does: MS-DETR is proposed based on DETR, significantly improving the quality of candidate boxes and training efficiency by adding one-to-many supervision to the queries of the main decoder.
🎯 What it does: The MTLoRA framework is proposed, achieving parameter-efficient fine-tuning in multi-task learning models through task-agnostic (TA-LoRA) and task-specific (TS-LoRA) low-rank adaptation modules.
🎯 What it does: Developed an untrained image layering pipeline and generated the MuLAn dataset, which includes multi-layer RGBA decompositions and instance occlusion information for 44,860 images.
🎯 What it does: A motion-aware robust communication network named MRCNet is proposed, aimed at enhancing the robustness and accuracy of multi-agent collaborative perception in the presence of real-world noise such as pose noise, perception noise, and motion blur.
🎯 What it does: This paper proposes a multi-attribute interactive Transformer that quantifies the causal effects of attributes on predictions through an attribute causal analysis module, and achieves fine-grained visual-language alignment for 3D visual localization tasks via an attribute interaction-driven cross-modal exchange fusion module.
🎯 What it does: This paper proposes a Multi-Criteria Token Fusion (MCTF) method that significantly reduces the number of tokens while maintaining the performance of visual Transformers.
Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer
Zhen Zhao (East China Normal University), Yuan Xie (East China Normal University)
CodeRecognitionTransformerLarge Language ModelImageTextMultimodality
🎯 What it does: A scene text recognition model named E STR is proposed, achieving rapid adaptation within a context learning framework without the need for additional fine-tuning.
Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering
Jiawei Yao (University of Washington), Juhua Hu (Alibaba Group)
CodeTransformerLarge Language ModelContrastive LearningMultimodality
🎯 What it does: A personalized multi-clustering method based on Multi-MaP (Multi-modal Proxy Learning) is proposed, which utilizes the CLIP encoder and GPT-4 to generate proxy words based on user keyword interests, and obtains the feature vectors required for clustering through similarity optimization.
🎯 What it does: A multi-scale dynamic and hierarchical relationship modeling (MDHR) framework is proposed for the precise recognition of facial action units (AUs) in videos.
Multi-Space Alignments Towards Universal LiDAR Segmentation
Youquan Liu (ShanghaiTech University), Yuexin Ma (ShanghaiTech University)
CodeSegmentationAutonomous DrivingVision Language ModelImageTextMultimodalityPoint Cloud
🎯 What it does: A general LiDAR segmentation framework named M3Net is proposed, capable of performing multi-task, multi-dataset, and multi-modal LiDAR semantic/panoptic segmentation under a single set of parameters.
Multi-Task Dense Prediction via Mixture of Low-Rank Experts
Yuqi Yang (Nankai University), Bo Li (vivo Mobile Communication Co., Ltd)
CodeObject DetectionSegmentationConvolutional Neural NetworkMixture of ExpertsImage
🎯 What it does: A decoder framework for multi-task dense prediction, MLoRE, is proposed, utilizing a Mixture-of-Low-Rank-Experts to achieve dynamic combination and global association of task features;
MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning
Matteo Farina (University of Trento), Elisa Ricci (University of Trento)
CodeGenerationRetrievalTransformerVision Language ModelMultimodality
🎯 What it does: A new task-agnostic visual-language model pruning method, called MULTIFLOW, is proposed, aiming to obtain a sparse model that can be transferred to multiple unknown downstream tasks through a single pruning.
🎯 What it does: This paper proposes a Multimodal Pathway Transformer (M2PT) that enhances the performance of unimodal Transformers on the target modality by using cross-modal unrelated data during the training phase and leveraging Cross-Modal Re-parameterization.
🎯 What it does: This paper proposes a single-stage action localization method called BMViT, which directly utilizes the spatiotemporal tokens output by Vision Transformers (such as MViTv2-S, ViT-B) to predict bounding boxes, actor confidence, and action categories through a simple MLP head, and trains the entire network using a bidirectional matching loss without the need for additional decoders or learnable queries.
🎯 What it does: An end-to-end multi-view point cloud stitching framework is proposed. First, high-precision dual point cloud registration is achieved through ODIN, followed by global pose rotation averaging, robust translation re-estimation, translation averaging, and diffusion graph optimization, ultimately unifying multiple partially overlapping point clouds into the same coordinate system.
🎯 What it does: This paper proposes a multi-view constrained illumination photometric stereo reconstruction method called MVCPS-NeuS, which utilizes the constraint of synchronized movement between the light source and the camera, combined with neural surface reconstruction to achieve high-precision 3D reconstruction under sparse views and lighting conditions.
🎯 What it does: This paper proposes StyleEntity, a model that utilizes named entities as training proxies to achieve zero-shot image editing, and further enhances stability during inference through Prompt Ensemble Latent Averaging (PELA).
🎯 What it does: A regularization strategy is proposed to enhance the generalization ability of blind image super-resolution models by aligning the first and second statistical moments of features from different degraded images of the same content.
CodeClassificationKnowledge DistillationConvolutional Neural NetworkLarge Language ModelImage
🎯 What it does: This paper proposes a data-independent knowledge distillation method called NAYER, which quickly synthesizes high-quality pseudo-samples for teacher-student distillation by transferring the noise source from the input to the noise layer and using label text embeddings generated by a pre-trained language model as input to the generator.
🎯 What it does: Proposes the NeRF Analogies method, which transfers the appearance of the source NeRF to the target geometry, generating a new 3D view-consistent NeRF.
🎯 What it does: A NeRF training method based on uncertainty prediction has been developed, which can automatically remove dynamic elements from 'wild' image sequences containing dynamic interfering objects and train high-quality static scene NeRF.
🎯 What it does: A visual feature extraction framework FEC based on neural clustering is proposed, treating the feature extraction process as an adaptive selection and updating of pixel representatives, replacing traditional grid-based partitioning.
🎯 What it does: This paper proposes a Neural Direction Encoding (NDE) that extends the spatial encoding of feature grids into the directional domain to better model high-frequency view-dependent reflections, such as those from smooth metals.
Guilherme Schardong (Institute of Systems and Robotics, University of Coimbra), Nuno Gonçalves (Institute of Systems and Robotics, University of Coimbra)
🎯 What it does: Utilizing a coordinate-based neural network to achieve implicit deformation and blending of facial images, completing continuous time facial modeling.
🎯 What it does: A completely data-driven Neural Markov Random Field (Neural MRF) framework is proposed for stereo matching, introducing a Disparity Proposal Network (DPN) to prune candidate disparity space at a coarse level, followed by refinement at a fine level.
🎯 What it does: A neural point cloud diffusion model is proposed, capable of generating the shape and appearance of objects in three-dimensional space, and supports independent control of shape and appearance.
🎯 What it does: This paper proposes a complete method for 4×4 super-resolution in real-time rendering, which first splits the rendered radiance into illumination and material components, performs super-resolution on the smooth illumination, and then overlaps it with high-resolution materials.
Jiahao Li (Microsoft Research), Yan Lu (Microsoft Research)
CodeCompressionOptical FlowVideo
🎯 What it does: A neural video compression model DCVC-FM based on conditional coding is proposed, utilizing feature modulation to achieve a single model with a wide quality range, long prediction chain processing, support for RGB/YUV color spaces, and low-precision inference.
🎯 What it does: NICE is proposed, a replay-free class-incremental learning architecture inspired by adult neurogenesis, capable of achieving zero forgetting and automatically locating the required subnetworks during testing.
Noisy One-point Homographies are Surprisingly Good
Yaqing Ding (Czech Technical University in Prague), Viktor Larsson (Lund University)
CodeOptimizationImageBenchmark
🎯 What it does: A minimum solver is proposed that can estimate the global homography matrix using only a single feature point with scale and orientation information, and it is integrated with existing multi-point solvers within the RANSAC framework.
🎯 What it does: A robust dual embedding method RDE is proposed to address the noise correspondence (NC) problem in text-image person re-identification, utilizing CCD to filter clean samples and employing TAL for stable triplet learning.
Normalizing Flows on the Product Space of SO(3) Manifolds for Probabilistic Human Pose Modeling
Olaf Dünkel, Florian Pfaff (University of Stuttgart)
CodePose EstimationFlow-based ModelMultimodality
🎯 What it does: This paper proposes and implements a normalized flow model HuProSO3 on the SO(3) product space for learning the probability distribution of human poses, and applies it to tasks such as unconditional pose priors, inverse kinematics, partially observed inverse kinematics, and 2D to 3D lifting.
🎯 What it does: This paper proposes a hardness-aware semantic scene completion framework (HASSC) that utilizes hardness voxel mining and self-distillation techniques to enhance the performance of semantic scene completion under single/multiple camera visual inputs.
Novel Class Discovery for Ultra-Fine-Grained Visual Categorization
Yu Liu (Dalian University of Technology), Nan Pu (University of Trento)
CodeClassificationRecognitionConvolutional Neural NetworkContrastive LearningImageAgriculture Related
🎯 What it does: This paper proposes the Ultra-Fine-Grained Novel Class Discovery (UFG-NCD) task and introduces the Region-Aligned Proxy Learning (RAPL) framework for discovering unlabeled ultra-fine-grained categories under partially labeled data.
🎯 What it does: This study proposes a neural target object 3D reconstruction method called NTO3D based on the Segment Anything Model (SAM). It utilizes a 3D occupancy field to elevate multi-view 2D SAM segmentation masks into a unified 3D space, and further maps the features of the SAM encoder to a 3D feature field, allowing users to obtain high-quality 3D reconstruction results of target objects with prompts given from a single viewpoint.
🎯 What it does: In this work, the authors utilize the intermediate features of the Stable Diffusion model and design three types of aggregation networks to achieve template matching-based object pose estimation, significantly improving the estimation accuracy for unseen objects.
🎯 What it does: A training framework called OGDM is proposed, which incorporates observations into diffusion models to correct the reverse process, allowing for high-quality generation results even with few sampling steps.
🎯 What it does: A pre-training method based on OCR text destylization (ODM) is proposed, which can unify diverse image texts into a consistent style and achieve precise alignment of text and image features.
OED: Towards One-stage End-to-End Dynamic Scene Graph Generation
Guan Wang (Peking University), Yang Liu (Peking University)
CodeObject DetectionGenerationTransformerVideo
🎯 What it does: This paper proposes a one-stage end-to-end dynamic scene graph generation framework called OED, which directly performs set prediction on video frame pairs to generate subject-object pairs and their relationship triples.
🎯 What it does: We propose OmniViD, a unified generative framework that integrates various video tasks such as action recognition, video captioning, question answering, event description, and object tracking into a token generation based on words, time, and boxes.
🎯 What it does: An **exact inverse method** for DDIM (first-order DPM-solver) and higher-order DPM-solvers (such as DPM-Solver++) is proposed, enabling the reverse mapping from generated images back to the initial noise;
🎯 What it does: An efficient dataset distillation method called RDED is proposed, which is non-optimized and based on patch stitching, capable of generating synthetic datasets usable for various networks with 10/50 samples per class on large-scale high-resolution datasets like ImageNet-1K.
On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation
Agneet Chatterjee (Arizona State University), Yezhou Yang (Arizona State University)
CodeDepth EstimationVision Language ModelDiffusion modelImage
🎯 What it does: This study systematically evaluates the robustness, generalization, and bias of natural language prompts in monocular depth estimation, exploring the impact of different sentence types on model performance.
On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning?
Maxime Zanella (UCLouvain UMons), Ismail Ben Ayed (TS Montréal E)
CodeClassificationRecognitionOptimizationComputational EfficiencyTransformerVision Language ModelContrastive LearningImageMultimodality
🎯 What it does: A robust test-time augmentation method MTA based on MeanShift is proposed, which clusters multi-view data directly in the final feature space of CLIP without the need for gradient training.
On Train-Test Class Overlap and Detection for Image Retrieval
Chull Hwan Song (Dealicious Inc), Yannis Avrithis (Institute of Advanced Research on Artificial Intelligence)
CodeRetrievalConvolutional Neural NetworkImage
🎯 What it does: This paper first reconstructs the Google Landmarks v2-clean dataset by removing category overlaps with the Oxford/Paris evaluation sets, resulting in R GLDv2-clean; then it proposes a single-stage, end-to-end detection-retrieval framework called CiDeR, which utilizes unsupervised attention to locate targets and directly generate global features;
One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
Lin Li (King's College London), Michael Spratling (Imperial College London)
CodeClassificationAdversarial AttackTransformerPrompt EngineeringVision Language ModelImageMultimodality
🎯 What it does: Proposes Adversarial Prompt Tuning (APT), which enhances the adversarial robustness of large-scale vision-language models (such as CLIP) by learning adjustable text context vectors.
🎯 What it does: A single-class face anti-spoofing framework OC-SCMNet is proposed, which distinguishes whether a facial image is from a live subject by learning zero-forgery clue maps (SCM).
🎯 What it does: A single image stylization method based on diffusion models, OSASIS, has been developed, which can transfer the style of any reference image to the input image while maintaining its structure.
OneLLM: One Framework to Align All Modalities with Language
Jiaming Han (Chinese University of Hong Kong), Xiangyu Yue (Shanghai Artificial Intelligence Laboratory)
CodeTransformerLarge Language ModelMixture of ExpertsImageVideoMultimodalityPoint CloudBiomedical DataMagnetic Resonance ImagingAudio
🎯 What it does: This paper designs and trains a unified framework called OneLLM, which can align eight different modalities (images, audio, video, point clouds, depth/normal maps, IMU, fMRI, etc.) with language and supports multi-modal instruction following.
🎯 What it does: This paper proposes a dynamic cluster memory management framework (DCM) for online continual learning without task boundaries, managing sample storage without the need for label information.
🎯 What it does: This paper constructs a Sketch Encoder based on the CLIP pre-trained ViT, employing a dual-layer training approach (global scene understanding + category-level refinement) to achieve semantic segmentation of freehand scene sketches, using only sketch-title pairs for weakly supervised training.
🎯 What it does: A new framework for open vocabulary semantic segmentation, EBSeg, is proposed, which combines the AdaB Decoder and SSC Loss, using frozen SAM and CLIP encoders to achieve high-quality segmentation.
🎯 What it does: This paper proposes a dual-decoder convolutional neural network that can perform closed-set semantic segmentation and identify unknown categories without the need for additional training data, distinguishing them into different new categories.
OpenBias: Open-set Bias Detection in Text-to-Image Generative Models
Moreno D'Incà (University of Trento), Nicu Sebe (University of Trento)
CodeGenerationData SynthesisTransformerLarge Language ModelDiffusion modelImageText
🎯 What it does: This paper proposes OpenBias, an end-to-end pipeline that utilizes LLM to generate potential bias concepts, employs VQA to identify and quantify biases in text-to-image generation models, and does not rely on a pre-defined list of biases.
🎯 What it does: Proposes the OpenESS framework, which achieves open vocabulary semantic segmentation on event cameras, enabling zero-shot inference with no labels or a small number of labels.
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
Qidong Huang (University of Science and Technology of China), Nenghai Yu (University of Science and Technology of China)
CodeGenerationData SynthesisTransformerLarge Language ModelVision Language ModelTextMultimodality
🎯 What it does: This paper proposes a novel decoding strategy called OPERA, which reduces the hallucination problem in multimodal large language models (MLLMs) by utilizing a penalty for the 'over-trust' aggregation pattern in self-attention and a backtracking redistribution mechanism, without adding extra data or knowledge.
OpticalDR: A Deep Optical Imaging Model for Privacy-Protective Depression Recognition
Yuchen Pan (Harbin Institute of Technology), Xianming Liu (Harbin Institute of Technology)
CodeRecognitionSafty and PrivacyTransformerImageVideoMultimodality
🎯 What it does: A learnable optical lens and deep network joint training framework for privacy-preserving depression recognition (OpticalDR) is proposed, which eliminates facial identity information through optical blurring while retaining depression-related features.
🎯 What it does: This paper proposes a single-stage visual scene localization model DINOv2 SALAD, which combines the DINOv2 visual Transformer as a feature extractor and an OT aggregation mechanism based on the Sinkhorn algorithm.
Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation
Hongwei Yan (Tsinghua University), Yi Zhong (Tsinghua University)
CodeKnowledge DistillationConvolutional Neural NetworkMixture of ExpertsContrastive LearningImage
🎯 What it does: The MOSE framework is proposed, utilizing multi-layer supervision and reverse self-distillation to address the underfitting-overfitting dilemma in online continual learning.
🎯 What it does: This paper proposes the OrthCaps Capsule Network, which significantly reduces capsule redundancy and parameter count by utilizing sparse attention routing and Householder orthogonalization.
🎯 What it does: Proposes the Selective Parameter Update (SPU) method, which updates only a small number of parameters in the first MLP layer of the pre-trained CLIP model that are most sensitive to new tasks in a continual learning scenario, while maintaining the original generalization ability.
OVMR: Open-Vocabulary Recognition with Multi-Modal References
Zehong Ma (Peking University), Qi Tian (Huawei Inc.)
CodeClassificationRecognitionTransformerVision Language ModelContrastive LearningImageTextMultimodality
🎯 What it does: This paper proposes OVMR, which generates an open vocabulary classifier using multimodal information from text descriptions and example images, without requiring additional training during inference.
PAD: Patch-Agnostic Defense against Adversarial Patch Attacks
Lihua Jing (Institute of Information Engineering, Chinese Academy of Sciences), Cong Zou (Institute of Information Engineering, Chinese Academy of Sciences)
CodeObject DetectionAdversarial AttackImageVideo
🎯 What it does: A Patch-Agnostic Defense (PAD) is designed, which does not require training or prior attack information, utilizing the semantic independence and spatial heterogeneity of patches to locate and remove them from images, compatible with any pre-trained object detector.
Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models
Xianfang Zeng (Tencent), Gang Yu (Tencent)
CodeRestorationGenerationDiffusion modelMesh
🎯 What it does: Paint3D proposes a two-stage generation framework that first uses a perspective-aware 2D diffusion model to generate rough textures, and then refines them in UV space, capable of producing high-resolution (2K) non-lit textures, and supports text or image conditional control.
🎯 What it does: The PAIR Diffusion framework is proposed to achieve independent editing of the structure and appearance of each object in an image, enabling multimodal, non-reversible image editing.
🎯 What it does: An end-to-end detection framework named PairDETR is proposed for simultaneously detecting faces and bodies and establishing associations.
🎯 What it does: PanoOcc is proposed, a camera-based 3D panoramic segmentation framework that achieves unified learning of occupancy representation and can perform semantic segmentation and instance detection end-to-end.
ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks
Kai Han (Huawei Noah's Ark Lab), Enhua Wu (State Key Lab of Computer Science, ISCAS & UCAS)
CodeClassificationComputational EfficiencyConvolutional Neural NetworkMixture of ExpertsImageText
🎯 What it does: This paper proposes ParameterNet, which significantly increases the model parameter count while maintaining low FLOPs by utilizing techniques such as dynamic convolution, thereby addressing the low FLOPs pre-training bottleneck and enhancing the performance of mobile networks after pre-training on ImageNet-22K.
Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition
Anqi Zhu (University of Melbourne), James Bailey (University of Melbourne)
CodeRecognitionGraph Neural NetworkLarge Language ModelContrastive LearningVideoTextMultimodality
🎯 What it does: A zero-shot action recognition method called PURLS is proposed, which enhances the recognition performance of unseen actions by aligning local and global semantics of language and skeleton data.
PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation
Ardian Umam (National Yang Ming Chiao Tung University), Yen-Yu Lin (National Yang Ming Chiao Tung University)
CodeSegmentationKnowledge DistillationVision Language ModelPoint Cloud
🎯 What it does: This paper proposes a cross-modal knowledge distillation framework called PartDistill, which utilizes knowledge from 2D visual language models to achieve 3D shape part segmentation.
Partial-to-Partial Shape Matching with Geometric Consistency
Viktoria Ehm (Technical University of Munich), Florian Bernard (University of Bonn)
CodeOptimizationMesh
🎯 What it does: A geometrically consistent part-to-part shape matching framework is proposed, which can simultaneously predict overlapping regions and provide correspondences.
🎯 What it does: This paper proposes PeerAiD, an online adversarial distillation method that utilizes peer networks to specifically generate adversarial samples targeting the student model, thereby enhancing the robustness and natural accuracy of the student network.
PEM: Prototype-based Efficient MaskFormer for Image Segmentation
Niccolò Cavagnero (Politecnico di Torino), Fabio Cermelli (Focoos AI)
CodeSegmentationTransformerImage
🎯 What it does: An end-to-end efficient Transformer architecture (PEM) is proposed, which supports both semantic segmentation and panoptic segmentation.
PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees
Chulin Xie (University of Illinois at Urbana-Champaign), Anima Anandkumar (University of Chicago)
CodeFederated LearningKnowledge DistillationImage
🎯 What it does: A parameter-efficient federated learning framework called PERADA is proposed, which utilizes adapters and knowledge distillation to achieve personalized models under data heterogeneity and enhance generalization ability against distribution shifts.
Perceptual Assessment and Optimization of HDR Image Rendering
Peibei Cao (City University of Hong Kong), Kede Ma (City University of Hong Kong)
CodeOptimizationNeural Radiance FieldImage
🎯 What it does: A method is proposed to decompose HDR images into multiple exposure LDR images based on an inverse display model, and to quantitatively evaluate HDR images using existing LDR quality assessment metrics, while using this evaluation as a perceptual loss for optimizing HDR image rendering.
Permutation Equivariance of Transformers and Its Applications
Hengyuan Xu (Shanghai Jiao Tong University), Baochun Li (University of Toronto)
CodeClassificationSegmentationGenerationSafty and PrivacyTransformerImageText
🎯 What it does: This paper proposes and proves the 'Permutation Equivariance' property of Transformers, which supports both cross-token exchanges and internal dimension permutations during forward and backward propagation, and demonstrates that this property remains invariant in existing models such as ViT, BERT, and GPT. Based on this property, a privacy-enhanced split-learning scheme and model encryption/authorization mechanism are constructed.
🎯 What it does: A framework for automatic skeleton generation based on associative attention is proposed for human-object interaction (HOI) image editing, and the generated skeleton is directly projected as a control signal into existing skeleton-guided generation models.
🎯 What it does: A method for adversarial attacks on customized diffusion models, called CAAT, is proposed. By adding small perturbations to the cross-attention layer, the model fails to generate fake images.
🎯 What it does: A semi-supervised learning-based breast ultrasound image lesion segmentation framework, PH-Net, is proposed, which improves segmentation accuracy by utilizing data with a lack of pixel annotations.
🎯 What it does: PIA is proposed, a plugin-based personalized image animation framework that converts any personalized text-to-image model into video animations.
🎯 What it does: A dynamic channel sampling module named PiX is proposed, which can dynamically select channels on a per-pixel basis, replacing traditional 1×1 convolutions, and achieving multifunctional capabilities such as channel compression, network scaling, and dynamic pruning.
🎯 What it does: Trained and evaluated two geographic localization models, PIGEON and PIGEOTTO, which achieved global image localization by combining semantic geographic units, haversine smooth labels, multi-task contrastive pre-training of CLIP, and cross-unit retrieval refinement.
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
Shiyu Xuan (Peking University), Shiliang Zhang (Ant Group)
CodeRecognitionObject DetectionTransformerLarge Language ModelSupervised Fine-TuningVision Language ModelImageTextMultimodality
🎯 What it does: A high-quality instruction tuning dataset was constructed by designing various reference reasoning (RC) tasks and utilizing a self-consistent bootstrapping method. Based on this, parameter-efficient tuning of the visual encoder of a multimodal large language model was performed, significantly improving the model's performance on fine-grained image understanding tasks.