🎯 What it does: A MAS (Merge and Split) framework is proposed to address the challenge of efficiently training multi-task federated learning on resource-constrained edge devices.
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
Mingdeng Cao (University of Tokyo), Yinqiang Zheng (University of Tokyo)
CodeGenerationData SynthesisDiffusion modelImage
🎯 What it does: A method called MasaCtrl is proposed, which does not require fine-tuning. By transforming self-attention in diffusion models into mutual self-attention, it enables querying the content of the source image, thereby achieving consistent image generation and complex non-rigid editing while maintaining texture and identity.
🎯 What it does: Proposes the Masked Diffusion Transformer (MDT), which enhances contextual association learning of diffusion probabilistic models through masked modeling in the latent space, significantly accelerating training and improving image generation quality.
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge
Wei Lin (Graz University of Technology), Horst Bischof (Graz University of Technology)
CodeRecognitionTransformerLarge Language ModelContrastive LearningVideoText
🎯 What it does: By unsupervisedly utilizing unlabeled videos and multi-source language knowledge (action dictionary, GPT-3 generated text, BLIP visual captions) to construct a text bag, and fine-tuning CLIP using multi-instance learning, we enhance zero-shot and few-shot action recognition performance.
🎯 What it does: MATE is proposed, a training method for 3D point cloud testing based on a mask autoencoder, which can self-supervise adaptation on a single sample and enhance robustness against disturbances such as noise and density.
MB-TaylorFormer: Multi-Branch Efficient Transformer Expanded by Taylor Formula for Image Dehazing
Yuwei Qiu (Sun Yat-sen University), Zhi Jin (Sun Yat-sen University)
CodeRestorationTransformerImage
🎯 What it does: This paper proposes a multi-branch, lightweight Transformer network called MB-TaylorFormer for single image dehazing. It mainly achieves linear complexity by embedding visual tokens through multi-scale deformable convolutions and implementing Taylor expansion softmax attention, while incorporating a multi-scale attention correction module (MSAR) to compensate for Taylor approximation errors.
MDCS: More Diverse Experts with Consistency Self-distillation for Long-tailed Recognition
Qihao Zhao (Beijing University of Chemical Technology), Jun Liu (Singapore University of Technology and Design)
CodeClassificationRecognitionKnowledge DistillationMixture of ExpertsImage
🎯 What it does: A multi-expert framework MDCS is proposed, which enhances expert diversity through diversity loss and reduces model variance via consistency self-distillation to improve long-tail classification performance.
Measuring Asymmetric Gradient Discrepancy in Parallel Continual Learning
Fan Lyu (Tianjin University), Wei Feng (Tianjin University)
CodeClassificationRecognitionOptimizationImage
🎯 What it does: This paper proposes and validates the use of Asymmetric Gradient Distance (AGD) and Maximum Difference Optimization (MaxDO) in Parallel Continual Learning (PCL) to alleviate the issues of gradient conflict and catastrophic forgetting.
🎯 What it does: This paper proposes a medical knowledge-enhanced visual-language pre-training model MedKLIP for the diagnosis and localization of X-ray images.
🎯 What it does: This study focuses on scene and plot segmentation in long videos (>60 minutes) and proposes MEGA (Multimodal Alignment Aggregation and Distillation), a unified multimodal Transformer to address the video segmentation problem.
🎯 What it does: This paper proposes the Motion Guided Masked Autoencoder (MGMAE), which utilizes a time-consistent masking strategy guided by optical flow to dynamically select visible tokens, enhancing the effectiveness of self-supervised pre-training for videos.
🎯 What it does: This paper proposes MIMO-NeRF, which replaces the single-input single-output MLP of NeRF with a multi-input multi-output MLP, and utilizes self-supervised learning to address the ambiguity of color and volume density, thereby accelerating rendering.
🎯 What it does: A framework for online video stabilization is proposed, which first estimates camera motion using deep mesh flow, then smooths the motion trajectory with a network, and finally generates a stable view from the last frame.
🎯 What it does: An unsupervised, bias-agnostic debiasing method is proposed, which identifies bias-target alignment information by extracting the Voronoi boundary distance of samples at the bottleneck layer, and uses this information to weight and regularize the training process, thereby reducing the deep network's dependence on bias in the data.
🎯 What it does: This paper proposes a lightweight RNN framework called MiniROAD for online action detection, addressing the mismatch between training and inference phases;
🎯 What it does: This paper proposes the StillMix method, which utilizes a 2D reference network to identify and mix static frames to suppress background and foreground static biases in video action recognition.
🎯 What it does: MixCycle is proposed, a semi-supervised 3D single object tracking framework that utilizes self-tracking loops, forward and backward loops, and SOTMixup data augmentation specifically designed for SOT.
🎯 What it does: By using a mixed voxel representation, the 4D dynamic scene is split into static and dynamic voxels, which are rendered using lightweight and dynamic networks respectively, achieving fast training and high frame rate rendering.
🎯 What it does: This paper proposes the first baseline for cross-lingual visual speech translation (lip reading translation) and constructs the AVMuST-TED dataset. It introduces the MixSpeech framework, which achieves audio-visual cross-modal knowledge transfer through mixed speech self-learning.
🎯 What it does: A three-stage audio-driven portrait animation system is designed, which includes a one-shot mapping network (MODA), a facial synthesis network, and a renderer with temporal position encoding, achieving multi-modal high-fidelity voice-driven portrait video synthesis.
ModelGiF: Gradient Fields for Model Functional Distance
Jie Song (Zhejiang University), Mingli Song (Zhejiang University)
CodeOptimizationSafty and PrivacyConvolutional Neural NetworkImage
🎯 What it does: Proposes the Model Gradient Field (ModelGiF) method, using the gradient field as a unified representation to measure the functional similarity of pre-trained models, and applies it to task similarity assessment, intellectual property protection, and model forgetting verification.
🎯 What it does: A skeleton action recognition framework based on self-supervised contrastive learning, RVTCLR, and its improved version RVTCLR+ are proposed. The framework jointly trains skeleton features through relative visual tempo learning and appearance consistency tasks, and further enhances high-level semantic representation by adding a Distribution-Consistency branch.
🎯 What it does: Designed and implemented MolGrapher, achieving complete recognition and reconstruction of chemical structure images through keypoint detection, hypergraph construction, and graph neural network classification.
🎯 What it does: This paper proposes MonoNeRF, which can learn transferable dynamic radiance fields from multiple segments of monocular video, enabling novel view synthesis, frame interpolation, and scene editing.
MoTIF: Learning Motion Trajectories with Local Implicit Neural Functions for Continuous Space-Time Video Super-Resolution
Yi-Hsin Chen (National Yang Ming Chiao Tung University), Wen-Hsiao Peng (National Yang Ming Chiao Tung University)
CodeRestorationSuper ResolutionOptical FlowVideo
🎯 What it does: This paper proposes a continuous spatiotemporal video super-resolution method called MoTIF, based on spatiotemporal local implicit neural functions, which reconstructs high-resolution frames using forward motion trajectories and reliability-aware splatting.
🎯 What it does: A visual Transformer model for secure multi-party computation, MPCViT, is proposed, achieving low-latency and high-accuracy inference through heterogeneous attention and NAS, and further extended to MPCViT+.
🎯 What it does: This paper proposes a compression and acceleration scheme for binary neural networks (BNN) based on the Minimum Spanning Tree (MST), utilizing MST to rearrange the computation order of convolution output channels, reducing XNOR operations and further minimizing MST distance during the learning phase, thereby decreasing the number of parameters and computational load, while also implementing a corresponding hardware accelerator.
Multi-label Affordance Mapping from Egocentric Vision
Lorenzo Mur-Labadia (Universidad de Zaragoza), Ruben Martinez-Cantin (Universidad de Zaragoza)
CodeObject DetectionSegmentationConvolutional Neural NetworkSupervised Fine-TuningSimultaneous Localization and MappingVideo
🎯 What it does: In first-person perspective videos, multi-label pixel-level affordance segmentation is achieved through automated interactive 3D geometry playback, and the largest EPIC-Aff dataset is constructed based on this.
🎯 What it does: This paper studies multi-label knowledge distillation methods and proposes the L2D framework, which combines multi-label logits distillation and label-level embedding distillation to enhance the performance of the student model.
🎯 What it does: This paper proposes the problem of Multi-view Active Fine-grained Visual Recognition (MAFR) and conducts research by collecting a multi-view fine-grained vehicle dataset (MvCars), designing experiments to validate the necessity and research value of MAFR.
🎯 What it does: This paper proposes a multi-view self-supervised separation framework (MeD) for image denoising, which utilizes multiple images of the same scene containing only noise to separate scene features from noise features in the latent space, achieving denoising without the need for clean images.
🎯 What it does: A unified multi-weather image restoration framework based on domain translation is proposed, which utilizes multi-weather variants to learn weather-invariant features, thereby restoring clear images under various weather conditions such as rain, fog, and snow.
🎯 What it does: This paper proposes a multi-object 3D visual localization task called Multi3DRefer, along with a corresponding dataset and evaluation metrics.
🎯 What it does: This paper proposes a framework based on multi-modal knowledge distillation, training an RGB-only student model for first-person action recognition, where the student uses only RGB video during inference.
🎯 What it does: A multi-modal clothing designer based on a latent diffusion model is proposed, enabling clothing image editing under three conditions: text, human pose, and clothing sketches.
🎯 What it does: This paper proposes a skeleton video anomaly detection method called MoCoDAD based on a diffusion probability model. It judges anomalies by conditioning on past actions to generate multimodal future poses and comparing them with real future poses.
Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction
Yingxue Xu (Hong Kong University of Science and Technology), Hao Chen (Hong Kong University of Science and Technology)
CodeClassificationSegmentationOptimizationTransformerMultimodalityBiomedical Data
🎯 What it does: This paper proposes a multi-modal optimal transport (OT) co-attention transformer (MOTCat), which matches whole slide image (WSI) patches with gene expression vectors through OT to filter out key information related to the tumor microenvironment, and aggregates multi-modal features using a transformer to ultimately achieve cancer survival prediction.
🎯 What it does: An Explicit Conditional Multimodal Variational Auto-Encoder (ECMVAE) is proposed for audio-visual segmentation, which fully utilizes audio instructions by explicitly decomposing shared and exclusive representations in the latent space and incorporating orthogonal and mutual information constraints.
🎯 What it does: This paper proposes a Transformer-based color quantization model called CQFormer, which can maintain a low-bit color space while balancing visual perceptual structure and machine recognition accuracy.
🎯 What it does: The NDC-Scene framework is proposed, utilizing Normalized Device Coordinates space and a depth-adaptive dual decoder to achieve monocular 3D semantic scene completion.
🎯 What it does: The Neural Microflake Field (NeMF) model is proposed, achieving inverse volume rendering to recover the distribution, density, chromaticity, and roughness of volumetric microcrystals from multi-view images, and supports relighting, material editing, and volumetric scattering effects.
🎯 What it does: We propose NEMTO, an end-to-end neural rendering pipeline that can recover the geometry and lighting of transparent objects from multi-view natural lighting images under unknown refractive indices, and synthesize new views and re-lit results.
🎯 What it does: This paper proposes a conditionally generated adversarial network based on characteristic functions (CCF-GAN), which learns the differences in joint distributions through neural characteristic functions (NCF) to achieve more stable conditional generation.
Jie Yang (International Digital Economy Academy), Lei Zhang (International Digital Economy Academy)
CodePose EstimationTransformerImage
🎯 What it does: An interactive 2D human keypoint detection framework called Click-Pose is proposed, where users can correct predicted keypoints with minimal clicks, and the remaining keypoints are automatically optimized, supporting end-to-end inference without the need for post-processing.
🎯 What it does: A pluggable video depth stabilizer NVDS is proposed, which can eliminate inter-frame flicker and enhance spatiotemporal consistency without modifying the single-image depth model; a large-scale natural scene video depth dataset VDW is also released.
🎯 What it does: Using unpaired all-weather (24-hour) visible and near-infrared video for low-light video enhancement, an end-to-end model based on physics-inspired illumination redirection, noise GAN, and temporal perception network is proposed.
🎯 What it does: A noise-aware learning framework called NoC is proposed, which utilizes alignment level control of the generator to train image description models on the crawled image-text data from the entire web.
Normalizing Flows for Human Pose Anomaly Detection
Or Hirschorn (Tel Aviv University), Shai Avidan (Tel Aviv University)
CodePose EstimationAnomaly DetectionGraph Neural NetworkFlow-based ModelVideoTime Series
🎯 What it does: This paper proposes a lightweight anomaly detection framework STG-NF that utilizes only human skeletal sequences, based on regularized flow to learn skeletal distribution and assess anomaly probability through log-likelihood.
🎯 What it does: A Selective Diffusion Distillation (SDD) framework is proposed, which uses a pre-trained diffusion model to guide lightweight image manipulators (such as the mapping network of StyleGAN) for image editing through a single forward inference, avoiding the trade-off between editability and fidelity in traditional diffusion editing.
🎯 What it does: This paper studies Adaptive Open Set Object Detection (AOOD), which considers new scenes in the target domain while also addressing the detection of new categories.
Zixu Zhao (Amazon Web Services), Tianjun Xiao (Amazon Web Services)
CodeObject TrackingVideo
🎯 What it does: This paper proposes an unsupervised multi-object tracking framework OC-MOT based on video object centric learning, utilizing a self-supervised memory module and an index-merge mechanism to address the issues of partial-whole segmentation and temporal consistency of objects.
🎯 What it does: This paper proposes a scheme that utilizes only key frames for object detection and predicting the target position in subsequent frames, which improves detection accuracy while significantly reducing computational and labeling costs.
OFVL-MS: Once for Visual Localization across Multiple Indoor Scenes
Tao Xie (Harbin Institute of Technology), Ruifeng Li (Harbin Institute of Technology)
CodePose EstimationConvolutional Neural NetworkSimultaneous Localization and MappingImage
🎯 What it does: This paper studies a unified visual localization framework based on multi-task learning, capable of predicting camera poses in multiple indoor scenes at once.
🎯 What it does: The ORAN model is proposed for the audio-visual navigation task, enhancing navigation capabilities by combining cross-task policy distillation (CCPD) and omnidirectional information gathering (OIG).
🎯 What it does: This paper studies the effect of spectral discriminators in GAN-based super-resolution and proposes the Spectral Transformer, which combines frequency domain information with Transformers, and the Dual Transformer, which integrates spatial and frequency domain discriminators, thereby improving super-resolution quality and no-reference image quality assessment.
On the Robustness of Open-World Test-Time Training: Self-Training with Dynamic Prototype Expansion
Yushu Li (South China University of Technology), Kui Jia (South China University of Technology)
CodeClassificationDomain AdaptationImage
🎯 What it does: This paper proposes a test-time training method aimed at open-world scenarios (including strong out-of-distribution samples), designed to maintain good performance even when strong OOD (unknown category) samples are present in the target domain.
🎯 What it does: This paper proposes a Training-Assisted Bit Flip Attack (TBA), which constructs a high-risk model during the training phase, allowing for the implantation of malicious functionality by flipping only a few bits after deployment.
🎯 What it does: A MatSim dataset and benchmark were proposed, and a Siamese network based on contrastive learning was trained to achieve one-shot recognition of any material state.
🎯 What it does: This paper proposes an online clustering vector quantization method called CVQ-VAE, which addresses the VQ codebook collapse problem using dynamic initialization and running average updates.
🎯 What it does: An online query propagation framework called OnlineRefer is proposed for real-time segmentation of target objects from videos based on natural language instructions.
Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network
Cong Han (Meituan Inc), Lin Ma (Meituan Inc)
CodeSegmentationTransformerVision Language ModelImage
🎯 What it does: An efficient open vocabulary semantic segmentation network, DeOP, is proposed, which completes segmentation with just one forward pass of the vision-language model.
🎯 What it does: Proposes the Open Vocabulary Video Question Answering (OVQA) benchmark and improves existing VideoQA models on this benchmark to support the prediction of rare and unseen answers.
🎯 What it does: Proposes the OpenOccupancy evaluation benchmark, extending nuScenes for a surround semantic occupancy perception task, constructing dense semantic occupancy annotations and providing multimodal baselines and a Cascade Occupancy Network.
🎯 What it does: A fully supervised and self-supervised hierarchical supervision joint learning framework named OPERA is designed and implemented to simultaneously utilize both supervised and self-supervised information from labeled and unlabeled data in image representation learning.
🎯 What it does: A knowledge distillation method based on network groups is proposed, employing an Online Role Change (ORC) mechanism that dynamically elevates the best-performing student networks to temporary teachers during the training process, forming a teacher group; this teacher group then transmits knowledge to the student group through three teaching methods (intensive teaching, private teaching, inter-group teaching).
Order-preserving Consistency Regularization for Domain Adaptation and Generalization
Mengmeng Jing (University of Electronic Science and Technology of China), Cees G. M. Snoek (University of Amsterdam)
CodeDomain AdaptationImage
🎯 What it does: A method of Order Consistency Regularization (OCR) is proposed, which reduces the model's sensitivity to domain-specific attributes by maximizing the entropy of the difference between the representations of the original and augmented images.
Ordered Atomic Activity for Fine-grained Interactive Traffic Scenario Understanding
Nakul Agarwal (Honda Research Institute), Yi-Ting Chen (National Yang Ming Chiao Tung University)
CodeRecognitionRetrievalGraph Neural NetworkVideo
🎯 What it does: This study proposes the Ordered Atomic Activity representation method, which splits interactive scenes into ordered atomic activities based on road topology and constructs the OATS dataset.
🎯 What it does: This paper proposes a method that builds on a trained monocular depth estimation model by adding a post-training image reconstruction decoder, using reconstruction error to detect out-of-distribution (OOD) inputs, thereby enhancing the safety of depth estimation.
🎯 What it does: This paper proposes the PADDLES method, which utilizes discrete Fourier transform to decompose intermediate features of the network into amplitude spectrum and phase spectrum, and sets different early stopping points for both to enhance robustness in noisy label environments.
🎯 What it does: A novel unsupervised deep image stitching method is proposed, capable of handling both large parallax and low-texture scenes, avoiding the limitations of traditional handcrafted geometric features.
🎯 What it does: This paper investigates the reasons for the failure of parametric classifiers in the task of Generalized Category Discovery (GCD) and proposes a simple and efficient baseline (SimGCD) based on self-distillation and entropy regularization. By improving pseudo-labels, joint training, and post-backbone features, it significantly enhances the ability to discover new categories.
🎯 What it does: Using parameterized depth distribution to upsample multi-view image features to BEV space, jointly achieving 3D object detection and BEV semantic segmentation, and generating visibility maps to suppress hallucinations.
🎯 What it does: Proposes the Parametric Information Maximization (PIM) model, targeting the Generalized Category Discovery (GCD) task, utilizing dual-layer optimization and supervised constraints to achieve mutual information maximization between features and labels.
🎯 What it does: A pure convolutional network ParCNetV2 is proposed, which efficiently models global context and attention mechanisms by using oversized convolution and bifurcate gate units.
Partition-And-Debias: Agnostic Biases Mitigation via a Mixture of Biases-Specific Experts
Jiaxuan Li (University of Tokyo), Hideki Nakayama (University of Tokyo)
CodeClassificationMixture of ExpertsImage
🎯 What it does: Proposes an agnostic biases scenario and achieves automatic decomposition and elimination of various unknown biases through the Partition-and-Debias (PnD) method, thereby enabling unbiased image classification.
🎯 What it does: In the domain transfer task from synthetic data to real data, a training enhancement method based on frequency domain amplitude spectrum ratio perturbation, called PASTA, is proposed;
🎯 What it does: Transform the multi-label image classification problem into a conditional transport (CT) problem of two discrete distributions: visual patches and text labels, and align the two modalities by minimizing the bidirectional CT distance.
🎯 What it does: A Persistent-Transient Duality model is proposed to predict human motion trajectories and object positions in human-object interaction (HOI).
🎯 What it does: A GAN based on triple latent variables has been designed and implemented, capable of end-to-end generating images that meet the needs of the color-blind community, while supporting varying degrees of personalization.
🎯 What it does: A mechanism called Personalized Semantic Incentive (PSE) is proposed to generate more accurate and locally adaptive image classification models for each client within the federated learning framework.
PGFed: Personalize Each Client's Global Objective for Federated Learning
Jun Luo (University of Pittsburgh), Shandong Wu (University of Pittsburgh)
CodeFederated LearningImage
🎯 What it does: A new personalized federated learning framework called PGFed is proposed, allowing each client to optimize its global objective by explicitly aggregating local and global empirical risks, and based on this, an accelerated version called PGFedMo is introduced.
🎯 What it does: A few-shot diffusion model based on staged content fusion is proposed, incorporating directional distribution consistency loss and iterative cross-domain structure guidance strategy to achieve few-shot image generation and domain adaptation.
🎯 What it does: Utilizing a pre-trained image diffusion model for text-driven editing of videos without the need for any additional training or fine-tuning;
Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image Reconstruction
Miaoyu Li (Beijing Institute of Technology), Yulun Zhang (ETH Zurich)
CodeRestorationTransformerImage
🎯 What it does: This paper proposes a Pixel Adaptive Deep Unfolding Transformer (PADUT) for reconstructing 3D hyperspectral image (HSI) cubes from CASSI coded snapshot spectral images, improving the numerical updates, prior learning, and stage interaction of traditional unfolding frameworks.
🎯 What it does: This paper proposes PARQ (Pixel-Aligned Recurrent Queries), a multi-view 3D object detection framework based on Transformer, which continuously updates the 3D positions using pixel-aligned query points during the recursive process and outputs 3D bounding boxes.
🎯 What it does: A diversified facial aging diffusion autoencoder (PADA) based on CLIP is proposed, achieving the generation of multimodal and diverse aging results conditioned on text or reference images.
🎯 What it does: This paper proposes Prompt-driven Zero-shot Domain Adaptation (PØDA), which utilizes CLIP text prompts to adapt source domain models to unseen target domains.
Point-Query Quadtree for Crowd Counting, Localization, and More
Chengxin Liu (Huazhong University of Science and Technology), Tongliang Liu (University of Sydney)
CodeRecognitionObject DetectionTransformerImage
🎯 What it does: Treating crowd counting as a decomposable point query process, we propose the Point Query Transformer (PET) model, which employs a point query quadtree and advanced rectangular window attention to support multi-tasks such as counting, localization, partial annotation learning, and point annotation refinement.
Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport
Wentong Li (Zhejiang University), Lei Zhang (HongKong Polytechnical University)
CodeObject DetectionSegmentationTransformerImage
🎯 What it does: A single-point supervised panoptic segmentation method called Point2Mask is proposed, which generates pseudo-masks through optimal transport and trains a panoptic segmentation network.
🎯 What it does: A large synthetic dataset called PointOdyssey and an improved point tracking method PIPs++ are proposed, focusing on fine-grained point tracking over long durations;
Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study
Myeongseob Ko (Virginia Tech), Ruoxi Jia (Virginia Tech)
CodeAdversarial AttackTransformerVision Language ModelContrastive LearningImageTextMultimodality
🎯 What it does: This paper proposes membership inference attacks against large multimodal models, including baseline attacks based on cosine similarity, enhanced attacks, and weakly supervised attacks.