arXiv subjects

Xi Shen

Publications and source records attributed to Xi Shen.

At least 37 records · Page 2Linked to original sources

arXiv2025

DEIM: DETR with Improved Matching for Fast Convergence

We introduce DEIM, an innovative and efficient training framework designed to accelerate convergence in real-time object detection with Transformer-based architectures (DETR). To mitigate the sparse supervision inherent in one-to-one (O2O) matching in DETR models, DEIM employs a Dense O2O matching strategy. This approach increases the number of positive samples per image by incorporating additional targets, using standard data augmentation techniques. While Dense O2O matching speeds up convergence, it also introduces numerous low-quality matches that could affect performance. To address this, we propose the Matchability-Aware Loss (MAL), a novel loss function that optimizes matches across various quality levels, enhancing the effectiveness of Dense O2O. Extensive experiments on the COCO dataset validate the efficacy of DEIM. When integrated with RT-DETR and D-FINE, it consistently boosts performance while reducing training time by 50%. Notably, paired with RT-DETRv2, DEIM achieves 53.2% AP in a single day of training on an NVIDIA 4090 GPU. Additionally, DEIM-trained real-time models outperform leading real-time object detectors, with DEIM-D-FINE-L and DEIM-D-FINE-X achieving 54.7% and 56.5% AP at 124 and 78 FPS on an NVIDIA T4 GPU, respectively, without the need for additional data. We believe DEIM sets a new baseline for advancements in real-time object detection. Our code and pre-trained models are available at https://github.com/ShihuaHuang95/DEIM.

cs.CV↗

arXiv2025

Automatic Event Recognition Employing Optically Excited Electro-Nuclear Spin Coherence

An analog automatic event recognition (AER) system can be realized by combining the technique of holographic image recognition with the process of temporal signal correlation employing stimulated photon echo in an ensemble of two-level atoms. For efficient operation of the AER system, the optical transition in the two-level system must have a large oscillator strength. However, this implies a rapid decay of the excited state, which highly constrains the duration of the event that can be recognized, even when the images are retrieved rapidly from a fast parallel optical memory device. This constraint can be overcome by using a three-level Lambda system to transfer optical coherence to long-lived electro-nuclear spin coherence and vice-versa. Here, we present an analysis of the AER process employing such a Lambda system. We also show that due to the longitudinal and transverse variation of the phases of the optical fields undergoing Fresnel diffraction, the density matrix equations must account for the differences in the phases of the coherence excited in each pixel and the corresponding phases of the optical fields in subsequent pulses.

quant-ph↗

arXiv2025

Shift, Scale and Rotation Invariant Multiple Object Detection using Balanced Joint Transform Correlator

The Polar Mellin Transform (PMT) is a well-known technique that converts images into shift, scale and rotation invariant signatures for object detection using opto-electronic correlators. However, this technique cannot be properly applied when there are multiple targets in a single input. Here, we propose a Segmented PMT (SPMT) that extends this methodology for cases where multiple objects are present within the same frame. Simulations show that this SPMT can be integrated into an opto-electronic joint transform correlator to create a correlation system capable of detecting multiple objects simultaneously, presenting robust detection capabilities across various transformation conditions, with remarkable discrimination between matching and non-matching targets.

eess.IV↗

arXiv2025

Orbital-excitation-dominated magnetization dissipation and quantum oscillation of Gilbert damping in Fe films

Using first-principles electronic structure calculation, we demonstrate the spin dissipation process in bulk Fe by orbital excitations within the energy bands of pure spin character. The variation of orbitals in the intraband transitions provides an efficient channel to convert spin to orbital angular momentum with spin-orbit interaction. This mechanism dominates the Gilbert damping of Fe below room temperature. The theoretical prediction is confirmed by the ferromagnetic resonance experiment performed on single-crystal Fe(001) films. A significant thickness-dependent damping oscillation is found at low temperature induced by the quantum well states of the corresponding energy bands. Our findings not only explain the microscopic nature of the recently reported ultralow damping of Fe-based alloys, but also help for the understanding of the transport and dissipation process of orbital currents.

cond-mat.mes-hall↗

arXiv2025

Temporal Scale and Shift Invariant Automatic Event Recognition using the Mellin Transform

The Spatio-temporal holographic correlator combines the traditional 2D optical image correlation techniques with inhomogeneously broadened arrays of cold atoms to achieve 3D time-space correlation to realize automatic event recognition at an ultra-high speed. Here we propose a method to realize such event recognition for videos running at different speeds. With this method, we can highly improve recognition accuracy and filter almost all the unwanted events in the video database.

cs.CV↗

arXiv2025

Ultra-fast Real-time Target Recognition Using a Shift, Scale, and Rotation Invariant Hybrid Opto-electronic Joint Transform Correlator

Hybrid Opto-electronic correlators (HOC) overcome many limitations of all-optical correlators (AOC) while maintaining high-speed operation. However, neither the OEC nor the AOC in their conventional configurations can detect targets that have been rotated or scaled relative to a reference. This can be addressed by using a polar Mellin transform (PMT) pre-processing step to convert input images into signatures that contain most of the relevant information, albeit represented in a shift, scale, and rotation invariant (SSRI) manner. The PMT requires the use of optics to perform the Fourier transform and electronics for a log-polar remapping step. Recently, we demonstrated a pipelined architecture that can perform the PMT at a speed of 720 frames per second (fps), enabling the construction of an efficient opto-electronic PMT pre-processor. Here, we present an experimental demonstration of a complete HOC that implements this technique to achieve real-time and ultra-fast SSRI target recognition for space situational awareness. For this demonstration, we make use of a modified version of the HOC that makes use of Joint Transform Correlation , thus rendering the system simpler and more compact.

physics.optics↗

arXiv2025

Mixed anion control of enhanced negative thermal expansion in the oxysulfide of PbTiO3

The rare physical property of negative thermal expansion (NTE) is intriguing because materials with large NTE over a wide temperature range can serve as high-performance thermal expansion compensators. However, applications of NTE are hindered by the fact that most of the available NTE materials show small magnitudes of NTE, and/or NTE occurs only in a narrow temperature range. Herein, for the first time, we investigated the effect of anion substitution instead of general Pb/Ti-site substitutions on the thermal expansion properties of a typical ferroelectric NTE material, PbTiO3. Intriguingly, the substitution of S for O in PbTiO3 further increases the tetragonality of PbTiO3. Consequently, an unusually enhanced NTE with an average volumetric coefficient of thermal expansion $\barα_V$ = -2.50 $\times$ 10$^{-5}$/K was achieved over a wide temperature range (300 -- 790 K), which is contrasted to that of pristine PbTiO3 ($\barα_V$ = -1.99 $\times$ 10$^{-5}$/K RT -- 763 K). The intensified NTE is attributed to the enhanced hybridization between Pb/Ti and O/S atoms by the substitution of S, as evidenced by our theoretical investigations. We therefore demonstrate a new technique for introducing mixed anions to achieve large NTE over a wide temperature range in PbTiO3-based ferroelectrics.

cond-mat.mtrl-sci↗

arXiv2024

ForgeryTTT: Zero-Shot Image Manipulation Localization with Test-Time Training

Social media is increasingly plagued by realistic fake images, making it hard to trust content. Previous algorithms to detect these fakes often fail in new, real-world scenarios because they are trained on specific datasets. To address the problem, we introduce ForgeryTTT, the first method leveraging test-time training (TTT) to identify manipulated regions in images. The proposed approach fine-tunes the model for each individual test sample, improving its performance. ForgeryTTT first employs vision transformers as a shared image encoder to learn both classification and localization tasks simultaneously during the training-time training using a large synthetic dataset. Precisely, the localization head predicts a mask to highlight manipulated areas. Given such a mask, the input tokens can be divided into manipulated and genuine groups, which are then fed into the classification head to distinguish between manipulated and genuine parts. During test-time training, the predicted mask from the localization head is used for the classification head to update the image encoder for better adaptation. Additionally, using the classical dropout strategy in each token group significantly improves performance and efficiency. We test ForgeryTTT on five standard benchmarks. Despite its simplicity, ForgeryTTT achieves a 20.1% improvement in localization accuracy compared to other zero-shot methods and a 4.3% improvement over non-zero-shot techniques. Our code and data will be released upon publication.

cs.CV↗

arXiv2024

HTR-VT: Handwritten Text Recognition with Vision Transformer

We explore the application of Vision Transformer (ViT) for handwritten text recognition. The limited availability of labeled data in this domain poses challenges for achieving high performance solely relying on ViT. Previous transformer-based models required external data or extensive pre-training on large datasets to excel. To address this limitation, we introduce a data-efficient ViT method that uses only the encoder of the standard transformer. We find that incorporating a Convolutional Neural Network (CNN) for feature extraction instead of the original patch embedding and employ Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements. Furthermore, our introduction of the span mask technique, which masks interconnected features in the feature map, acts as an effective regularizer. Empirically, our approach competes favorably with traditional CNN-based models on small datasets like IAM and READ2016. Additionally, it establishes a new benchmark on the LAM dataset, currently the largest dataset with 19,830 training text lines. The code is publicly available at: https://github.com/YutingLi0606/HTR-VT.

cs.CV↗

arXiv2024

From COCO to COCO-FP: A Deep Dive into Background False Positives for COCO Detectors

Reducing false positives is essential for enhancing object detector performance, as reflected in the mean Average Precision (mAP) metric. Although object detectors have achieved notable improvements and high mAP scores on the COCO dataset, analysis reveals limited progress in addressing false positives caused by non-target visual clutter-background objects not included in the annotated categories. This issue is particularly critical in real-world applications, such as fire and smoke detection, where minimizing false alarms is crucial. In this study, we introduce COCO-FP, a new evaluation dataset derived from the ImageNet-1K dataset, designed to address this issue. By extending the original COCO validation dataset, COCO-FP specifically assesses object detectors' performance in mitigating background false positives. Our evaluation of both standard and advanced object detectors shows a significant number of false positives in both closed-set and open-set scenarios. For example, the AP50 metric for YOLOv9-E decreases from 72.8 to 65.7 when shifting from COCO to COCO-FP. The dataset is available at https://github.com/COCO-FP/COCO-FP.

cs.CV↗

arXiv2024

Technique Report of CVPR 2024 PBDL Challenges

The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, and medium properties from images. In recent years, deep learning has shown promising improvements for various vision tasks, and when combined with physics-based vision, these approaches can enhance the robustness and accuracy of vision systems. This technical report summarizes the outcomes of the Physics-Based Vision Meets Deep Learning (PBDL) 2024 challenge, held in CVPR 2024 workshop. The challenge consisted of eight tracks, focusing on Low-Light Enhancement and Detection as well as High Dynamic Range (HDR) Imaging. This report details the objectives, methodologies, and results of each track, highlighting the top-performing solutions and their innovative approaches.

cs.CV↗

arXiv2024

Depth-aware Test-Time Training for Zero-shot Video Object Segmentation

Zero-shot Video Object Segmentation (ZSVOS) aims at segmenting the primary moving object without any human annotations. Mainstream solutions mainly focus on learning a single model on large-scale video datasets, which struggle to generalize to unseen videos. In this work, we introduce a test-time training (TTT) strategy to address the problem. Our key insight is to enforce the model to predict consistent depth during the TTT process. In detail, we first train a single network to perform both segmentation and depth prediction tasks. This can be effectively learned with our specifically designed depth modulation layer. Then, for the TTT process, the model is updated by predicting consistent depth maps for the same frame under different data augmentations. In addition, we explore different TTT weight updating strategies. Our empirical results suggest that the momentum-based weight initialization and looping-based training scheme lead to more stable improvements. Experiments show that the proposed method achieves clear improvements on ZSVOS. Our proposed video TTT strategy provides significant superiority over state-of-the-art TTT methods. Our code is available at: https://nifangbaage.github.io/DATTT.

cs.CV↗

arXiv2024

SURE: SUrvey REcipes for building reliable and robust deep networks

In this paper, we revisit techniques for uncertainty estimation within deep neural networks and consolidate a suite of techniques to enhance their reliability. Our investigation reveals that an integrated application of diverse techniques--spanning model regularization, classifier and optimization--substantially improves the accuracy of uncertainty predictions in image classification tasks. The synergistic effect of these techniques culminates in our novel SURE approach. We rigorously evaluate SURE against the benchmark of failure prediction, a critical testbed for uncertainty estimation efficacy. Our results showcase a consistently better performance than models that individually deploy each technique, across various datasets and model architectures. When applied to real-world challenges, such as data corruption, label noise, and long-tailed class distribution, SURE exhibits remarkable robustness, delivering results that are superior or on par with current state-of-the-art specialized methods. Particularly on Animal-10N and Food-101N for learning with noisy labels, SURE achieves state-of-the-art performance without any task-specific adjustments. This work not only sets a new benchmark for robust uncertainty estimation but also paves the way for its application in diverse, real-world scenarios where reliability is paramount. Our code is available at \url{https://yutingli0606.github.io/SURE/}.

cs.CV↗

arXiv2024

Room-temperature sub-100 nm Néel-type skyrmions in non-stoichiometric van der Waals ferromagnet $\rm Fe_{3-x}GaTe_{2}$ with ultrafast laser writability

Realizing room-temperature magnetic skyrmions in two-dimensional van der Waals ferromagnets offers unparalleled prospects for future spintronic applications. However, due to the intrinsic spin fluctuations that suppress atomic long-range magnetic order and the inherent inversion crystal symmetry that excludes the presence of the Dzyaloshinskii-Moriya interaction, achieving room-temperature skyrmions in 2D magnets remains a formidable challenge. In this study, we target room-temperature 2D magnet $\rm Fe_3GaTe_2$ and unveil that the introduction of iron-deficient into this compound enables spatial inversion symmetry breaking, thus inducing a significant Dzyaloshinskii-Moriya interaction that brings about room-temperature Néel-type skyrmions with unprecedentedly small size. To further enhance the practical applications of this finding, we employ a homemade in-situ optical Lorentz transmission electron microscopy to demonstrate ultrafast writing of skyrmions in $\rm Fe_{3-x}GaTe_2$ using a single femtosecond laser pulse. Our results manifest the $\rm Fe_{3-x}GaTe_2$ as a promising building block for realizing skyrmion-based magneto-optical functionalities.

cond-mat.mtrl-sci↗

arXiv2023

High-speed Opto-electronic Pre-processing of Polar Mellin Transform for Shift, Scale and Rotation Invariant Image Recognition at Record-Breaking Speeds

Space situational awareness demands efficient monitoring of terrestrial sites and celestial bodies, necessitating advanced target recognition systems. Current target recognition systems exhibit limited operational speed due to challenges in handling substantial image data. While machine learning has improved this scenario, highresolution images remain a concern. Optical correlators, relying on analog processes, provide a potential alternative but are hindered by material limitations. Recent advancements in hybrid opto-electronic correlators (HOC) have addressed such limitations, additionally achieving shift, scale, and rotation invariant (SSRI) target recognition through use of the polar Mellin transform (PMT). However, there are currently no techniques for obtaining the PMT at speeds fast enough to take advantage of the inherent speed of the HOC. To that end, we demonstrate an optoelectronic PMT pre-processor that can operate at record-breaking millisecond frame rates using commercially available components for use in an automated SSRI HOC image recognition system for space situational awareness.

eess.IV↗

arXiv2023

TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut

In this paper, we describe a graph-based algorithm that uses the features obtained by a self-supervised transformer to detect and segment salient objects in images and videos. With this approach, the image patches that compose an image or video are organised into a fully connected graph, where the edge between each pair of patches is labeled with a similarity score between patches using features learned by the transformer. Detection and segmentation of salient objects is then formulated as a graph-cut problem and solved using the classical Normalized Cut algorithm. Despite the simplicity of this approach, it achieves state-of-the-art results on several common image and video detection and segmentation tasks. For unsupervised object discovery, this approach outperforms the competing approaches by a margin of 6.1%, 5.7%, and 2.6%, respectively, when tested with the VOC07, VOC12, and COCO20K datasets. For the unsupervised saliency detection task in images, this method improves the score for Intersection over Union (IoU) by 4.4%, 5.6% and 5.2%. When tested with the ECSSD, DUTS, and DUT-OMRON datasets, respectively, compared to current state-of-the-art techniques. This method also achieves competitive results for unsupervised video object segmentation tasks with the DAVIS, SegTV2, and FBMS datasets.

cs.CV↗

arXiv2023

T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations

In this work, we investigate a simple and must-known conditional generative framework based on Vector Quantised-Variational AutoEncoder (VQ-VAE) and Generative Pre-trained Transformer (GPT) for human motion generation from textural descriptions. We show that a simple CNN-based VQ-VAE with commonly used training recipes (EMA and Code Reset) allows us to obtain high-quality discrete representations. For GPT, we incorporate a simple corruption strategy during the training to alleviate training-testing discrepancy. Despite its simplicity, our T2M-GPT shows better performance than competitive approaches, including recent diffusion-based approaches. For example, on HumanML3D, which is currently the largest dataset, we achieve comparable performance on the consistency between text and generated motion (R-Precision), but with FID 0.116 largely outperforming MotionDiffuse of 0.630. Additionally, we conduct analyses on HumanML3D and observe that the dataset size is a limitation of our approach. Our work suggests that VQ-VAE still remains a competitive approach for human motion generation.

cs.CV↗

arXiv2023

LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation

Gestures are non-verbal but important behaviors accompanying people's speech. While previous methods are able to generate speech rhythm-synchronized gestures, the semantic context of the speech is generally lacking in the gesticulations. Although semantic gestures do not occur very regularly in human speech, they are indeed the key for the audience to understand the speech context in a more immersive environment. Hence, we introduce LivelySpeaker, a framework that realizes semantics-aware co-speech gesture generation and offers several control handles. In particular, our method decouples the task into two stages: script-based gesture generation and audio-guided rhythm refinement. Specifically, the script-based gesture generation leverages the pre-trained CLIP text embeddings as the guidance for generating gestures that are highly semantically aligned with the script. Then, we devise a simple but effective diffusion-based gesture generation backbone simply using pure MLPs, that is conditioned on only audio signals and learns to gesticulate with realistic motions. We utilize such powerful prior to rhyme the script-guided gestures with the audio signals, notably in a zero-shot setting. Our novel two-stage generation framework also enables several applications, such as changing the gesticulation style, editing the co-speech gestures via textual prompting, and controlling the semantic awareness and rhythm alignment with guided diffusion. Extensive experiments demonstrate the advantages of the proposed framework over competing methods. In addition, our core diffusion-based generative model also achieves state-of-the-art performance on two benchmarks. The code and model will be released to facilitate future research.

cs.CV↗