arXiv subjects

Fengyun Rao

Publications and source records attributed to Fengyun Rao.

47 records · Page 3Linked to original sources

arXiv2024

Spatial-Semantic Collaborative Cropping for User Generated Content

A large amount of User Generated Content (UGC) is uploaded to the Internet daily and displayed to people world-widely through the client side (e.g., mobile and PC). This requires the cropping algorithms to produce the aesthetic thumbnail within a specific aspect ratio on different devices. However, existing image cropping works mainly focus on landmark or landscape images, which fail to model the relations among the multi-objects with the complex background in UGC. Besides, previous methods merely consider the aesthetics of the cropped images while ignoring the content integrity, which is crucial for UGC cropping. In this paper, we propose a Spatial-Semantic Collaborative cropping network (S2CNet) for arbitrary user generated content accompanied by a new cropping benchmark. Specifically, we first mine the visual genes of the potential objects. Then, the suggested adaptive attention graph recasts this task as a procedure of information association over visual nodes. The underlying spatial and semantic relations are ultimately centralized to the crop candidate through differentiable message passing, which helps our network efficiently to preserve both the aesthetics and the content integrity. Extensive experiments on the proposed UGCrop5K and other public datasets demonstrate the superiority of our approach over state-of-the-art counterparts. Our project is available at https://github.com/suyukun666/S2CNet.

cs.CV↗

arXiv2023

Inter-X: Towards Versatile Human-Human Interaction Analysis

The analysis of the ubiquitous human-human interactions is pivotal for understanding humans as social beings. Existing human-human interaction datasets typically suffer from inaccurate body motions, lack of hand gestures and fine-grained textual descriptions. To better perceive and generate human-human interactions, we propose Inter-X, a currently largest human-human interaction dataset with accurate body movements and diverse interaction patterns, together with detailed hand gestures. The dataset includes ~11K interaction sequences and more than 8.1M frames. We also equip Inter-X with versatile annotations of more than 34K fine-grained human part-level textual descriptions, semantic interaction categories, interaction order, and the relationship and personality of the subjects. Based on the elaborate annotations, we propose a unified benchmark composed of 4 categories of downstream tasks from both the perceptual and generative directions. Extensive experiments and comprehensive analysis show that Inter-X serves as a testbed for promoting the development of versatile human-human interaction analysis. Our dataset and benchmark will be publicly available for research purposes.

cs.CV↗

arXiv2023

Image Captioning with Multi-Context Synthetic Data

Image captioning requires numerous annotated image-text pairs, resulting in substantial annotation costs. Recently, large models (e.g. diffusion models and large language models) have excelled in producing high-quality images and text. This potential can be harnessed to create synthetic image-text pairs for training captioning models. Synthetic data can improve cost and time efficiency in data collection, allow for customization to specific domains, bootstrap generalization capability for zero-shot performance, and circumvent privacy concerns associated with real-world data. However, existing methods struggle to attain satisfactory performance solely through synthetic data. We identify the issue as generated images from simple descriptions mostly capture a solitary perspective with limited context, failing to align with the intricate scenes prevalent in real-world imagery. To tackle this, we present an innovative pipeline that introduces multi-context data generation. Beginning with an initial text corpus, our approach employs a large language model to extract multiple sentences portraying the same scene from diverse viewpoints. These sentences are then condensed into a single sentence with multiple contexts. Subsequently, we generate intricate images using the condensed captions through diffusion models. Our model is exclusively trained on synthetic image-text pairs crafted through this process. The effectiveness of our pipeline is validated through experimental results in both the in-domain and cross-domain settings, where it achieves state-of-the-art performance on well-known datasets such as MSCOCO, Flickr30k, and NoCaps.

cs.CV↗

arXiv2023

A Similarity Alignment Model for Video Copy Segment Matching

With the development of multimedia technology, Video Copy Detection has been a crucial problem for social media platforms. Meta AI hold Video Similarity Challenge on CVPR 2023 to push the technology forward. In this report, we share our winner solutions on Matching Track. We propose a Similarity Alignment Model(SAM) for video copy segment matching. Our SAM exhibits superior performance compared to other competitors, with a 0.108 / 0.144 absolute improvement over the second-place competitor in Phase 1 / Phase 2. Code is available at https://github.com/FeipengMa6/VSC22-Submission/tree/main/VSC22-Matching-Track-1st.

cs.CV↗

arXiv2023

A Dual-level Detection Method for Video Copy Detection

With the development of multimedia technology, Video Copy Detection has been a crucial problem for social media platforms. Meta AI hold Video Similarity Challenge on CVPR 2023 to push the technology forward. In this paper, we share our winner solutions on both tracks to help progress in this area. For Descriptor Track, we propose a dual-level detection method with Video Editing Detection (VED) and Frame Scenes Detection (FSD) to tackle the core challenges on Video Copy Detection. Experimental results demonstrate the effectiveness and efficiency of our proposed method. Code is available at https://github.com/FeipengMa6/VSC22-Submission.

cs.CV↗

arXiv2022

CA-SSL: Class-Agnostic Semi-Supervised Learning for Detection and Segmentation

To improve instance-level detection/segmentation performance, existing self-supervised and semi-supervised methods extract either task-unrelated or task-specific training signals from unlabeled data. We show that these two approaches, at the two extreme ends of the task-specificity spectrum, are suboptimal for the task performance. Utilizing too little task-specific training signals causes underfitting to the ground-truth labels of downstream tasks, while the opposite causes overfitting to the ground-truth labels. To this end, we propose a novel Class-Agnostic Semi-Supervised Learning (CA-SSL) framework to achieve a more favorable task-specificity balance in extracting training signals from unlabeled data. CA-SSL has three training stages that act on either ground-truth labels (labeled data) or pseudo labels (unlabeled data). This decoupling strategy avoids the complicated scheme in traditional SSL methods that balances the contributions from both data types. Especially, we introduce a warmup training stage to achieve a more optimal balance in task specificity by ignoring class information in the pseudo labels, while preserving localization training signals. As a result, our warmup model can better avoid underfitting/overfitting when fine-tuned on the ground-truth labels in detection and segmentation tasks. Using 3.6M unlabeled data, we achieve a significant performance gain of 4.7% over ImageNet-pretrained baseline on FCOS object detection. In addition, our warmup model demonstrates excellent transferability to other detection and segmentation frameworks.

cs.CV↗

arXiv2021

CLIP4Caption ++: Multi-CLIP for Video Caption

This report describes our solution to the VALUE Challenge 2021 in the captioning task. Our solution, named CLIP4Caption++, is built on X-Linear/X-Transformer, which is an advanced model with encoder-decoder architecture. We make the following improvements on the proposed CLIP4Caption++: We employ an advanced encoder-decoder model architecture X-Transformer as our main framework and make the following improvements: 1) we utilize three strong pre-trained CLIP models to extract the text-related appearance visual features. 2) we adopt the TSN sampling strategy for data enhancement. 3) we involve the video subtitle information to provide richer semantic information. 3) we introduce the subtitle information, which fuses with the visual features as guidance. 4) we design word-level and sentence-level ensemble strategies. Our proposed method achieves 86.5, 148.4, 64.5 CIDEr scores on VATEX, YC2C, and TVC datasets, respectively, which shows the superior performance of our proposed CLIP4Caption++ on all three datasets.

cs.CV↗

arXiv2021

CLIP4Caption: CLIP for Video Caption

Video captioning is a challenging task since it requires generating sentences describing various diverse and complex videos. Existing video captioning models lack adequate visual representation due to the neglect of the existence of gaps between videos and texts. To bridge this gap, in this paper, we propose a CLIP4Caption framework that improves video captioning based on a CLIP-enhanced video-text matching network (VTM). This framework is taking full advantage of the information from both vision and language and enforcing the model to learn strongly text-correlated video features for text generation. Besides, unlike most existing models using LSTM or GRU as the sentence decoder, we adopt a Transformer structured decoder network to effectively learn the long-range visual and language dependency. Additionally, we introduce a novel ensemble strategy for captioning tasks. Experimental results demonstrate the effectiveness of our method on two datasets: 1) on MSR-VTT dataset, our method achieved a new state-of-the-art result with a significant gain of up to 10% in CIDEr; 2) on the private test data, our method ranking 2nd place in the ACM MM multimedia grand challenge 2021: Pre-training for Video Understanding Challenge. It is noted that our model is only trained on the MSR-VTT dataset.

cs.CV↗

arXiv2010

Detection of Strong Short-Term Variability in NGC 6946 X-1

Using two archival XMM-Newton observations, we identify strong X-ray flux variations in NGC 6946 X-1 indicating it is the most variable ultraluminous X-ray source (ULX) on mHz time scales known so far. The 1-10 keV lightcurve exhibits variability with a fractional rms amplitude of 60% integrated in the frequency range of 1-100 mHz. The power spectral density of the source shows a flat-topped spectrum that breaks at about 3 mHz with possible quasi-periodic oscillations (QPOs) near 8.5 mHz. Black hole binaries usually produce strong fast variability in the hard or intermediate state. The energy spectrum of NGC 6946 X-1 is dominated by two components, a 0.18~keV thermal disk and a power law with a photon index of ~2.2, which is consistent with the intermediate state. The characteristic time scales of the X-ray emission suggests that the ULX may contain a black hole with a mass on the order of 10^3.

astro-ph.HE↗

arXiv2010

Low-frequency oscillations in XTE J1550-564

We present the results of timing analysis of the low-frequency Quasi-Periodic Oscillation (QPO) in the Rossi X-Ray Timing Explorer data of the black hole binary XTE J1550--564 during its 1998 outburst. The QPO frequency is observed to vary on timescales between $\sim$100 s and days, correlated with the count rate contribution from the optically thick accretion disk: we studied this correlation and discuss its influence on the QPO width. In all observations, the quality factors ($ν_0$/FWHM) of the fundamental and second harmonic peaks were observed to be consistent, suggesting that the quasi-periodic nature of the oscillation is due to frequency modulation. In addition to the QPO and its harmonic peaks, a new 1.5$ν$ component was detected in the power spectra. This component is broad, with a quality factor of $\sim$0.6. From this, we argue what the peak observed at half the QPO frequency, usually referred to as "sub-harmonic" could be the fundamental frequency, leading to the sequence 1:2:3:4. We also studied the energy dependence of the timing features and conclude that the two continuum components observed in the power spectrum, although both more intense at high energies, show a different dependence on energy. At low energies, the lowest-frequency component dominates, while at high energies the higher-frequency one has a higher fractional rms. An interplay between these two components was also observed as a function of their characteristic frequency. In this source, the transition between low/hard state and hard-intermediate state appears to be a smooth process.

astro-ph.HE↗

arXiv2010

Discovery of mHz X-ray Oscillations in a Transient Ultraluminous X-ray Source in M82

We report the discovery of X-ray quasi-periodic oscillations (QPOs) at frequencies of 3-4 mHz from a transient ultraluminous X-ray source (ULX) X42.3+59 in M82. The QPOs are strong and broad and appear with weak or absent red noise, and are detected only in Chandra observations when the source is brighter than 10^40 ergs/s. The QPO behavior is similar to the type A-I QPOs found in XTE J1550-564, which is a subclass of low frequency QPOs with properties in between type A and B. Therefore, we identify the QPOs in X42.3+59 as of type A or B, and rule out the possibility of type C. With this identification, the mass of the black hole in X42.3+59 can be inferred as in the range of 12,000-43,000 solar masses by scaling the QPO frequency to that of the type A/B QPOs in stellar mass black holes. Cool disk emission is detected in one Chandra observation, and the disk inner radius suggests a similar black hole mass range. Black holes of such a high mass are able to produce an energy output in a manner similar to X42.3+59 by accreting from the interstellar medium directly.

astro-ph.HE↗