arXiv subjects

Ying Cheng

Publications and source records attributed to Ying Cheng.

28 records · Page 2Linked to original sources

arXiv2022

Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection

Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early or intermediate manner, yet overlooking the modality heterogeneousness over the weakly-supervised setting. In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual learning. To address these issues, we propose a modality-aware contrastive instance learning with self-distillation (MACIL-SD) strategy. Specifically, we leverage a lightweight two-stream network to generate audio and visual bags, in which unimodal background, violent, and normal instances are clustered into semi-bags in an unsupervised way. Then audio and visual violent semi-bag representations are assembled as positive pairs, and violent semi-bags are combined with background and normal instances in the opposite modality as contrastive negative pairs. Furthermore, a self-distillation module is applied to transfer unimodal visual knowledge to the audio-visual model, which alleviates noises and closes the semantic gap between unimodal and multimodal features. Experiments show that our framework outperforms previous methods with lower complexity on the large-scale XD-Violence dataset. Results also demonstrate that our proposed approach can be used as plug-in modules to enhance other networks. Codes are available at https://github.com/JustinYuu/MACIL_SD.

cs.CV↗

arXiv2022

Self-Supervised Video Representation Learning with Motion-Contrastive Perception

Visual-only self-supervised learning has achieved significant improvement in video representation learning. Existing related methods encourage models to learn video representations by utilizing contrastive learning or designing specific pretext tasks. However, some models are likely to focus on the background, which is unimportant for learning video representations. To alleviate this problem, we propose a new view called long-range residual frame to obtain more motion-specific information. Based on this, we propose the Motion-Contrastive Perception Network (MCPNet), which consists of two branches, namely, Motion Information Perception (MIP) and Contrastive Instance Perception (CIP), to learn generic video representations by focusing on the changing areas in videos. Specifically, the MIP branch aims to learn fine-grained motion features, and the CIP branch performs contrastive learning to learn overall semantics information for each instance. Experiments on two benchmark datasets UCF-101 and HMDB-51 show that our method outperforms current state-of-the-art visual-only self-supervised approaches.

cs.CV↗

arXiv2021

Domain Adaptive Cascade R-CNN for MItosis DOmain Generalization (MIDOG) Challenge

We present a summary of the domain adaptive cascade R-CNN method for mitosis detection of digital histopathology images. By comprehensive data augmentation and adapting existing popular detection architecture, our proposed method has achieved an F1 score of 0.7500 on the preliminary test set in MItosis DOmain Generalization (MIDOG) Challenge at MICCAI 2021.

eess.IV↗

arXiv2021

MPN: Multimodal Parallel Network for Audio-Visual Event Localization

Audio-visual event localization aims to localize an event that is both audible and visible in the wild, which is a widespread audio-visual scene analysis task for unconstrained videos. To address this task, we propose a Multimodal Parallel Network (MPN), which can perceive global semantics and unmixed local information parallelly. Specifically, our MPN framework consists of a classification subnetwork to predict event categories and a localization subnetwork to predict event boundaries. The classification subnetwork is constructed by the Multimodal Co-attention Module (MCM) and obtains global contexts. The localization subnetwork consists of Multimodal Bottleneck Attention Module (MBAM), which is designed to extract fine-grained segment-level contents. Extensive experiments demonstrate that our framework achieves the state-of-the-art performance both in fully supervised and weakly supervised settings on the Audio-Visual Event (AVE) dataset.

cs.CV↗

arXiv2020

Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication

Visual storytelling aims to generate a narrative paragraph from a sequence of images automatically. Existing approaches construct text description independently for each image and roughly concatenate them as a story, which leads to the problem of generating semantically incoherent content. In this paper, we propose a new way for visual storytelling by introducing a topic description task to detect the global semantic context of an image stream. A story is then constructed with the guidance of the topic description. In order to combine the two generation tasks, we propose a multi-agent communication framework that regards the topic description generator and the story generator as two agents and learn them simultaneously via iterative updating mechanism. We validate our approach on VIST dataset, where quantitative results, ablations, and human evaluation demonstrate our method's good ability in generating stories with higher quality compared to state-of-the-art methods.

cs.CL↗

arXiv2020

Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning

When watching videos, the occurrence of a visual event is often accompanied by an audio event, e.g., the voice of lip motion, the music of playing instruments. There is an underlying correlation between audio and visual events, which can be utilized as free supervised information to train a neural network by solving the pretext task of audio-visual synchronization. In this paper, we propose a novel self-supervised framework with co-attention mechanism to learn generic cross-modal representations from unlabelled videos in the wild, and further benefit downstream tasks. Specifically, we explore three different co-attention modules to focus on discriminative visual regions correlated to the sounds and introduce the interactions between them. Experiments show that our model achieves state-of-the-art performance on the pretext task while having fewer parameters compared with existing methods. To further evaluate the generalizability and transferability of our approach, we apply the pre-trained model on two downstream tasks, i.e., sound source localization and action recognition. Extensive experiments demonstrate that our model provides competitive results with other self-supervised methods, and also indicate that our approach can tackle the challenging scenes which contain multiple sound sources.

cs.MM↗

arXiv2020

Significant reduced traffic in Beijing failed to relieve haze pollution during the COVID-19 lockdown: implications for haze mitigation

The COVID-19 outbreak greatly limited human activities and reduced primary emissions particularly from urban on-road vehicles, but coincided with Beijing experiencing pandemic haze, raising the public concerns of the validity and effectiveness of the imposed traffic policies to improve the air pollution. Here, we explored the relationship between local vehicle emissions and the winter haze in Beijing before and during the COVID-19 lockdown period based on an integrated analysis framework, which combines a real-time on-road emission inventory, in-situ air quality observations and a localized chemical transport modeling system. We found that traffic emissions decreased substantially affected by the pandemic, with a higher reduction for NOx (75.9%, 125.3 Mg/day) compared to VOCs (53.1%, 52.9 Mg/day). Unexpectedly, our results show that the imbalanced emission abatement of NOx and VOCs from vehicles led to a significant rise of the atmospheric oxidizing capacity in urban areas, but only resulting in modest increases in secondary aerosols due to the inadequate precursors. However, the enhanced oxidizing capacity in the surrounding regions greatly increased the secondary particles with relatively abundant precursors, which is mainly responsible for Beijing haze during the lockdown period. Our results indicate that the winter haze in Beijing was insensitive to the local vehicular emissions reduction due to the complicated nonlinear response of the fine particle and air pollutant emissions. We suggest mitigation policies should focus on accelerating VOC and NH3 emissions reduction and synchronously controlling regional sources to release the benefits on local traffic emission control.

physics.ao-ph↗

arXiv2020

Applying the Network Item Response Model to Student Assessment Data

This study discusses an alternative tool for modeling student assessment data. The model constructs networks from a matrix item responses and attempts to represent these data in low dimensional Euclidean space. This procedure has advantages over common methods used for modeling student assessment data such as Item Response Theory because it relaxes the highly restrictive local-independence assumption. This article provides a deep discussion of the model and the steps one must take to estimate it. To enable extending a present model by adding data, two methods for estimating the positions of new individuals in the network are discussed. Then, a real data analysis is then provided as a case study on using the model and how to interpret the results. Finally, the model is compared and contrasted to other popular models in psychological and educational measurement: Item response theory (IRT) and network psychometric Ising model for binary data.

stat.AP↗

arXiv2019

The citation advantage of foreign language references for Chinese social science papers

Contemporary scientific exchanges are international, yet language continues to be a persistent barrier to scientific communication, particularly for non-native English-speaking scholars. Since the ability to absorb knowledge has a strong impact on how researchers create new scientific knowledge, a comprehensive access to and understanding of both domestic and international scientific publications is essential for scientific performance. This study explores the effect of absorbed knowledge on research impact by analyzing the relationship between the language diversity of cited references and the number of citations received by the citing paper. Chinese social sciences are taken as the research object, and the data, 950,302 papers published between 1998 and 2013 with 8,151,327 cited references, were collected from the Chinese Social Sciences Citation Index. Results show that there is a stark increase in the consumption of foreign language material within the Chinese social science community, and English material accounts for the vast majority of this consumption. Papers with foreign language references receive significantly more citations than those without, and the citation advantage of these internationalized work holds when we control for characteristics of the citing papers. However, the citation advantage has decreased from 1998 to 2008, largely as an artifact of the increased number of papers citing foreign language material. After 2008, the decline of the citation advantage subsided and became relatively stable, which suggests that incorporating foreign language literature continues to increase scientific impact, even as the scientific community itself becomes increasingly international. However, internationalization is not without concerns: the work closes with a discussion of the benefits and potential problems of the lack of linguistic diversity in scientific communication.

cs.DL↗

arXiv2018

Topological Sound

Recently, we witnessed a tremendous effort to conquer the realm of acoustics as a possible playground to test with sound waves topologically protected wave propagation. Acoustics differ substantially from photonic and electronic systems since longitudinal sound waves lack intrinsic spin polarization and breaking the time-reversal symmetry requires additional complexities that both are essential in mimicking the quantum effects leading to topologically robust sound propagation. In this article, we review the latest efforts to explore with sound waves topological states of quantum matter in two- and three-dimensional systems where we discuss how spin and valley degrees of freedom appear as highly novel ingredients to tailor the flow of sound in the form of one-way edge modes and defect-immune protected acoustic waves. Both from a theoretical stand point and based on contemporary experimental verifications, we summarize the latest advancements of the flourishing research frontier on topological sound.

cond-mat.mes-hall↗