arXiv subjects

Guangyao Li

Publications and source records attributed to Guangyao Li.

At least 37 records · Page 2Linked to original sources

arXiv2024

Collective modes in an unconventional superconductor with $j=3/2$ fermions

The $j = 3/2$ fermions in cubic crystals or cold atomic gases can form Cooper pairs in both singlet ($J = 0$) and unconventional quintet ($J = 2$) $s$-wave states. Our study utilizes analytical field theory to examine fluctuations in these states within the framework of the Luttinger-Kohn model. We investigate how collective modes evolve with varying spin-orbit coupling (SOC) strength. In the singlet state, quintet Bardasis-Schrieffer modes soften at a finite wavevector, hinting at Fulde-Ferrell-Larkin-Ovchinnikov physics. In the quintet state, we identify additional gapless and gapped modes originating from the partially broken symmetry due to SOC. Our results can be readily detected using current experimental techniques.

cond-mat.supr-con↗

arXiv2024

Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer

Never having seen an object and heard its sound simultaneously, can the model still accurately localize its visual position from the input audio? In this work, we concentrate on the Audio-Visual Localization and Segmentation tasks but under the demanding zero-shot and few-shot scenarios. To achieve this goal, different from existing approaches that mostly employ the encoder-fusion-decoder paradigm to decode localization information from the fused audio-visual feature, we introduce the encoder-prompt-decoder paradigm, aiming to better fit the data scarcity and varying data distribution dilemmas with the help of abundant knowledge from pre-trained models. Specifically, we first propose to construct Semantic-aware Audio Prompt (SAP) to help the visual foundation model focus on sounding objects, meanwhile, the semantic gap between the visual and audio modalities is also encouraged to shrink. Then, we develop a Correlation Adapter (ColA) to keep minimal training efforts as well as maintain adequate knowledge of the visual foundation model. By equipping with these means, extensive experiments demonstrate that this new paradigm outperforms other fusion-based methods in both the unseen class and cross-dataset settings. We hope that our work can further promote the generalization study of Audio-Visual Localization and Segmentation in practical application scenarios.

cs.CV↗

arXiv2023

CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing

Audio-visual video parsing is the task of categorizing a video at the segment level with weak labels, and predicting them as audible or visible events. Recent methods for this task leverage the attention mechanism to capture the semantic correlations among the whole video across the audio-visual modalities. However, these approaches have overlooked the importance of individual segments within a video and the relationship among them, and tend to rely on a single modality when learning features. In this paper, we propose a novel interactive-enhanced cross-modal perception method~(CM-PIE), which can learn fine-grained features by applying a segment-based attention module. Furthermore, a cross-modal aggregation block is introduced to jointly optimize the semantic representation of audio and visual signals by enhancing inter-modal interactions. The experimental results show that our model offers improved parsing performance on the Look, Listen, and Parse dataset compared to other methods.

cs.CV↗

arXiv2023

Progressive Spatio-temporal Perception for Audio-Visual Question Answering

Audio-Visual Question Answering (AVQA) task aims to answer questions about different visual objects, sounds, and their associations in videos. Such naturally multi-modal videos are composed of rich and complex dynamic audio-visual components, where most of which could be unrelated to the given questions, or even play as interference in answering the content of interest. Oppositely, only focusing on the question-aware audio-visual content could get rid of influence, meanwhile enabling the model to answer more efficiently. In this paper, we propose a Progressive Spatio-Temporal Perception Network (PSTP-Net), which contains three modules that progressively identify key spatio-temporal regions w.r.t. questions. Specifically, a temporal segment selection module is first introduced to select the most relevant audio-visual segments related to the given question. Then, a spatial region selection module is utilized to choose the most relevant regions associated with the question from the selected temporal segments. To further refine the selection of features, an audio-guided visual attention module is employed to perceive the association between auido and selected spatial regions. Finally, the spatio-temporal features from these modules are integrated for answering the question. Extensive experimental results on the public MUSIC-AVQA and AVQA datasets provide compelling evidence of the effectiveness and efficiency of PSTP-Net. Code is available at: \href{https://github.com/GeWu-Lab/PSTP-Net}{https://github.com/GeWu-Lab/PSTP-Net}

cs.CV↗

arXiv2023

Towards Long Form Audio-visual Video Understanding

We live in a world filled with never-ending streams of multimodal information. As a more natural recording of the real scenario, long form audio-visual videos are expected as an important bridge for better exploring and understanding the world. In this paper, we propose the multisensory temporal event localization task in long form videos and strive to tackle the associated challenges. To facilitate this study, we first collect a large-scale Long Form Audio-visual Video (LFAV) dataset with 5,175 videos and an average video length of 210 seconds. Each of the collected videos is elaborately annotated with diversified modality-aware events, in a long-range temporal sequence. We then propose an event-centric framework for localizing multisensory events as well as understanding their relations in long form videos. It includes three phases in different levels: snippet prediction phase to learn snippet features, event extraction phase to extract event-level features, and event interaction phase to study event relations. Experiments demonstrate that the proposed method, utilizing the new LFAV dataset, exhibits considerable effectiveness in localizing multiple modality-aware events within long form videos. Project website: http://gewu-lab.github.io/LFAV/

cs.MM↗

arXiv2023

Multi-Scale Attention for Audio Question Answering

Audio question answering (AQA), acting as a widely used proxy task to explore scene understanding, has got more attention. The AQA is challenging for it requires comprehensive temporal reasoning from different scales' events of an audio scene. However, existing methods mostly extend the structures of visual question answering task to audio ones in a simple pattern but may not perform well when perceiving a fine-grained audio scene. To this end, we present a Multi-scale Window Attention Fusion Model (MWAFM) consisting of an asynchronous hybrid attention module and a multi-scale window attention module. The former is designed to aggregate unimodal and cross-modal temporal contexts, while the latter captures sound events of varying lengths and their temporal dependencies for a more comprehensive understanding. Extensive experiments are conducted to demonstrate that the proposed MWAFM can effectively explore temporal information to facilitate AQA in the fine-grained scene.Code: https://github.com/GeWu-Lab/MWAFM

cs.SD↗

arXiv2023

An efficient online successive reanalysis method for dynamic topology optimization

In this study, an efficient reanalysis strategy for dynamic topology optimization is proposed. Compared with other related studies, an online successive dynamic reanalysis method and POD-based approximate dynamic displacement strategy are integrated. In dynamic reanalysis, the storage of the stiffness matrix decomposition can be avoided and the reduced basis vectors should be updated successively according to the structural status in each iteration. Therefore, the bottleneck of combined approximation method for large-scale dynamic topology optimization can be handled. Sequentially, the Proper Orthogonal Decomposition (POD) is employed to obtain the approximate dynamic displacement, in which the Proper Orthogonal Mode (POM) of the displacement field is employed to establish the approximated equivalent static loads of Equivalent Static Load (ESL) method. Compared with the exact equivalent static loads at all the time intervals, the number of equivalent static loads is significantly reduced. Finally, the 2D and 3D test results indicate that the proposed method has remarkable speed-up effect on the premise of small relative error, support the strength of the proposed strategy.

math.NA↗

arXiv2023

MECPformer: Multi-estimations Complementary Patch with CNN-Transformers for Weakly Supervised Semantic Segmentation

The initial seed based on the convolutional neural network (CNN) for weakly supervised semantic segmentation always highlights the most discriminative regions but fails to identify the global target information. Methods based on transformers have been proposed successively benefiting from the advantage of capturing long-range feature representations. However, we observe a flaw regardless of the gifts based on the transformer. Given a class, the initial seeds generated based on the transformer may invade regions belonging to other classes. Inspired by the mentioned issues, we devise a simple yet effective method with Multi-estimations Complementary Patch (MECP) strategy and Adaptive Conflict Module (ACM), dubbed MECPformer. Given an image, we manipulate it with the MECP strategy at different epochs, and the network mines and deeply fuses the semantic information at different levels. In addition, ACM adaptively removes conflicting pixels and exploits the network self-training capability to mine potential target information. Without bells and whistles, our MECPformer has reached new state-of-the-art 72.0% mIoU on the PASCAL VOC 2012 and 42.4% on MS COCO 2014 dataset. The code is available at https://github.com/ChunmengLiu1/MECPformer.

cs.CV↗

arXiv2023

Equatorial Waves in Rotating Bubble-Trapped Superfluids

As the Earth rotates, the Coriolis force causes several oceanic and atmospheric waves to be trapped along the equator, including Kelvin, Yanai, Rossby, and Poincaré modes. It has been demonstrated that the mathematical origin of these waves is related to the nontrivial topology of the underlying hydrodynamic equations. Inspired by recent observations of Bose-Einstein condensation (BEC) in bubble-shaped traps in microgravity ultracold quantum gas experiments, we show that equatorial modes are supported by a rapidly rotating condensate in a spherical geometry. Based on a zero-temperature coarse-grained hydrodynamic framework, we reformulate the coupled oscillations of the superfluid and the Abrikosov vortex lattice resulting from rotation by a Schrödinger-like eigenvalue problem. The obtained non-Hermitian Hamiltonian is topologically nontrivial. Furthermore, we solve the hydrodynamic equations for a spherical geometry and find that the rotating superfluid hosts Kelvin, Yanai, and Poincaré equatorial modes, but not the Rossby mode. Our predictions can be tested with state-of-the-art bubble-shaped trapped BEC experiments.

cond-mat.quant-gas↗

arXiv2023

Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning

Federated Learning (FL) recently emerges as a paradigm to train a global machine learning model across distributed clients without sharing raw data. Knowledge Graph (KG) embedding represents KGs in a continuous vector space, serving as the backbone of many knowledge-driven applications. As a promising combination, federated KG embedding can fully take advantage of knowledge learned from different clients while preserving the privacy of local data. However, realistic problems such as data heterogeneity and knowledge forgetting still remain to be concerned. In this paper, we propose FedLU, a novel FL framework for heterogeneous KG embedding learning and unlearning. To cope with the drift between local optimization and global convergence caused by data heterogeneity, we propose mutual knowledge distillation to transfer local knowledge to global, and absorb global knowledge back. Moreover, we present an unlearning method based on cognitive neuroscience, which combines retroactive interference and passive decay to erase specific knowledge from local clients and propagate to the global model by reusing knowledge distillation. We construct new datasets for assessing realistic performance of the state-of-the-arts. Extensive experiments show that FedLU achieves superior results in both link prediction and knowledge forgetting.

cs.LG↗

arXiv2022

EventEA: Benchmarking Entity Alignment for Event-centric Knowledge Graphs

Entity alignment is to find identical entities in different knowledge graphs (KGs) that refer to the same real-world object. Embedding-based entity alignment techniques have been drawing a lot of attention recently because they can help solve the issue of symbolic heterogeneity in different KGs. However, in this paper, we show that the progress made in the past was due to biased and unchallenging evaluation. We highlight two major flaws in existing datasets that favor embedding-based entity alignment techniques, i.e., the isomorphic graph structures in relation triples and the weak heterogeneity in attribute triples. Towards a critical evaluation of embedding-based entity alignment methods, we construct a new dataset with heterogeneous relations and attributes based on event-centric KGs. We conduct extensive experiments to evaluate existing popular methods, and find that they fail to achieve promising performance. As a new approach to this difficult problem, we propose a time-aware literal encoder for entity alignment. The dataset and source code are publicly available to foster future research. Our work calls for more effective and practical embedding-based solutions to entity alignment.

cs.CL↗

arXiv2022

I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning

Knowledge graph (KG) embedding seeks to learn vector representations for entities and relations. Conventional models reason over graph structures, but they suffer from the issues of graph incompleteness and long-tail entities. Recent studies have used pre-trained language models to learn embeddings based on the textual information of entities and relations, but they cannot take advantage of graph structures. In the paper, we show empirically that these two kinds of features are complementary for KG embedding. To this end, we propose CoLE, a Co-distillation Learning method for KG Embedding that exploits the complementarity of graph structures and text information. Its graph embedding model employs Transformer to reconstruct the representation of an entity from its neighborhood subgraph. Its text embedding model uses a pre-trained language model to generate entity representations from the soft prompts of their names, descriptions, and relational neighbors. To let the two model promote each other, we propose co-distillation learning that allows them to distill selective knowledge from each other's prediction logits. In our co-distillation learning, each model serves as both a teacher and a student. Experiments on benchmark datasets demonstrate that the two models outperform their related baselines, and the ensemble method CoLE with co-distillation learning advances the state-of-the-art of KG embedding.

cs.CL↗

arXiv2022

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos. The problem requires comprehensive multimodal understanding and spatio-temporal reasoning over audio-visual scenes. To benchmark this task and facilitate our study, we introduce a large-scale MUSIC-AVQA dataset, which contains more than 45K question-answer pairs covering 33 different question templates spanning over different modalities and question types. We develop several baselines and introduce a spatio-temporal grounded audio-visual network for the AVQA problem. Our results demonstrate that AVQA benefits from multisensory perception and our model outperforms recent A-, V-, and AVQA approaches. We believe that our built dataset has the potential to serve as testbed for evaluating and promoting progress in audio-visual scene understanding and spatio-temporal reasoning. Code and dataset: http://gewu-lab.github.io/MUSIC-AVQA/

cs.CV↗

arXiv2022

WegFormer: Transformers for Weakly Supervised Semantic Segmentation

Although convolutional neural networks (CNNs) have achieved remarkable progress in weakly supervised semantic segmentation (WSSS), the effective receptive field of CNN is insufficient to capture global context information, leading to sub-optimal results. Inspired by the great success of Transformers in fundamental vision areas, this work for the first time introduces Transformer to build a simple and effective WSSS framework, termed WegFormer. Unlike existing CNN-based methods, WegFormer uses Vision Transformer (ViT) as a classifier to produce high-quality pseudo segmentation masks. To this end, we introduce three tailored components in our Transformer-based framework, which are (1) a Deep Taylor Decomposition (DTD) to generate attention maps, (2) a soft erasing module to smooth the attention maps, and (3) an efficient potential object mining (EPOM) to filter noisy activation in the background. Without any bells and whistles, WegFormer achieves state-of-the-art 70.5% mIoU on the PASCAL VOC dataset, significantly outperforming the previous best method. We hope WegFormer provides a new perspective to tap the potential of Transformer in weakly supervised semantic segmentation. Code will be released.

cs.CV↗

arXiv2021

Microscopic calculation of polariton scattering in semiconductor microcavities

Recent experiments in exciton-polariton systems have provided high-precision measurements of the value of the polariton-polariton interaction constant, which is a key parameter that governs the nonlinear dynamics of polariton condensates and potentially enables quantum correlated polaritons. Yet, until now, this parameter has only been addressed theoretically using perturbative treatments or approximations that do not include the composite nature of the excitons. Here, we use a recently developed microscopic description of polaritons involving electrons, holes, and photons, where the interactions between charged particles are assumed to be highly screened. Within this model, we perform an exact four-body calculation of the spin-polarized polariton-polariton and polariton-exciton interaction constants. In the limit of weak light-matter coupling relevant to an atomically thin semiconductor in a microcavity, we obtain excellent agreement with a recently proposed universal form of low-energy polariton-polariton scattering [O. Bleu et al., Phys. Rev. Res. 2, 043185 (2020)]. At stronger light-matter coupling, of relevance to multilayer microcavities, we observe that the interaction constant increases towards that predicted by the Born approximation. We show that in all regimes of interest the interaction constant can be accurately obtained from the exciton-exciton scattering phase shift at negative collision energy, and we argue that this has important implications for interactions in other systems featuring strong light-matter coupling.

cond-mat.mes-hall↗

arXiv2021

Microscopic description of exciton-polaritons in microcavities

We investigate the microscopic description of exciton-polaritons that involves electrons, holes and photons within a two-dimensional microcavity. We show that in order to recover the simplified exciton-photon model that is typically used to describe polaritons, one must correctly define the exciton-photon detuning and exciton-photon (Rabi) coupling in terms of the bare microscopic parameters. For the case of unscreened Coulomb interactions, we find that the exciton-photon detuning is strongly shifted from its bare value in a manner akin to renormalization in quantum electrodynamics. Within the renormalized theory, we exactly solve the problem of a single exciton-polariton for the first time and obtain the full spectral response of the microcavity. In particular, we find that the electron-hole wave function of the polariton can be significantly modified by the very strong Rabi couplings achieved in current experiments. Our microscopic approach furthermore allows us to obtain the effective interaction between identical polaritons for any light-matter coupling. Crucially, we show that the standard treatment of polariton-polariton interactions in the very strong coupling regime is incorrect, since it neglects the light-induced modification of the exciton size and thus greatly overestimates the effect of Pauli exclusion on the Rabi coupling, i.e., the saturation of exciton oscillator strength. Our findings thus provide the foundations for understanding and characterizing exciton-polariton systems across the whole range of polariton densities.

cond-mat.quant-gas↗

arXiv2021

Theory of polariton-electron interactions in semiconductor microcavities

We develop a microscopic description of an electron-doped two-dimensional semiconductor embedded in a microcavity. Specifically, we investigate the interactions between exciton-polaritons and electrons for the case where the interactions between charges are strongly screened and the system is spin polarized. As a starting point, we obtain an analytic expression for the exciton-polariton wave function, and we relate the microscopic parameters of the light-matter system to experimentally measurable quantities, such as the Rabi coupling and the cavity photon frequency. We then derive the polariton-electron interaction within the standard Born approximation and compare it with the exact polariton-electron scattering $T$ matrix that we obtain from a diagrammatic approach that has proven highly successful in the context of nuclear physics and ultracold atomic gases. In particular, we show that the Born approximation provides an upper bound on the polariton-electron coupling strength at vanishing momentum. Using our exact microscopic calculation, we demonstrate that polariton-electron scattering can be strongly enhanced compared to the exciton-electron case, which is the opposite of that expected from the Born approximation. We furthermore expose a resonance-like peak at scattering momenta near the polariton inflection point, whose size is set by the strength of the light-matter coupling. Our results arise from the non-Galilean nature of the polariton system and should thus be applicable to a range of semiconductor microcavities such as GaAs quantum wells and atomically thin materials.

cond-mat.mes-hall↗

arXiv2021

Enhanced scattering between electrons and exciton-polaritons in a microcavity

The interplay between strong light-matter interactions and charge doping represents an important frontier in the pursuit of exotic many-body physics and optoelectronics. Here, we consider a simplified model of a two-dimensional semiconductor embedded in a microcavity, where the interactions between electrons and holes are strongly screened, allowing us to develop a diagrammatic formalism for this system with an analytic expression for the exciton-polariton propagator. We apply this to the scattering of spin-polarized polaritons and electrons, and show that this is strongly enhanced compared with exciton-electron interactions. As we argue, this counter-intuitive result is a consequence of the shift of the collision energy due to the strong light-matter coupling, and hence this is a generic feature that applies also for more realistic electron-hole and electron-electron interactions. We furthermore demonstrate that the lack of Galilean invariance inherent in the light-matter coupled system can lead to a narrow resonance-like feature for polariton-electron interactions close to the polariton inflection point. Our results are potentially important for realizing tunable light-mediated interactions between charged particles.

cond-mat.mes-hall↗