arXiv subjects

Xiao Dong

Publications and source records attributed to Xiao Dong.

At least 37 records · Page 2Linked to original sources

arXiv2024

A Survey of Foundation Models for Music Understanding

Music is essential in daily life, fulfilling emotional and entertainment needs, and connecting us personally, socially, and culturally. A better understanding of music can enhance our emotions, cognitive skills, and cultural connections. The rapid advancement of artificial intelligence (AI) has introduced new ways to analyze music, aiming to replicate human understanding of music and provide related services. While the traditional models focused on audio features and simple tasks, the recent development of large language models (LLMs) and foundation models (FMs), which excel in various fields by integrating semantic information and demonstrating strong reasoning abilities, could capture complex musical features and patterns, integrate music with language and incorporate rich musical, emotional and psychological knowledge. Therefore, they have the potential in handling complex music understanding tasks from a semantic perspective, producing outputs closer to human perception. This work, to our best knowledge, is one of the early reviews of the intersection of AI techniques and music understanding. We investigated, analyzed, and tested recent large-scale music foundation models in respect of their music comprehension abilities. We also discussed their limitations and proposed possible future directions, offering insights for researchers in this field.

cs.SD↗

arXiv2024

OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion

Open-vocabulary detection is a challenging task due to the requirement of detecting objects based on class names, including those not encountered during training. Existing methods have shown strong zero-shot detection capabilities through pre-training and pseudo-labeling on diverse large-scale datasets. However, these approaches encounter two main challenges: (i) how to effectively eliminate data noise from pseudo-labeling, and (ii) how to efficiently leverage the language-aware capability for region-level cross-modality fusion and alignment. To address these challenges, we propose a novel unified open-vocabulary detection method called OV-DINO, which is pre-trained on diverse large-scale datasets with language-aware selective fusion in a unified framework. Specifically, we introduce a Unified Data Integration (UniDI) pipeline to enable end-to-end training and eliminate noise from pseudo-label generation by unifying different data sources into detection-centric data format. In addition, we propose a Language-Aware Selective Fusion (LASF) module to enhance the cross-modality alignment through a language-aware query selection and fusion process. We evaluate the performance of the proposed OV-DINO on popular open-vocabulary detection benchmarks, achieving state-of-the-art results with an AP of 50.6% on the COCO benchmark and 40.1% on the LVIS benchmark in a zero-shot manner, demonstrating its strong generalization ability. Furthermore, the fine-tuned OV-DINO on COCO achieves 58.4% AP, outperforming many existing methods with the same backbone. The code for OV-DINO is available at https://github.com/wanghao9610/OV-DINO.

cs.CV↗

arXiv2024

SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction

Dynamic reconstruction of deformable tissues in endoscopic video is a key technology for robot-assisted surgery. Recent reconstruction methods based on neural radiance fields (NeRFs) have achieved remarkable results in the reconstruction of surgical scenes. However, based on implicit representation, NeRFs struggle to capture the intricate details of objects in the scene and cannot achieve real-time rendering. In addition, restricted single view perception and occluded instruments also propose special challenges in surgical scene reconstruction. To address these issues, we develop SurgicalGaussian, a deformable 3D Gaussian Splatting method to model dynamic surgical scenes. Our approach models the spatio-temporal features of soft tissues at each time stamp via a forward-mapping deformation MLP and regularization to constrain local 3D Gaussians to comply with consistent movement. With the depth initialization strategy and tool mask-guided training, our method can remove surgical instruments and reconstruct high-fidelity surgical scenes. Through experiments on various surgical videos, our network outperforms existing method on many aspects, including rendering quality, rendering speed and GPU usage. The project page can be found at https://surgicalgaussian.github.io.

cs.CV↗

arXiv2024

Representations of non-finitely graded Lie algebras related to Virasoro algebra

In this paper, we study representations of non-finitely graded Lie algebras $\mathcal{W}(ε)$ related to Virasoro algebra, where $ε= \pm 1$. Precisely speaking, we completely classify the free $\mathcal{U}(\mathfrak h)$-modules of rank one over $\mathcal{W}(ε)$,and find that these module structures are rather different from those of other graded Lie algebras. We also determine the simplicity and isomorphism classes of these modules.

math.RT↗

arXiv2024

DRSM: efficient neural 4d decomposition for dynamic reconstruction in stationary monocular cameras

With the popularity of monocular videos generated by video sharing and live broadcasting applications, reconstructing and editing dynamic scenes in stationary monocular cameras has become a special but anticipated technology. In contrast to scene reconstructions that exploit multi-view observations, the problem of modeling a dynamic scene from a single view is significantly more under-constrained and ill-posed. Inspired by recent progress in neural rendering, we present a novel framework to tackle 4D decomposition problem for dynamic scenes in monocular cameras. Our framework utilizes decomposed static and dynamic feature planes to represent 4D scenes and emphasizes the learning of dynamic regions through dense ray casting. Inadequate 3D clues from a single-view and occlusion are also particular challenges in scene reconstruction. To overcome these difficulties, we propose deep supervised optimization and ray casting strategies. With experiments on various videos, our method generates higher-fidelity results than existing methods for single-view dynamic scene representation.

cs.CV↗

arXiv2023

Expanding the Pressure Frontier in Grüneisen Parameter Measurement: Study of Sodium Chloride

The Grüneisen parameter (γ) is crucial for determining many thermal properties, including the anharmonic effect, thermostatistics, and equation of state (EOS) of materials. However, the isentropic adiabatic compression conditions required to measure the Grüneisen parameter under high pressure are difficult to achieve. Thus, direct experimental Grüneisen parameter data in a wide range of pressures is sparse. In this work, we developed a new device that can apply pressure (up to tens of GPa) with an extremely short time about 0.5 ms, confidently achieving isentropic adiabatic compression. Then, we applied our new technique to sodium chloride and measured its Grüneisen parameter, which conforms to previous theoretical predictions. According to our obtained sodium chloride Grüneisen parameters, the calculated Hugoniot curve of the NaCl B1 phase appears up to 20 GPa and 960 K, which compares very well with the shock compression experiment data by Fritz et al. and other calculation works. Our results suggest that this new method can reliably measure the Grüneisen parameter of even more materials, which is significant for researching the equation of state in substances.

cond-mat.mtrl-sci↗

arXiv2023

Superconductivity in graphite intercalation compounds with sodium

The discovery of superconductivity in CaC6 with a critical temperature (Tc) of 11.5 K reignites much interest in exploring high-temperature superconductivity in graphite intercalation compounds (GICs). Here we identify a GIC NaC4, discovered by ab initio evolutionary structure search, as a superconductor with a computed Tc of 41.2 K at 5 GPa. This value is eight times higher than that of the synthesized GIC NaC2 and possesses the highest Tc among available GICs. The remarkable superconductivity of GIC NaC4 mainly arises from the coupling of π electrons in graphene with the low-frequency vibrations involving both Na and C atoms. These findings suggest that Na-GICs may hold great promise as high-Tc superconductors.

cond-mat.supr-con↗

arXiv2023

UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning

Recent advances in vision-language pre-training have enabled machines to perform better in multimodal object discrimination (e.g., image-text semantic alignment) and image synthesis (e.g., text-to-image generation). On the other hand, fine-tuning pre-trained models with discriminative or generative capabilities such as CLIP and Stable Diffusion on domain-specific datasets has shown to be effective in various tasks by adapting to specific domains. However, few studies have explored the possibility of learning both discriminative and generative capabilities and leveraging their synergistic effects to create a powerful and personalized multimodal model during fine-tuning. This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC). UniDiff effectively learns aligned semantics and mitigates the issue of semantic collapse during fine-tuning on small datasets by leveraging RSC on visual features from CLIP and diffusion models, without altering the pre-trained model's basic architecture. UniDiff demonstrates versatility in both multi-modal understanding and generative tasks. Experimental results on three datasets (Fashion-man, Fashion-woman, and E-commercial Product) showcase substantial enhancements in vision-language retrieval and text-to-image generation, illustrating the advantages of combining discriminative and generative fine-tuning. The proposed UniDiff model establishes a robust pipeline for personalized modeling and serves as a benchmark for future comparisons in the field.

cs.CV↗

arXiv2022

elBERto: Self-supervised Commonsense Learning for Question Answering

Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context. Typically, existing approaches first retrieve external evidence and then perform commonsense reasoning using these evidence. In this paper, we propose a Self-supervised Bidirectional Encoder Representation Learning of Commonsense (elBERto) framework, which is compatible with off-the-shelf QA model architectures. The framework comprises five self-supervised tasks to force the model to fully exploit the additional training signals from contexts containing rich commonsense. The tasks include a novel Contrastive Relation Learning task to encourage the model to distinguish between logically contrastive contexts, a new Jigsaw Puzzle task that requires the model to infer logical chains in long contexts, and three classic SSL tasks to maintain pre-trained models language encoding ability. On the representative WIQA, CosmosQA, and ReClor datasets, elBERto outperforms all other methods, including those utilizing explicit graph reasoning and external knowledge retrieval. Moreover, elBERto achieves substantial improvements on out-of-paragraph and no-effect questions where simple lexical similarity comparison does not help, indicating that it successfully learns commonsense and is able to leverage it when given dynamic context.

cs.CL↗

arXiv2022

Helium-bearing superconductor at high pressure

Helium (He) is the most inert noble gas at ambient conditions. It adopts a hexagonal close packed structure (P63/mmc) and remains in the insulating phase up to 32 TPa. In contrast, lithium (Li) is one of the most reactive metals at zero pressure, while its cubic high-pressure phase (Fd-3m) is a weak metallic electride above 475 GPa. Strikingly, a stable compound of Li5He2 (R-3m) was formed by mixing Fd-3m Li with P63/mmc He above 700 GPa. The presence of helium promotes the lattice transformation from Fd-3m Li to Pm-3m Li, and tuns the three-dimensional distributed interstitial electrons into the mixture of zero- and two-dimensional anionic electrons. This significantly increases the degree of metallization at the Fermi level, consequently, the coupling of conductive anionic electrons with the Li-dominated vibrations is the key factor to the formation of superconducting electride Li5He2 with a transition temperature up to 26 K, dynamically stable to pressures down to 210 GPa.

cond-mat.supr-con↗

arXiv2022

Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval

Our goal in this research is to study a more realistic environment in which we can conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets, and define two real practical instance-level retrieval tasks to enable the evaluations on the price comparison and personalized recommendations. For both instance-level tasks, how to accurately pinpoint the product target mentioned in the visual-linguistic data and effectively decrease the influence of irrelevant contents is quite challenging. To address this, we exploit to train a more effective cross-modal pertaining model which is adaptively capable of incorporating key concept information from the multi-modal data, by using an entity graph whose node and edge respectively denote the entity and the similarity relation between entities. Specifically, a novel Entity-Graph Enhanced Cross-Modal Pretraining (EGE-CMP) model is proposed for instance-level commodity retrieval, that explicitly injects entity knowledge in both node-based and subgraph-based ways into the multi-modal networks via a self-supervised hybrid-stream transformer, which could reduce the confusion between different object contents, thereby effectively guiding the network to focus on entities with real semantic. Experimental results well verify the efficacy and generalizability of our EGE-CMP, outperforming several SOTA cross-modal baselines like CLIP, UNITER and CAPTURE.

cs.MM↗

arXiv2022

Worst-Case Dynamic Power Distribution Network Noise Prediction Using Convolutional Neural Network

Worst-case dynamic PDN noise analysis is an essential step in PDN sign-off to ensure the performance and reliability of chips. However, with the growing PDN size and increasing scenarios to be validated, it becomes very time- and resource-consuming to conduct full-stack PDN simulation to check the worst-case noise for different test vectors. Recently, various works have proposed machine learning based methods for supply noise prediction, many of which still suffer from large training overhead, inefficiency, or non-scalability. Thus, this paper proposed an efficient and scalable framework for the worst-case dynamic PDN noise prediction. The framework first reduces the spatial and temporal redundancy in the PDN and input current vector, and then employs efficient feature extraction as well as a novel convolutional neural network architecture to predict the worst-case dynamic PDN noise. Experimental results show that the proposed framework consistently outperforms the commercial tool and the state-of-the-art machine learning method with only 0.63-1.02% mean relative error and 25-69$\times$ speedup.

cs.LG↗

arXiv2022

Unusual phase transition of layer-stacked borophene under pressure

The 8-Pmmn borophene, a boron analogue of graphene, hosts tilted and anisotropic massless Dirac fermion quasiparticles owing to the presence of the distorted graphene-like sublattice. First-principles calculations show that the stacked 8-Pmmn borophene is transformed into the fused three-dimensional borophene under pressure, being accompanied by the partially bond-breaking and bond-reforming. Strikingly, the fused 8-Pmmn borophene inherits the Dirac band dispersion resulting in an unusual semimetal-semimetal transition. A simple tight-binding model derived from graphene qualitatively reveals the underlying physics due to the maximum preservation of graphene-like substructure after the phase transition, which contrasts greatly to the transformation of graphite into diamond associated with the semimetal-insulator transition.

cond-mat.mtrl-sci↗

arXiv2022

M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining

Despite the potential of multi-modal pre-training to learn highly discriminative feature representations from complementary data modalities, current progress is being slowed by the lack of large-scale modality-diverse datasets. By leveraging the natural suitability of E-commerce, where different modalities capture complementary semantic information, we contribute a large-scale multi-modal pre-training dataset M5Product. The dataset comprises 5 modalities (image, text, table, video, and audio), covers over 6,000 categories and 5,000 attributes, and is 500 larger than the largest publicly available dataset with a similar number of modalities. Furthermore, M5Product contains incomplete modality pairs and noise while also having a long-tailed distribution, resembling most real-world problems. We further propose Self-harmonized ContrAstive LEarning (SCALE), a novel pretraining framework that integrates the different modalities into a unified model through an adaptive feature fusion mechanism, where the importance of each modality is learned directly from the modality embeddings and impacts the inter-modality contrastive learning and masked tasks within a multi-modal transformer model. We evaluate the current multi-modal pre-training state-of-the-art approaches and benchmark their ability to learn from unlabeled data when faced with the large number of modalities in the M5Product dataset. We conduct extensive experiments on four downstream tasks and demonstrate the superiority of our SCALE model, providing insights into the importance of dataset scale and diversity.

cs.CV↗

arXiv2022

Ultrahigh-Pressure Magnesium Hydrosilicates as Reservoirs of Water in Early Earth

The origin of water on the Earth is a long-standing mystery, requiring a comprehensive search for hydrous compounds, stable at conditions of the deep Earth and made of Earth-abundant elements. Previous studies usually focused on the current range of pressure-temperature conditions in the Earth's mantle and ignored a possible difference in the past, such as the stage of the core-mantle separation. Here, using ab initio evolutionary structure prediction, we find that only two magnesium hydrosilicate phases are stable at megabar pressures, $α$-Mg$_2$SiO$_5$H$_2$ and $β$-Mg$_2$SiO$_5$H$_2$, stable at 262-338 GPa and >338 GPa,respectively (all these pressures now lie within the Earth's iron core). Both are superionic conductors with quasi-one-dimensional proton diffusion at relevant conditions. In the first 30 million years of Earth's history, before the Earth's core was formed, these must have existed in the Earth, hosting much of Earth's water. As dense iron alloys segregated to form the Earth's core, Mg$_2$SiO$_5$H$_2$ phases decomposed and released water. Thus, now-extinct Mg$_2$SiO$_5$H$_2$ phases have likely contributed in a major way to the evolution of our planet.

physics.geo-ph↗

arXiv2022

Electronegativity and chemical hardness of the elements under pressure

Abundant evidence has shown the emergence of exotic chemical phenomena under pressure, including the formation of unexpected compounds and strange crystal structures. In many cases, there is no convincing explanation for these phenomena and there are virtually no chemical rules or models capable of predicting or even rationalizing these phenomena. Here we calculate, as a function of pressure, two central chemical properties of atoms, electronegativity and chemical hardness, which can be seen as the first and second-order chemical potentials. Mulliken electronegativity, which equals minus the chemical potential of the electron relative to the vacuum, is appropriately modified - instead of taking the vacuum (impossible under high pressure), we take the homogeneous electron gas as reference. We find that for most elements, chemical hardness and electronegativity decrease with pressure, consistent with pressure-induced metallization. Furthermore, we discover that pressure-induced s-d orbital transfer makes Ni, Pd and Pt "pseudo-noble-gas" atoms with a closed d-shell configuration, and the elements preceding them (Fe and especially Co, Rh, Ir) electron acceptors, while the elements right after them (Cu, Ag, Zn, Cd, for example) become highly electropositive. We show the explicative and predictive power of our electronegativity and chemical hardness scales under pressure.

cond-mat.mtrl-sci↗

arXiv2021

Formation of copper boride on Cu(111)

Boron forms compounds with nearly all metals, with notable exception of copper and other group IB and IIB elements. Here, we report an unexpected discovery of ordered copper boride grown epitaxially on Cu(111) under ultrahigh vacuum. Scanning tunneling microscopy experiments combined with ab initio evolutionary structure prediction reveal a remarkably complex structure of 2D-Cu8B14. Strong intra-layer p-d hybridization and a large amount of charge transfer between Cu and B atoms are the key factors for the emergence of copper boride. This makes the discovered material unique and opens up the possibility of synthesizing ordered low-dimensional structures in similar immiscible systems.

cond-mat.mtrl-sci↗

arXiv2021

Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining

Nowadays, customer's demands for E-commerce are more diversified, which introduces more complications to the product retrieval industry. Previous methods are either subject to single-modal input or perform supervised image-level product retrieval, thus fail to accommodate real-life scenarios where enormous weakly annotated multi-modal data are present. In this paper, we investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval among fine-grained product categories. To promote the study of this challenging task, we contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. Notably, Product1M contains over 1 million image-caption pairs and consists of two sample types, i.e., single-product and multi-product samples, which encompass a wide variety of cosmetics brands. In addition to the great diversity, Product1M enjoys several appealing characteristics including fine-grained categories, complex combinations, and fuzzy correspondence that well mimic the real-world scenes. Moreover, we propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE), that excels in capturing the potential synergy between multi-modal inputs via a hybrid-stream transformer in a self-supervised manner.CAPTURE generates discriminative instance features via masked multi-modal learning as well as cross-modal contrastive pretraining and it outperforms several SOTA cross-modal baselines. Extensive ablation studies well demonstrate the effectiveness and the generalization capacity of our model. Dataset and codes are available at https: //github.com/zhanxlin/Product1M.

cs.CV↗