arXiv subjects

Yin Fang

Publications and source records attributed to Yin Fang.

36 records · Page 2Linked to original sources

arXiv2024

Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey

Knowledge Graphs (KGs) play a pivotal role in advancing various AI applications, with the semantic web community's exploration into multi-modal dimensions unlocking new avenues for innovation. In this survey, we carefully review over 300 articles, focusing on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm. We begin by defining KGs and MMKGs, then explore their construction progress. Our review includes two primary task categories: KG-aware multi-modal learning tasks, such as Image Classification and Visual Question Answering, and intrinsic MMKG tasks like Multi-modal Knowledge Graph Completion and Entity Alignment, highlighting specific research trajectories. For most of these tasks, we provide definitions, evaluation benchmarks, and additionally outline essential insights for conducting relevant research. Finally, we discuss current challenges and identify emerging trends, such as progress in Large Language Modeling and Multi-modal Pre-training strategies. This survey aims to serve as a comprehensive reference for researchers already involved in or considering delving into KG and multi-modal learning research, offering insights into the evolving landscape of MMKG research and supporting future work.

cs.AI↗

arXiv2024

Revisit and Outstrip Entity Alignment: A Perspective of Generative Models

Recent embedding-based methods have achieved great successes in exploiting entity alignment from knowledge graph (KG) embeddings of multiple modalities. In this paper, we study embedding-based entity alignment (EEA) from a perspective of generative models. We show that EEA shares similarities with typical generative models and prove the effectiveness of the recently developed generative adversarial network (GAN)-based EEA methods theoretically. We then reveal that their incomplete objective limits the capacity on both entity alignment and entity synthesis (i.e., generating new entities). We mitigate this problem by introducing a generative EEA (GEEA) framework with the proposed mutual variational autoencoder (M-VAE) as the generative model. M-VAE enables entity conversion between KGs and generation of new entities from random noise vectors. We demonstrate the power of GEEA with theoretical analysis and empirical experiments on both entity alignment and entity synthesis tasks.

cs.CL↗

arXiv2024

ChatCell: Facilitating Single-Cell Analysis with Natural Language

As Large Language Models (LLMs) rapidly evolve, their influence in science is becoming increasingly prominent. The emerging capabilities of LLMs in task generalization and free-form dialogue can significantly advance fields like chemistry and biology. However, the field of single-cell biology, which forms the foundational building blocks of living organisms, still faces several challenges. High knowledge barriers and limited scalability in current methods restrict the full exploitation of LLMs in mastering single-cell data, impeding direct accessibility and rapid iteration. To this end, we introduce ChatCell, which signifies a paradigm shift by facilitating single-cell analysis with natural language. Leveraging vocabulary adaptation and unified sequence generation, ChatCell has acquired profound expertise in single-cell biology and the capability to accommodate a diverse range of analysis tasks. Extensive experiments further demonstrate ChatCell's robust performance and potential to deepen single-cell insights, paving the way for more accessible and intuitive exploration in this pivotal field. Our project homepage is available at https://zjunlp.github.io/project/ChatCell.

cs.CL↗

arXiv2023

Knowledge-informed Molecular Learning: A Survey on Paradigm Transfer

Machine learning, notably deep learning, has significantly propelled molecular investigations within the biochemical sphere. Traditionally, modeling for such research has centered around a handful of paradigms. For instance, the prediction paradigm is frequently deployed for tasks such as molecular property prediction. To enhance the generation and decipherability of purely data-driven models, scholars have integrated biochemical domain knowledge into these molecular study models. This integration has sparked a surge in paradigm transfer, which is solving one molecular learning task by reformulating it as another one. With the emergence of Large Language Models, these paradigms have demonstrated an escalating trend towards harmonized unification. In this work, we delineate a literature survey focused on knowledge-informed molecular learning from the perspective of paradigm transfer. We classify the paradigms, scrutinize their methodologies, and dissect the contribution of domain knowledge. Moreover, we encapsulate prevailing trends and identify intriguing avenues for future exploration in molecular learning.

cs.LG↗

arXiv2023

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

As a crucial extension of entity alignment (EA), multi-modal entity alignment (MMEA) aims to identify identical entities across disparate knowledge graphs (KGs) by exploiting associated visual information. However, existing MMEA approaches primarily concentrate on the fusion paradigm of multi-modal entity features, while neglecting the challenges presented by the pervasive phenomenon of missing and intrinsic ambiguity of visual images. In this paper, we present a further analysis of visual modality incompleteness, benchmarking latest MMEA models on our proposed dataset MMEA-UMVM, where the types of alignment KGs covering bilingual and monolingual, with standard (non-iterative) and iterative training paradigms to evaluate the model performance. Our research indicates that, in the face of modality incompleteness, models succumb to overfitting the modality noise, and exhibit performance oscillations or declines at high rates of missing modality. This proves that the inclusion of additional multi-modal data can sometimes adversely affect EA. To address these challenges, we introduce UMAEA , a robust multi-modal entity alignment approach designed to tackle uncertainly missing and ambiguous visual modalities. It consistently achieves SOTA performance across all 97 benchmark splits, significantly surpassing existing baselines with limited parameters and time consumption, while effectively alleviating the identified limitations of other models. Our code and benchmark data are available at https://github.com/zjukg/UMAEA.

cs.AI↗

arXiv2023

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs (KGs) whose entities are associated with relevant images. However, current MMEA algorithms rely on KG-level modality fusion strategies for multi-modal entity representation, which ignores the variations of modality preferences of different entities, thus compromising robustness against noise in modalities such as blurry images and relations. This paper introduces MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, which dynamically predicts the mutual correlation coefficients among modalities for more fine-grained entity-level modality fusion and alignment. Experimental results demonstrate that our model not only achieves SOTA performance in multiple training scenarios, including supervised, unsupervised, iterative, and low-resource settings, but also has a limited number of parameters, efficient runtime, and interpretability. Our code is available at https://github.com/zjukg/MEAformer.

cs.AI↗

arXiv2023

Graph Sampling-based Meta-Learning for Molecular Property Prediction

Molecular property is usually observed with a limited number of samples, and researchers have considered property prediction as a few-shot problem. One important fact that has been ignored by prior works is that each molecule can be recorded with several different properties simultaneously. To effectively utilize many-to-many correlations of molecules and properties, we propose a Graph Sampling-based Meta-learning (GS-Meta) framework for few-shot molecular property prediction. First, we construct a Molecule-Property relation Graph (MPG): molecule and properties are nodes, while property labels decide edges. Then, to utilize the topological information of MPG, we reformulate an episode in meta-learning as a subgraph of the MPG, containing a target property node, molecule nodes, and auxiliary property nodes. Third, as episodes in the form of subgraphs are no longer independent of each other, we propose to schedule the subgraph sampling process with a contrastive loss function, which considers the consistency and discrimination of subgraphs. Extensive experiments on 5 commonly-used benchmarks show GS-Meta consistently outperforms state-of-the-art methods by 5.71%-6.93% in ROC-AUC and verify the effectiveness of each proposed module. Our code is available at https://github.com/HICAI-ZJU/GS-Meta.

cs.LG↗

arXiv2023

DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning

Zero-shot learning (ZSL) aims to predict unseen classes whose samples have never appeared during training. One of the most effective and widely used semantic information for zero-shot image classification are attributes which are annotations for class-level visual characteristics. However, the current methods often fail to discriminate those subtle visual distinctions between images due to not only the shortage of fine-grained annotations, but also the attribute imbalance and co-occurrence. In this paper, we present a transformer-based end-to-end ZSL method named DUET, which integrates latent semantic knowledge from the pre-trained language models (PLMs) via a self-supervised multi-modal learning paradigm. Specifically, we (1) developed a cross-modal semantic grounding network to investigate the model's capability of disentangling semantic attributes from the images; (2) applied an attribute-level contrastive learning strategy to further enhance the model's discrimination on fine-grained visual characteristics against the attribute co-occurrence and imbalance; (3) proposed a multi-task learning policy for considering multi-model objectives. We find that our DUET can achieve state-of-the-art performance on three standard ZSL benchmarks and a knowledge graph equipped ZSL benchmark. Its components are effective and its predictions are interpretable.

cs.CV↗

arXiv2022

LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection

Visual question answering (VQA) often requires an understanding of visual concepts and language semantics, which relies on external knowledge. Most existing methods exploit pre-trained language models or/and unstructured text, but the knowledge in these resources are often incomplete and noisy. Some other methods prefer to use knowledge graphs (KGs) which often have intensive structured knowledge, but the research is still quite preliminary. In this paper, we propose LaKo, a knowledge-driven VQA method via Late Knowledge-to-text Injection. To effectively incorporate an external KG, we transfer triples into textual format and propose a late injection mechanism for knowledge fusion. Finally we address VQA as a text generation task with an effective encoder-decoder paradigm, which achieves state-of-the-art results on OKVQA dataset.

cs.CV↗

arXiv2022

Predicting nonlinear dynamics of optical solitons in optical fiber via the SCPINN

The strongly-constrained physics-informed neural network (SCPINN) is proposed by adding the information of compound derivative embedded into the soft-constraint of physics-informed neural network(PINN). It is used to predict nonlinear dynamics and the formation process of bright and dark picosecond optical solitons, and femtosecond soliton molecule in the single-mode fiber, and reveal the variation of physical quantities including the energy, amplitude, spectrum and phase of pulses during the soliton transmission. The adaptive weight is introduced to accelerate the convergence of loss function in this new neural network. Compared with the PINN, the accuracy of SCPINN in predicting soliton dynamics is improved by 5-11 times. Therefore, the SCPINN is a forward-looking method to study the modeling and analysis of soliton dynamics in the fiber.

nlin.PS↗

arXiv2022

Data-driven soliton solutions and model parameters of nonlinear wave models via the conservation-law constrained neural network method

In the process of the deep learning, we integrate more integrable information of nonlinear wave models, such as the conservation law obtained from the integrable theory, into the neural network structure, and propose a conservation-law constrained neural network method with the flexible learning rate to predict solutions and parameters of nonlinear wave models. As some examples, we study real and complex typical nonlinear wave models, including nonlinear Schrödinger equation, Korteweg-de Vries and modified Korteweg-de Vries equations. Compared with the traditional physics-informed neural network method, this new method can more accurately predict solutions and parameters of some specific nonlinear wave models even when less information is needed, for example, in the absence of the boundary conditions. This provides a reference to further study solutions of nonlinear wave models by combining the deep learning and the integrable theory.

nlin.PS↗

arXiv2022

Molecular Contrastive Learning with Chemical Element Knowledge Graph

Molecular representation learning contributes to multiple downstream tasks such as molecular property prediction and drug design. To properly represent molecules, graph contrastive learning is a promising paradigm as it utilizes self-supervision signals and has no requirements for human annotations. However, prior works fail to incorporate fundamental domain knowledge into graph semantics and thus ignore the correlations between atoms that have common attributes but are not directly connected by bonds. To address these issues, we construct a Chemical Element Knowledge Graph (KG) to summarize microscopic associations between elements and propose a novel Knowledge-enhanced Contrastive Learning (KCL) framework for molecular representation learning. KCL framework consists of three modules. The first module, knowledge-guided graph augmentation, augments the original molecular graph based on the Chemical Element KG. The second module, knowledge-aware graph representation, extracts molecular representations with a common graph encoder for the original molecular graph and a Knowledge-aware Message Passing Neural Network (KMPNN) to encode complex information in the augmented molecular graph. The final module is a contrastive objective, where we maximize agreement between these two views of molecular graphs. Extensive experiments demonstrated that KCL obtained superior performances against state-of-the-art baselines on eight molecular datasets. Visualization experiments properly interpret what KCL has learned from atoms and attributes in the augmented molecular graphs. Our codes and data are available at https://github.com/ZJU-Fangyin/KCL.

cs.LG↗

arXiv2021

Predicting the dynamic process and model parameters of the vector optical solitons in birefringent fibers via the modified PINN

A modified physics-informed neural network is used to predict the dynamics of optical pulses including one-soliton, two-soliton, and rogue wave based on the coupled nonlinear Schrödinger equation in birefringent fibers. At the same time, the elastic collision process of the mixed bright-dark soliton is predicted. Compared the predicted results with the exact solution, the modified physics-informed neural network method is proven to be effective to solve the coupled nonlinear Schrödinger equation. Moreover, the dispersion coefficients and nonlinearity coefficients of the coupled nonlinear Schrodinger equation can be learned by modified physics-informed neural network. This provides a reference for us to use deep learning methods to study the dynamic characteristics of solitons in optical fibers.

nlin.PS↗

arXiv2021

Modified physics-informed neural network method based on the conservation law constraint and its prediction of optical solitons

Based on conservation laws as one of the important integrable properties of nonlinear physical models, we design a modified physics-informed neural network method based on the conservation law constraint. From a global perspective, this method imposes physical constraints on the solution of nonlinear physical models by introducing the conservation law into the mean square error of the loss function to train the neural network. Using this method, we mainly study the standard nonlinear Schrödinger equation and predict various data-driven optical soliton solutions, including one-soliton, soliton molecules, two-soliton interaction, and rogue wave. In addition, based on various exact solutions, we use the modified physics-informed neural network method based on the conservation law constraint to predict the dispersion and nonlinear coefficients of the standard nonlinear Schrödinger equation. Compared with the traditional physics-informed neural network method, the modified method can significantly improve the calculation accuracy.

nlin.PS↗

arXiv2021

Data-driven femtosecond optical soliton excitations and parameters discovery of the high-order NLSE using the PINN

We use the physics-informed neural network to solve a variety of femtosecond optical soliton solutions of the high order nonlinear Schrödinger equation, including one-soliton solution, two-soliton solution, rogue wave solution, W-soliton solution and M-soliton solution. The prediction error for one-soliton, W-soliton and M-soliton is smaller. As the prediction distance increases, the prediction error will gradually increase. The unknown physical parameters of the high order nonlinear Schrödinger equation are studied by using rogue wave solutions as data sets. The neural network is optimized from three aspects including the number of layers of the neural network, the number of neurons, and the sampling points. Compared with previous research, our error is greatly reduced. This is not a replacement for the traditional numerical method, but hopefully to open up new ideas.

nlin.PS↗

arXiv2021

Knowledge-aware Contrastive Molecular Graph Learning

Leveraging domain knowledge including fingerprints and functional groups in molecular representation learning is crucial for chemical property prediction and drug discovery. When modeling the relation between graph structure and molecular properties implicitly, existing works can hardly capture structural or property changes and complex structure, with much smaller atom vocabulary and highly frequent atoms. In this paper, we propose the Contrastive Knowledge-aware GNN (CKGNN) for self-supervised molecular representation learning to fuse domain knowledge into molecular graph representation. We explicitly encode domain knowledge via knowledge-aware molecular encoder under the contrastive learning framework, ensuring that the generated molecular embeddings equipped with chemical domain knowledge to distinguish molecules with similar chemical formula but dissimilar functions. Extensive experiments on 8 public datasets demonstrate the effectiveness of our model with a 6\% absolute improvement on average against strong competitors. Ablation study and further investigation also verify the best of both worlds: incorporation of chemical domain knowledge into self-supervised learning.

cs.LG↗

arXiv2019

Self-dispersion of Two Natural Polysaccharides for Granular Composites

We envision that dispersion between two polymeric materials on mesoscales would create new composites with properties that are much more superior to the components alone. Here we elucidate the dispersion between two of most abundant natural polysaccharides, starch and chitosan, which form mesoscale composites that may promise many applications. By using X-ray microscopic imaging, small-angle X-ray scattering, and differential scanning calorimetry, we were able to characterize the interactions of chitosan and starch in the mesoscale composites. The morphology of the composite is far more complex from the simple mixture of starch granules with a nominal size of a few micrometers and chitosan microbundles of tens and hundreds of micrometers. This unique morphology can only be explained by the enhanced miscibility of chitosan in a starch granular matrix. It is evidenced that there is a possible ionic interaction between the amino group in chitosan and the hydroxyl groups in starch granules. Despite the limited solubility of chitosan in water, this ionic interaction allows for the disassembly of chitosan microbundles within the starch suspension. The result is a chemically stronger and more stable granular composite formed by two biocompatible and biodegradable polysaccharide polymers. The mechanism of chitosan to disperse throughout starch granules has implications for the application of chitosan in water and other solvents.

physics.chem-ph↗

arXiv2019

Dynamic and programmable cellular-scale granules enable tissue-like materials

Tissue-like materials are required in many robotic systems to improve human-machine interactions. However, the mechanical properties of living tissues are difficult to replicate. Synthetic materials are not usually capable of simultaneously displaying the behaviors of the cellular ensemble and the extracellular matrix. A particular challenge is identification of a cell-like synthetic component which is tightly integrated with its matrix and also responsive to external stimuli at the population level. Here, we demonstrate that cellular-scale hydrated starch granules, an underexplored component in materials science, can turn conventional hydrogels into tissue-like materials when composites are formed. Using several synchrotron-based X-ray techniques, we reveal the mechanically-induced motion and training dynamics of the starch granules in the hydrogel matrix. These dynamic behaviors enable multiple tissue-like properties such as strain-stiffening, anisotropy, mechanical heterogeneity, programmability, mechanochemistry, impact absorption, and self-healability. The starch-hydrogel composites can be processed as robotic skins that maintain these tissue-like characteristics.

cond-mat.soft↗