arXiv subjects

Yong Zhu

Publications and source records attributed to Yong Zhu.

At least 19 recordsLinked to original sources

arXiv2025

Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs

Sparse large language models (LLMs) with Mixture of Experts (MoE) and close to a trillion parameters are dominating the realm of most capable language models. However, the massive model scale poses significant challenges for the underlying software and hardware systems. In this paper, we aim to uncover a recipe to harness such scale on Ascend NPUs. The key goals are better usage of the computing resources under the dynamic sparse model structures and materializing the expected performance gain on the actual hardware. To select model configurations suitable for Ascend NPUs without repeatedly running the expensive experiments, we leverage simulation to compare the trade-off of various model hyperparameters. This study led to Pangu Ultra MoE, a sparse LLM with 718 billion parameters, and we conducted experiments on the model to verify the simulation results. On the system side, we dig into Expert Parallelism to optimize the communication between NPU devices to reduce the synchronization overhead. We also optimize the memory efficiency within the devices to further reduce the parameter and activation management overhead. In the end, we achieve an MFU of 30.0% when training Pangu Ultra MoE, with performance comparable to that of DeepSeek R1, on 6K Ascend NPUs, and demonstrate that the Ascend system is capable of harnessing all the training stages of the state-of-the-art language models. Extensive experiments indicate that our recipe can lead to efficient training of large-scale sparse language models with MoE. We also study the behaviors of such models for future reference.

cs.CL↗

arXiv2024

SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens

We propose an acceleration scheme for large language models (LLMs) through Speculative Decoding with Semantic Adaptive Tokens (SDSAT). The primary objective of this design is to enhance the LLM model's ability to generate draft tokens more accurately without compromising the model's accuracy. The core strategies involve: 1) Fine-tune the model by incorporating semantic adaptive tokens that possess flexible decoding capabilities without changing its structure, allowing them to generate high-quality draft tokens. 2) By employing a training method that does not affect the standard tokens, the model can acquire parallel decoding abilities atop its original framework with minimal training overhead. 3) We have designed the "two-step-draft-then-verify" generation strategies using both greedy search and nucleus sampling. Experiments conducted on the CodeLlama-13B and 7B models have yielded speed increases of over 3.5X and 3.0X, respectively. Please refer to https://github.com/hasuoshenyun/SDSAT.

cs.CL↗

arXiv2024

Optimizing single-photon quantum radar detection through partially postselected filtering

In this study, we explore an approach aimed at enhancing the transmission or reflection coefficients of absorbing materials through the utilization of joint measurements of entangled photon states. On the one hand, through the implementation of photon catalysis in the reflected channel, we can effectively modify the state of the transmission channel, leading to a notable improvement in the transmission ratio. Similarly, this approach holds potential for significantly amplifying the reflection ratio of absorbing materials, which is useful for detecting cooperative targets. On the other hand, employing statistical counting methods based on the technique of heralding on zero photons, we evaluate the influence of our reflection enhancement protocol for detecting noncooperative targets, which is validated through Monte Carlo simulations of a quantum radar setup affected by Gaussian white noise. Our results demonstrate a remarkable enhancement in the signal-to-noise ratio of imaging, albeit with an increase in mean-square error. These findings highlight the potential practical applications of our approach in the implementation of quantum radar.

quant-ph↗

arXiv2022

Mixture of Experts for Biomedical Question Answering

Biomedical Question Answering (BQA) has attracted increasing attention in recent years due to its promising application prospect. It is a challenging task because the biomedical questions are professional and usually vary widely. Existing question answering methods answer all questions with a homogeneous model, leading to various types of questions competing for the shared parameters, which will confuse the model decision for each single type of questions. In this paper, in order to alleviate the parameter competition problem, we propose a Mixture-of-Expert (MoE) based question answering method called MoEBQA that decouples the computation for different types of questions by sparse routing. To be specific, we split a pretrained Transformer model into bottom and top blocks. The bottom blocks are shared by all the examples, aiming to capture the general features. The top blocks are extended to an MoE version that consists of a series of independent experts, where each example is assigned to a few experts according to its underlying question type. MoEBQA automatically learns the routing strategy in an end-to-end manner so that each expert tends to deal with the question types it is expert in. We evaluate MoEBQA on three BQA datasets constructed based on real examinations. The results show that our MoE extension significantly boosts the performance of question answering models and achieves new state-of-the-art performance. In addition, we elaborately analyze our MoE modules to reveal how MoEBQA works and find that it can automatically group the questions into human-readable clusters.

cs.CL↗

arXiv2022

Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning

A Temporal Knowledge Graph (TKG) is a sequence of KGs corresponding to different timestamps. TKG reasoning aims to predict potential facts in the future given the historical KG sequences. One key of this task is to mine and understand evolutional patterns of facts from these sequences. The evolutional patterns are complex in two aspects, length-diversity and time-variability. Existing models for TKG reasoning focus on modeling fact sequences of a fixed length, which cannot discover complex evolutional patterns that vary in length. Furthermore, these models are all trained offline, which cannot well adapt to the changes of evolutional patterns from then on. Thus, we propose a new model, called Complex Evolutional Network (CEN), which uses a length-aware Convolutional Neural Network (CNN) to handle evolutional patterns of different lengths via an easy-to-difficult curriculum learning strategy. Besides, we propose to learn the model under the online setting so that it can adapt to the changes of evolutional patterns over time. Extensive experiments demonstrate that CEN obtains substantial performance improvement under both the traditional offline and the proposed online settings.

cs.AI↗

arXiv2022

Building Chinese Biomedical Language Models via Multi-Level Text Discrimination

Pre-trained language models (PLMs), such as BERT and GPT, have revolutionized the field of NLP, not only in the general domain but also in the biomedical domain. Most prior efforts in building biomedical PLMs have resorted simply to domain adaptation and focused mainly on English. In this work we introduce eHealth, a Chinese biomedical PLM built from scratch with a new pre-training framework. This new framework pre-trains eHealth as a discriminator through both token- and sequence-level discrimination. The former is to detect input tokens corrupted by a generator and recover their original identities from plausible candidates, while the latter is to further distinguish corruptions of a same original sequence from those of others. As such, eHealth can learn language semantics at both token and sequence levels. Extensive experiments on 11 Chinese biomedical language understanding tasks of various forms verify the effectiveness and superiority of our approach. We release the pre-trained model at \url{https://github.com/PaddlePaddle/Research/tree/master/KG/eHealth} and will also release the code later.

cs.CL↗

arXiv2021

Set-Estimation based Networked Model Predictive Control for Energy Management of Faulty Microgrids

This paper addresses the issue of power flow control for partially faulty microgrids. In microgrid control systems, faults may occur in both electrical and communication layers. This may have severe effects on the operation of microgrids. In addition, disturbances always coexist with faults in microgrids, which may further deteriorate system performance. To address the faults and disturbances simultaneously, a model predictive control (MPC) method based on set-membership estimation (SME) that transmits information via a communication network is proposed. When electrical devices are nonfunctional or communication failures occur, the corresponding system states will become unavailable. To this end, the SME method is employed to estimate the states with the existence of unknown-but-bounded process and measurement disturbances. The networked MPC method is designed to schedule the power dispatch by using the forecasts of photovoltaic (PV) generation and load demand. With these two methods, the fault-tolerant control can be achieved. Further, a deviation compensation method is proposed to compensate for the forecast errors. The effectiveness of the proposed control strategy is demonstrated through wireless communication tests using Raspberry Pis.

eess.SY↗

arXiv2021

Link Prediction on N-ary Relational Facts: A Graph-based Approach

Link prediction on knowledge graphs (KGs) is a key research topic. Previous work mainly focused on binary relations, paying less attention to higher-arity relations although they are ubiquitous in real-world KGs. This paper considers link prediction upon n-ary relational facts and proposes a graph-based approach to this task. The key to our approach is to represent the n-ary structure of a fact as a small heterogeneous graph, and model this graph with edge-biased fully-connected attention. The fully-connected attention captures universal inter-vertex interactions, while with edge-aware attentive biases to particularly encode the graph structure and its heterogeneity. In this fashion, our approach fully models global and local dependencies in each n-ary fact, and hence can more effectively capture associations therein. Extensive evaluation verifies the effectiveness and superiority of our approach. It performs substantially and consistently better than current state-of-the-art across a variety of n-ary relational benchmarks. Our code is publicly available.

cs.AI↗

arXiv2021

Boundary curvature guided shape-programming kirigami sheets

Kirigami, an ancient paper cutting art, offers a promising strategy for 2D-to-3D shape morphing through cut-guided deformation. Existing kirigami designs for target 3D curved shapes rely on intricate cut patterns in thin sheets, making the inverse design challenging. Motivated by the Gauss-Bonnet theorem that correlates the geodesic curvature along the boundary with the topological Gaussian curvature, here, we exploit programming the curvature of cut boundaries rather than complex cut patterns in kirigami sheets for target 3D curved topologies through both forward and inverse designs. Such a new strategy largely simplifies the inverse design. We demonstrate the achievement of varieties of dynamic 3D shape shifting under both mechanical stretching and remote magnetic actuation, and its potential application as an untethered predator-like kirigami soft robot. This study opens a new avenue to encode boundary curvatures for shape-programing materials with potential applications in shape-morphing structures, soft robots, and multifunctional devices.

physics.app-ph↗

arXiv2021

Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction

Entities, as the essential elements in relation extraction tasks, exhibit certain structure. In this work, we formulate such structure as distinctive dependencies between mention pairs. We then propose SSAN, which incorporates these structural dependencies within the standard self-attention mechanism and throughout the overall encoding stage. Specifically, we design two alternative transformation modules inside each self-attention building block to produce attentive biases so as to adaptively regularize its attention flow. Our experiments demonstrate the usefulness of the proposed entity structure and the effectiveness of SSAN. It significantly outperforms competitive baselines, achieving new state-of-the-art results on three popular document-level relation extraction datasets. We further provide ablation and visualization to show how the entity structure guides the model for better relation extraction. Our code is publicly available.

cs.CL↗

arXiv2020

CoKE: Contextualized Knowledge Graph Embedding

Knowledge graph embedding, which projects symbolic entities and relations into continuous vector spaces, is gaining increasing attention. Previous methods allow a single static embedding for each entity or relation, ignoring their intrinsic contextual nature, i.e., entities and relations may appear in different graph contexts, and accordingly, exhibit different properties. This work presents Contextualized Knowledge Graph Embedding (CoKE), a novel paradigm that takes into account such contextual nature, and learns dynamic, flexible, and fully contextualized entity and relation embeddings. Two types of graph contexts are studied: edges and paths, both formulated as sequences of entities and relations. CoKE takes a sequence as input and uses a Transformer encoder to obtain contextualized representations. These representations are hence naturally adaptive to the input, capturing contextual meanings of entities and relations therein. Evaluation on a wide variety of public benchmarks verifies the superiority of CoKE in link prediction and path query answering. It performs consistently better than, or at least equally well as current state-of-the-art in almost every case, in particular offering an absolute improvement of 21.0% in H@10 on path query answering. Our code is available at \url{https://github.com/PaddlePaddle/Research/tree/master/KG/CoKE}.

cs.AI↗

arXiv2018

High resolution electron microscopy for heterogeneous catalysis research

Heterogeneous catalysts are the most important catalysts in industrial reactions. Nanocatalysts, with size ranging from hundreds of nanometers to the atomic scale, possess activities that are closely connected to their structural characteristics such as particle size, surface morphology, and three-dimensional topography. Recently, the development of advanced analytical transmission electron microscopy (TEM) techniques, especially quantitative high-angle annular dark-field (HAADF) imaging and high-energy resolution spectroscopy analysis in scanning transmission electron microscopy (STEM) at the atomic scale, strengthens the power of (S)TEM in analyzing the structural/chemical information of heterogeneous catalysts. Three-dimensional reconstruction from two-dimensional projected images and the real-time recording of structural evolution during catalytic reactions using in-situ (S)TEM methods further broaden the scope of (S)TEM observation. The atomic-scale structural information obtained from high resolution (S)TEM has proven to be of significance for better understanding and designing of new catalysts with enhanced performance.

cond-mat.mtrl-sci↗

arXiv2017

Finite size scaling theory for percolation phase transition

The finite-size scaling theory for continuous phase transition plays an important role in determining critical point and critical exponents from the size-dependent behaviors of quantities in the thermodynamic limit. For percolation phase transition, the finite-size scaling form for the reduced size of largest cluster has been extended to cluster ranked $R$. However, this is invalid for explosive percolation as our results show. Besides, the behaviors of largest increase of largest cluster induced by adding single link or node have also been used to investigate the critical properties of percolation and several new exponents $β_1$, $β_2$, $1/ν_1$ and $1/ν_2$ are defined while their relation with $β/ν$ and $1/ν$ is unknown. Through the analysis of asymptotic properties of size jump behaviors, we obtain correct critical exponents and develop a new approach to finite size scaling theory where sizes of ranked clusters are averaged at same distances from the sample-dependent pseudo-critical point in each realization rather than averaging at same value of control parameter.

cond-mat.stat-mech↗

arXiv2017

Revealing the phase transition behaviors of k-core percolation in random networks

The $k$-core percolation is a fundamental structural transition in complex networks. Through the analysis of the size jump behaviors of $k$-core in the evolution process of networks, we confirm that $k$-core percolation is continuous phase transition when $k=1,2$ while it is a hybrid first-order-second-order phase transition when $k\ge 3$. $2$-core percolation belongs to different universality class from that of $1$-core (giant component) percolation. The discontinuity of $k$-core percolation with $k\ge 3$ can be concluded from largest size jump of $k$-core which will not disappear in the thermodynamic limit while its continuous characteristic is reflected by second largest size jump which converges to zero in power law as $N\to \infty$. Furthermore, along with the previously known exponent $β=0.5$, we obtain a set of exponents which are independent of $k$ when $k\ge 3$ and also different from those critical exponents of $1$-core and $2$-core percolation.

cond-mat.stat-mech↗

arXiv2017

Finite size scaling theory for percolation with multiple giant clusters

A approach of finite size scaling theory for discontinous percolation with multiple giant clusters is developed in this paper. The percolation in generalized Bohman-Frieze-Wormald (BFW) model has already been proved to be discontinuous phase transition. In the evolution process, the size of largest cluster $s_1$ increases in a stairscase way and its fluctuation shows a series of peaks corresponding to the jumps of $s_1$ from one stair to another. Several largest jumps of the size of largest cluster from single edge are studied by extensive Monte Carlo simulation. $\overlineΔ_k(N)$ which is the mean of the $k$th largest jump of largest cluster, $\overline{r}_k(N)$ which is the corresponding averaged edge density, $σ_{Δ,k}(N)$ which is the standard deviation of $Δ_k$ and $σ_{r,k}(N)$ which is the standard deviation of $r_k$ are analyzed. Rich power law behaviours are found for $\overline{r}_k(N)$, $σ_{Δ,k}(N)$ and $σ_{r,k}(N)$ with critical exponents denoted as $1/ν_1$, $(β/ν)_2$ and $1/ν_2$. Unlike continuous percolation where the exact critical thresholds and critical exponent $1/ν_1$ are used for finite size scaling, the size-dependent pseudo critical thresholds $\overline{r}_k(N)$ and $1/ν_2$ works for the data collapse of the curves of largest cluster and its fluctuation in discontinuous percolation in BFW model. Further, data collapse can be obtained part by part. That is, $s_1(r,N)$ can be collapsed for each jump from one stair to another and its fluctuation can be collapsed around each peak with the corresponding $\overline{r}_k(N)$ and $1/ν_2$.

cond-mat.stat-mech↗

arXiv2016

A Review on Mechanics and Mechanical Properties of 2D Materials - Graphene and Beyond

Since the first successful synthesis of graphene just over a decade ago, a variety of two-dimensional (2D) materials (e.g., transition metal-dichalcogenides, hexagonal boron-nitride, etc.) have been discovered. Among the many unique and attractive properties of 2D materials, mechanical properties play important roles in manufacturing, integration and performance for their potential applications. Mechanics is indispensable in the study of mechanical properties, both experimentally and theoretically. The coupling between the mechanical and other physical properties (thermal, electronic, optical) is also of great interest in exploring novel applications, where mechanics has to be combined with condensed matter physics to establish a scalable theoretical framework. Moreover, mechanical interactions between 2D materials and various substrate materials are essential for integrated device applications of 2D materials, for which the mechanics of interfaces (adhesion and friction) has to be developed for the 2D materials. Here we review recent theoretical and experimental works related to mechanics and mechanical properties of 2D materials. While graphene is the most studied 2D material to date, we expect continual growth of interest in the mechanics of other 2D materials beyond graphene.

cond-mat.mes-hall↗

arXiv2015

Critical behaviors and universality classes of percolation phase transitions on two-dimensional square lattice

We have investigated both site and bond percolation on two dimensional lattice under the random rule and the product rule respectively. With the random rule, sites or bonds are added randomly into the lattice. From two candidates picked randomly, the site or bond with the smaller size product of two connected clusters is added when the product rule is taken. Not only the size of the largest cluster but also its size jump are studied to characterize the universality class of percolation. The finite-size scaling forms of giant cluster size and size jump are proposed and used to determine the critical exponents of percolation from Monte Carlo data. It is found that the critical exponents of both size and size jump in random site percolation are equal to that in random bond percolation. With the random rule, site and bond percolation belong to the same universality class. We obtain the critical exponents of the site percolation under the product rule, which are different from that of both random percolation and the bond percolation under the product rule. The universality class of site percolation differs different from that of bond percolation when the product rule is used.

cond-mat.stat-mech↗

arXiv2011

Growth, Characterization and Fermi Surface of Heavy Fermion CeCoIn5 Superconductor

High quality single crystals of heavy Fermion CeCoIn5 superconductor have been grown by flux method with a typical size of (1~2)mm x (1~2)mm x ~0.1 mm. The single crystals are characterized by structural analysis from X-ray diffraction and Laue diffraction, as well as compositional analysis. Magnetic and electrical measurements on the single crystals show a sharp superconducting transition with a transition temperature at Tc(onset) ~ 2.3 K and a transition width of ~0.15 K. The resistivity of the CeCoIn5 crystal exhibits a hump at ~45 K which is typical of a heavy Fermion system. High resolution angle-resolved photoemission spectroscopy (ARPES) measurements of CeCoIn5 reveal clear Fermi surface sheets that are consistent with the band structure calculations when assuming itinerant Ce 4f electrons at low temperature. This work provides important information on the electronic structure of heavy Fermion CeCoIn5 superconductor. It also lays a foundation for further studies on the physical properties and superconducting mechanism of the heavy Fermion superconductors.

cond-mat.supr-con↗