Searcharxiv⌕ Search

arXiv subjects

Heng Li

Publications and source records attributed to Heng Li.

At least 199 records · Page 11Linked to original sources

Twelve years of SAMtools and BCFtools

Background SAMtools and BCFtools are widely used programs for processing and analysing high-throughput sequencing data. Findings The first version appeared online twelve years ago and has been maintained and further developed ever since, with many new features and improvements added over the years. The SAMtools and BCFtools packages represent a unique collection of tools that have been used in numerous other software projects and countless genomic pipelines. Conclusion Both SAMtools and BCFtools are freely available on GitHub under the permissive MIT licence, free for both non-commercial and commercial use. Both packages have been installed over a million times via Bioconda. The source code and documentation are available from http://www.htslib.org.

q-bio.GN↗

BGM: Building a Dynamic Guidance Map without Visual Images for Trajectory Prediction

Visual images usually contain the informative context of the environment, thereby helping to predict agents' behaviors. However, they hardly impose the dynamic effects on agents' actual behaviors due to the respectively fixed semantics. To solve this problem, we propose a deterministic model named BGM to construct a guidance map to represent the dynamic semantics, which circumvents to use visual images for each agent to reflect the difference of activities in different periods. We first record all agents' activities in the scene within a period close to the current to construct a guidance map and then feed it to a Context CNN to obtain their context features. We adopt a Historical Trajectory Encoder to extract the trajectory features and then combine them with the context feature as the input of the social energy based trajectory decoder, thus obtaining the prediction that meets the social rules. Experiments demonstrate that BGM achieves state-of-the-art prediction accuracy on the two widely used ETH and UCY datasets and handles more complex scenarios.

cs.CV↗

Haplotype-resolved de novo assembly with phased assembly graphs

Haplotype-resolved de novo assembly is the ultimate solution to the study of sequence variations in a genome. However, existing algorithms either collapse heterozygous alleles into one consensus copy or fail to cleanly separate the haplotypes to produce high-quality phased assemblies. Here we describe hifiasm, a new de novo assembler that takes advantage of long high-fidelity sequence reads to faithfully represent the haplotype information in a phased assembly graph. Unlike other graph-based assemblers that only aim to maintain the contiguity of one haplotype, hifiasm strives to preserve the contiguity of all haplotypes. This feature enables the development of a graph trio binning algorithm that greatly advances over standard trio binning. On three human and five non-human datasets, including California redwood with a $\sim$30-gigabase hexaploid genome, we show that hifiasm frequently delivers better assemblies than existing tools and consistently outperforms others on haplotype-resolved assembly.

q-bio.GN↗

Low Threshold Bound State in the Continuum Lasers in Hybrid Lattice Resonance Metasurfaces

Bound states in the continuum (BICs) have attracted much attention in recent years due to the infinite quality factor (Q-factor) resonance and extremely localized field. In this study, BICs have been demonstrated by dielectric metasurfaces with hybrid surface lattice resonance (SLR) in the experiment. By breaking the symmetry of geometry, SLR can be easily switched between BICs and quasi-BICs. Comparing with literature, switching between BICs and quasi-BICs is usually accompanied by wavelength shift. Here, a design rule is proposed to prevent the wavelength shift when the Q-factor is changing. Also, such a design also makes subsequent identification of the laser threshold more credible. Due to the high Q-factor, low threshold laser is one of the intuitive applications of BICs. Utilize the high localized ability of BICs, low threshold BICs laser can be achieved by the dielectric metasurface immersed with Rhodamine 6G. Interestingly, due to the high Q-factor resonance of BICs, the laser signals and images can be observed in almost transparent samples. Not only the BICs laser is demonstrated in the experiment, but also the mechanism of BICs is deeply analyzed. This study can help readers better understand this novel feature of BICs, and provide the way for engineer BICs metasurfaces. The device can provide various applications, including laser, optical sensing, non-linear optics enhancement, and single-photon source.

physics.app-ph↗

The design and construction of reference pangenome graphs

The recent advances in sequencing technologies enables the assembly of individual genomes to the reference quality. How to integrate multiple genomes from the same species and to make the integrated representation accessible to biologists remain an open challenge. Here we propose a graph-based data model and associated formats to represent multiple genomes while preserving the coordinate of the linear reference genome. We implemented our ideas in the minigraph toolkit and demonstrate that we can efficiently construct a pangenome graph and compactly encode tens of thousands of structural variants missing from the current reference genome.

q-bio.GN↗

Logram: Efficient Log Parsing Using n-Gram Dictionaries

Software systems usually record important runtime information in their logs. Logs help practitioners understand system runtime behaviors and diagnose field failures. As logs are usually very large in size, automated log analysis is needed to assist practitioners in their software operation and maintenance efforts. Typically, the first step of automated log analysis is log parsing, i.e., converting unstructured raw logs into structured data. However, log parsing is challenging, because logs are produced by static templates in the source code (i.e., logging statements) yet the templates are usually inaccessible when parsing logs. Prior work proposed automated log parsing approaches that have achieved high accuracy. However, as the volume of logs grows rapidly in the era of cloud computing, efficiency becomes a major concern in log parsing. In this work, we propose an automated log parsing approach, Logram, which leverages n-gram dictionaries to achieve efficient log parsing. We evaluated Logram on 16 public log datasets and compared Logram with five state-of-the-art log parsing approaches. We found that Logram achieves a similar parsing accuracy to the best existing approaches while outperforms these approaches in efficiency (i.e., 1.8 to 5.1 times faster than the second fastest approaches). Furthermore, we deployed Logram on Spark and we found that Logram scales out efficiently with the number of Spark nodes (e.g., with near-linear scalability) without sacrificing parsing accuracy. In addition, we demonstrated that Logram can support effective online parsing of logs, achieving similar parsing results and efficiency with the offline mode.

cs.SE↗

Deep Learning of Subsurface Flow via Theory-guided Neural Network

Active researches are currently being performed to incorporate the wealth of scientific knowledge into data-driven approaches (e.g., neural networks) in order to improve the latter's effectiveness. In this study, the Theory-guided Neural Network (TgNN) is proposed for deep learning of subsurface flow. In the TgNN, as supervised learning, the neural network is trained with available observations or simulation data while being simultaneously guided by theory (e.g., governing equations, other physical constraints, engineering controls, and expert knowledge) of the underlying problem. The TgNN can achieve higher accuracy than the ordinary Artificial Neural Network (ANN) because the former provides physically feasible predictions and can be more readily generalized beyond the regimes covered with the training data. Furthermore, the TgNN model is proposed for subsurface flow with heterogeneous model parameters. Several numerical cases of two-dimensional transient saturated flow are introduced to test the performance of the TgNN. In the learning process, the loss function contains data mismatch, as well as PDE constraint, engineering control, and expert knowledge. After obtaining the parameters of the neural network by minimizing the loss function, a TgNN model is built that not only fits the data, but also adheres to physical/engineering constraints. Predicting the future response can be easily realized by the TgNN model. In addition, the TgNN model is tested in more complicated scenarios, such as prediction with changed boundary conditions, learning from noisy data or outliers, transfer learning, and engineering controls. Numerical results demonstrate that the TgNN model achieves much better predictability, reliability, and generalizability than ANN models due to the physical/engineering constraints in the former.

cs.LG↗

Efficient Architecture-Aware Acceleration of BWA-MEM for Multicore Systems

Innovations in Next-Generation Sequencing are enabling generation of DNA sequence data at ever faster rates and at very low cost. Large sequencing centers typically employ hundreds of such systems. Such high-throughput and low-cost generation of data underscores the need for commensurate acceleration in downstream computational analysis of the sequencing data. A fundamental step in downstream analysis is mapping of the reads to a long reference DNA sequence, such as a reference human genome. Sequence mapping is a compute-intensive step that accounts for more than 30% of the overall time of the GATK workflow. BWA-MEM is one of the most widely used tools for sequence mapping and has tens of thousands of users. In this work, we focus on accelerating BWA-MEM through an efficient architecture aware implementation, while maintaining identical output. The volume of data requires distributed computing environment, usually deploying multicore processors. Since the application can be easily parallelized for distributed memory systems, we focus on performance improvements on a single socket multicore processor. BWA-MEM run time is dominated by three kernels, collectively responsible for more than 85% of the overall compute time. We improved the performance of these kernels by 1) improving cache reuse, 2) simplifying the algorithms, 3) replacing small fragmented memory allocations with a few large contiguous ones, 4) software prefetching, and 5) SIMD utilization wherever applicable - and massive reorganization of the source code enabling these improvements. As a result, we achieved nearly 2x, 183x, and 8x speedups on the three kernels, respectively, resulting in up to 3.5x and 2.4x speedups on end-to-end compute time over the original BWA-MEM on single thread and single socket of Intel Xeon Skylake processor. To the best of our knowledge, this is the highest reported speedup over BWA-MEM.

cs.DC↗

Unified Flow and Thermal Law-of-the-wall Complete Formulations for Type-A Turbulent Boundary Layers

In-depth analyses of existing direct numerical simulations (DNS) data from various sources supported a logical and important classification of generic turbulent boundary layers (TBL), namely Type-A, -B and -C TBL, based on distribution patterns of time-averaged wall-shear stress. Among these types, Type-A TBL and its related law, as represented by the DNS data of turbulence on a zero-pressure-gradient semi-infinite flat-plate, was investigated in terms of analytical formulations of velocity independent on Reynolds ( ) number. With reference to the analysis from von Karman in developing the conventional law-of-the-wall, the current study first physically distinguished the time-averaged local scale used by von Karman from the ensemble-averaged scale defined in the paper, and then derived the governing equations with the -independency under the ensemble-averaged scales. Based on indicator function (IDF) and TBL thickness, the sublayer partitions were rigorously defined. The analytical formulations for entire TBL, namely the complete law-of-the-wall, were established, including the formula in inner, buffer, semi-logarithmic (semi-log) and wake layer. The researches were featured by introducing the general damping and enhancing functions (GDF and GEF) and applying these functions to both linear and logarithmic coordinates. These law formulations were proved uniform and consistent in time-averaged local and ensemble-averaged scales, which were validated by the existing DNS and experiment data. Based on the similarity of relevant properly-scaled governing equations, the law formulations were logically reasoned being applicable to the temperature in Type-A thermal TBL. The findings advance the current understandings of the conventional TBL theory and its well-known foundations of law-of-the-wall.

physics.flu-dyn↗

Identifying centromeric satellites with dna-brnn

Summary: Human alpha satellite and satellite 2/3 contribute to several percent of the human genome. However, identifying these sequences with traditional algorithms is computationally intensive. Here we develop dna-brnn, a recurrent neural network to learn the sequences of the two classes of centromeric repeats. It achieves high similarity to RepeatMasker and is times faster. Dna-brnn explores a novel application of deep learning and may accelerate the study of the evolution of the two repeat classes. Availability and implementation: https://github.com/lh3/dna-nn Contact: hli@jimmy.harvard.edu

q-bio.GN↗

Unified Flow and Thermal Law-of-the-wall for Type-A and B Turbulent Boundary Layers

Systematic study of the existing statistical data from direct numerical simulations (DNS) lead to a logical and important classification of generic turbulent boundary layers (TBL), namely Type A,B and C TBL, based on the distribution patterns of time averaged wall shear stress. Among these three types, Type A and B TBLs are investigated in terms of its universal statistical laws of velocity and temperature with independency on both Reynolds (Re) and Prandtl (Pr) numbers. With reference to the analysis by von Karman in developing the conventional law of the wall for Type A TBL, the current study first physically distinguishes the time averaged local velocity scale from the ensemble averaged velocity scale, and properly defines the TBL temperature scale based on wall heat flux, similar to the well known frictional velocity scale based on wall shear stress. The unified law of the wall formulations are then derived for the inner layer of Type A and B thermal TBLs using these scales. The formulations are constructed by introducing the general damping and enhancing functions in inner layer, and further, these functions are extended to logarithmic coordinate based on the multiscale characteristics of TBL velocity profiles. The validations of the law are conducted both by the DNS guided near wall integration of Type B TBL governing equations, which uncovered the Re and Pr independency mechanisms of the law. The research advances the current understandings of the conventional TBL theory and develop the complete analytical forms of law of the wall for Type A and B TBLs, including inner layer, transition layer, logarithmic layer and wake layer.

physics.flu-dyn↗

Giant, anomalous piezo-impedance of silicon-on-insulator

A giant, anomalous piezo-response of fully-depleted silicon-on-insulator (FD-SOI) devices under mechanical stress is demonstrated using impedance spectroscopy. This piezo-response strongly depends on the measurement frequency, $ω$, and consists of both a piezoresistance (PZR) and piezocapacitance whose maximum values are $π_R = -1100 \times 10^{-11}$ Pa$^{-1}$ and $π_C = -900 \times 10^{-11}$ Pa$^{-1}$ respectively. These values should be compared with the usual bulk PZR in p-type silicon, $π_R= 70 \times 10^{-11}$ Pa$^{-1}$. The observations are well described using models of space charge limited electron and hole currents in the presence of fast electronic traps having stress-dependent capture ($ω_c$) and emission rates. Under steady-state conditions (i.e. when $ω\ll ω_c$) where the impedance spectroscopy measurements yield results that are directly comparable with previously published reports of PZR in depleted, silicon nano-objects, the overall piezo-response is just the usual, bulk silicon PZR. Anomalous PZR is observed only under non-steady-state conditions when $ω\approx ω_c$, with a symmetry suggesting that the electro-mechanically active fast traps are native Pb$_0$ interface defects. The observations suggest new functionalities for FD-SOI, and shed light on the debate over the PZR of carrier depleted nano-silicon.

cond-mat.mes-hall↗

Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning

Ranking is a fundamental and widely studied problem in scenarios such as search, advertising, and recommendation. However, joint optimization for multi-scenario ranking, which aims to improve the overall performance of several ranking strategies in different scenarios, is rather untouched. Separately optimizing each individual strategy has two limitations. The first one is lack of collaboration between scenarios meaning that each strategy maximizes its own objective but ignores the goals of other strategies, leading to a sub-optimal overall performance. The second limitation is the inability of modeling the correlation between scenarios meaning that independent optimization in one scenario only uses its own user data but ignores the context in other scenarios. In this paper, we formulate multi-scenario ranking as a fully cooperative, partially observable, multi-agent sequential decision problem. We propose a novel model named Multi-Agent Recurrent Deterministic Policy Gradient (MA-RDPG) which has a communication component for passing messages, several private actors (agents) for making actions for ranking, and a centralized critic for evaluating the overall performance of the co-working actors. Each scenario is treated as an agent (actor). Agents collaborate with each other by sharing a global action-value function (the critic) and passing messages that encodes historical information across scenarios. The model is evaluated with online settings on a large E-commerce platform. Results show that the proposed model exhibits significant improvements against baselines in terms of the overall performance.

cs.AI↗

Heterogeneity of Synaptic Input Connectivity Regulates Spike-based Neuronal Avalanches

Our mysterious brain is believed to operate near a non-equilibrium point and generate critical self-organized avalanches in neuronal activity. Recent experimental evidence has revealed significant heterogeneity in both synaptic input and output connectivity, but whether the structural heterogeneity participates in the regulation of neuronal avalanches remains poorly understood. By computational modelling, we predict that different types of structural heterogeneity contribute distinct effects on avalanche neurodynamics. In particular, neuronal avalanches can be triggered at an intermediate level of input heterogeneity, but heterogeneous output connectivity cannot evoke avalanche dynamics. In the criticality region, the co-emergence of multi-scale cortical activities is observed, and both the avalanche dynamics and neuronal oscillations are modulated by the input heterogeneity. Remarkably, we show similar results can be reproduced in networks with various types of in- and out-degree distributions. Overall, these findings not only provide details on the underlying circuitry mechanisms of nonrandom synaptic connectivity in the regulation of neuronal avalanches, but also inspire testable hypotheses for future experimental studies.

q-bio.NC↗

Minimap2: pairwise alignment for nucleotide sequences

Motivation: Recent advances in sequencing technologies promise ultra-long reads of $\sim$100 kilo bases (kb) in average, full-length mRNA or cDNA reads in high throughput and genomic contigs over 100 mega bases (Mb) in length. Existing alignment programs are unable or inefficient to process such data at scale, which presses for the development of new alignment algorithms. Results: Minimap2 is a general-purpose alignment program to map DNA or long mRNA sequences against a large reference database. It works with accurate short reads of $\ge$100bp in length, $\ge$1kb genomic reads at error rate $\sim$15%, full-length noisy Direct RNA or cDNA reads, and assembly contigs or closely related full chromosomes of hundreds of megabases in length. Minimap2 does split-read alignment, employs concave gap cost for long insertions and deletions (INDELs) and introduces new heuristics to reduce spurious alignments. It is 3-4 times faster than mainstream short-read mappers at comparable accuracy and $\ge$30 times faster at higher accuracy for both genomic and mRNA reads, surpassing most aligners specialized in one type of alignment. Availability and implementation: https://github.com/lh3/minimap2 Contact: hengli@broadinstitute.org

q-bio.GN↗

BGT: efficient and flexible genotype query across many samples

Summary: BGT is a compact format, a fast command line tool and a simple web application for efficient and convenient query of whole-genome genotypes and frequencies across tens to hundreds of thousands of samples. On real data, it encodes the haplotypes of 32,488 samples across 39.2 million SNPs into a 7.4GB database and decodes a couple of hundred million genotypes per CPU second. The high performance enables real-time responses to complex queries. Availability and implementation: https://github.com/lh3/bgt Contact: hengli@broadinstitute.org

q-bio.GN↗

Minimap and miniasm: fast mapping and de novo assembly for noisy long sequences

Motivation: Single Molecule Real-Time (SMRT) sequencing technology and Oxford Nanopore technologies (ONT) produce reads over 10kbp in length, which have enabled high-quality genome assembly at an affordable cost. However, at present, long reads have an error rate as high as 10-15%. Complex and computationally intensive pipelines are required to assemble such reads. Results: We present a new mapper, minimap, and a de novo assembler, miniasm, for efficiently mapping and assembling SMRT and ONT reads without an error correction stage. They can often assemble a sequencing run of bacterial data into a single contig in a few minutes, and assemble 45-fold C. elegans data in 9 minutes, orders of magnitude faster than the existing pipelines. We also introduce a pairwise read mapping format (PAF) and a graphical fragment assembly format (GFA), and demonstrate the interoperability between ours and current tools. Availability and implementation: https://github.com/lh3/minimap and https://github.com/lh3/miniasm Contact: hengli@broadinstitute.org

q-bio.GN↗