arXiv subjects

Hao Ma

Publications and source records attributed to Hao Ma.

At least 127 records · Page 7Linked to original sources

arXiv2020

Language Models as Fact Checkers?

Recent work has suggested that language models (LMs) store both common-sense and factual knowledge learned from pre-training data. In this paper, we leverage this implicit knowledge to create an effective end-to-end fact checker using a solely a language model, without any external knowledge or explicit retrieval components. While previous work on extracting knowledge from LMs have focused on the task of open-domain question answering, to the best of our knowledge, this is the first work to examine the use of language models as fact checkers. In a closed-book setting, we show that our zero-shot LM approach outperforms a random baseline on the standard FEVER task, and that our fine-tuned LM compares favorably with standard baselines. Though we do not ultimately outperform methods which use explicit knowledge bases, we believe our exploration shows that this method is viable and has much room for exploration.

cs.CL↗

arXiv2020

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the benefits of pretrained models as a function of the number of training samples used in the downstream task. On several text classification tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1%. Our findings indicate that MLM-based models might reach a diminishing return point as the supervised data size increases significantly.

cs.CL↗

arXiv2020

Linformer: Self-Attention with Linear Complexity

Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences, as the standard self-attention mechanism of the Transformer uses $O(n^2)$ time and space with respect to sequence length. In this paper, we demonstrate that the self-attention mechanism can be approximated by a low-rank matrix. We further exploit this finding to propose a new self-attention mechanism, which reduces the overall self-attention complexity from $O(n^2)$ to $O(n)$ in both time and space. The resulting linear transformer, the \textit{Linformer}, performs on par with standard Transformer models, while being much more memory- and time-efficient.

cs.LG↗

arXiv2020

A Combined Data-driven and Physics-driven Method for Steady Heat Conduction Prediction using Deep Convolutional Neural Networks

With several advantages and as an alternative to predict physics field, machine learning methods can be classified into two distinct types: data-driven relying on training data and physics-driven using physics law. Choosing heat conduction problem as an example, we compared the data- and physics-driven learning process with deep Convolutional Neural Networks (CNN). It shows that the convergences of the error to ground truth solution and the residual of heat conduction equation exhibit remarkable differences. Based on this observation, we propose a combined-driven method for learning acceleration and more accurate solutions. With a weighted loss function, reference data and physical equation are able to simultaneously drive the learning. Several numerical experiments are conducted to investigate the effectiveness of the combined method. For the data-driven based method, the introduction of physical equation not only is able to speed up the convergence, but also produces physically more consistent solutions. For the physics-driven based method, it is observed that the combined method is able to speed up the convergence up to 49.0\% by using a not very restrictive coarse reference.

physics.comp-ph↗

arXiv2020

Performance of Linear Classification Algorithms on alpha/gamma Discrimination for LaBr3:Ce Scintillation Detectors with Various Pulse Digitizer Properties

With the development of high-speed readout electronics, the digital pulse shape discrimination (PSD) methods have attracted the attention of more researchers, especially in the field of high energy physics and neutron detection. How to choose a PSD algorithm and corresponding data acquisition system (DAQ) naturally becomes a critical problem to settle down for the detection system designers. In this paper, the relationship between the classification performance of different PSD algorithms and digitizers' sampling properties (including sampling rate and the effective number of bits) has been researched based on $\mathrm{LaBr_3}$:Ce scintillation detectors. A self-developed integrated digitizer configured with five different ADCs and a WavePro 404HD oscilloscope were deployed to digitize the waveforms from $\mathrm{LaBr_3}$:Ce scintillators. Moreover, three PSD methods, charge comparison method (CCM), least square for classification method (LS) and Fisher's linear discriminant analysis (LDA), based on linear model were applied to discriminate the alpha signals from the intrinsic isotope $^{227}$Ac. With the LS method and a 125 MSPS 14-Bit ADC, the FoM value was 1.424$\pm$0.042, which is similar to the result from LDA but 31\% better than the result of CCM. The discrimination results showed that the performances of LS and LDA are less affected by the sampling rate with respect to the CCM method, which reflects in a better PSD capability. The results of this paper can help the developers of detector systems to make a trade-off among sampling properties, desirable discrimination results and the cost of systems.

physics.ins-det↗

arXiv2019

Principle Study of Image Reconstruction Algorithms in Muon Tomography

Muon tomography is a relatively new method of radiography that utilizes muons from cosmic rays and their multiple Coulomb scattering property to distinguish materials. Researchers around the world have been developing various detection systems and image reconstruction algorithms for muon tomography applications, such as nuclear reactor monitoring and cargo inspection for contraband. This paper studies the principle in image reconstruction of muon tomography. Implementation and comparison of some popular algorithms with our simulation dataset will be presented as well as some ideas of future improvements for better image qualities and material discrimination performances.

physics.ins-det↗

arXiv2019

Results of direct dark matter detection with CDEX experiment at CJPL

The China Dark Matter Experiment (CDEX), located at the China Jinping Underground Laboratory (CJPL) whose overburden is about 2400m rock, aims at direct searches of light Weakly Interacting Massive Particles (WIMPs). A single-element 994-gram p-type point contact (PPC) germanium detector (CDEX-1B) runs inside a solid passive shielding system. To achieve lower background, a prototype 10kg PPC germanium detector array (CDEX-10), consisting of three detector strings with three germanium crystals each, is directly immersed in the liquid nitrogen. With the energy threshold of 160eV, the limits on WIMP-nucleus scattering are set by energy spectra and annual modulation analysis, respectively. Incorporating Migdal effect, the data of CDEX-1B are re-analyzed to search sub-GeV WIMPs. Finally, the future plan of CDEX experiment in CJPL-II is introduced.

hep-ex↗

arXiv2019

NetSMF: Large-Scale Network Embedding as Sparse Matrix Factorization

We study the problem of large-scale network embedding, which aims to learn latent representations for network mining applications. Previous research shows that 1) popular network embedding benchmarks, such as DeepWalk, are in essence implicitly factorizing a matrix with a closed form, and 2)the explicit factorization of such matrix generates more powerful embeddings than existing methods. However, directly constructing and factorizing this matrix---which is dense---is prohibitively expensive in terms of both time and space, making it not scalable for large networks. In this work, we present the algorithm of large-scale network embedding as sparse matrix factorization (NetSMF). NetSMF leverages theories from spectral sparsification to efficiently sparsify the aforementioned dense matrix, enabling significantly improved efficiency in embedding learning. The sparsified matrix is spectrally close to the original dense one with a theoretically bounded approximation error, which helps maintain the representation power of the learned embeddings. We conduct experiments on networks of various scales and types. Results show that among both popular benchmarks and factorization based methods, NetSMF is the only method that achieves both high efficiency and effectiveness. We show that NetSMF requires only 24 hours to generate effective embeddings for a large-scale academic collaboration network with tens of millions of nodes, while it would cost DeepWalk months and is computationally infeasible for the dense matrix factorization solution. The source code of NetSMF is publicly available (https://github.com/xptree/NetSMF).

cs.SI↗

arXiv2019

CDEX dark matter experiment: Status and prospects

The China Dark Matter Experiment (CDEX) aims at direct searches of light Weakly Interacting Massive Particles (WIMPs) at the China Jinping Underground Laboratory (CJPL) with an overburden of about 2400m rock. Results from a prototype CDEX-1 994 g p-type Point Contact Germanium(pPCGe) detector are reported. Research programs are pursued to further reduce the physics threshold by improving hardware and data analysis. The CDEX-10 experiment with a pPCGe array of 10 kg target mass range is being tested. The evolution of CDEX program into "CDEX-1T Experiment" with ton-scale germanium detector arrays will also be introduced in this study.

hep-ex↗

arXiv2018

DeepInf: Social Influence Prediction with Deep Learning

Social and information networking activities such as on Facebook, Twitter, WeChat, and Weibo have become an indispensable part of our everyday life, where we can easily access friends' behaviors and are in turn influenced by them. Consequently, an effective social influence prediction for each user is critical for a variety of applications such as online recommendation and advertising. Conventional social influence prediction approaches typically design various hand-crafted rules to extract user- and network-specific features. However, their effectiveness heavily relies on the knowledge of domain experts. As a result, it is usually difficult to generalize them into different domains. Inspired by the recent success of deep neural networks in a wide range of computing applications, we design an end-to-end framework, DeepInf, to learn users' latent feature representation for predicting social influence. In general, DeepInf takes a user's local network as the input to a graph neural network for learning her latent social representation. We design strategies to incorporate both network structures and user-specific features into convolutional neural and attention networks. Extensive experiments on Open Academic Graph, Twitter, Weibo, and Digg, representing different types of social and information networks, demonstrate that the proposed end-to-end model, DeepInf, significantly outperforms traditional feature engineering-based approaches, suggesting the effectiveness of representation learning for social applications.

cs.SI↗

arXiv2018

Measurements of Radon Concentrations Using CR-39 Detectors in China JinPing Underground Laboratory (2015-2016)

Radon background is one of the most critical influences on the ultra-low background and rare-event physical experiments in underground laboratories. To conduct a comprehensive investigation of the radon concentrations in China JinPing underground Laboratory (CJPL), long-term cumulative measurements were carried out using time-integrated passive radon dosimeters containing CR-39 solid-state nuclear track detectors at four sites of CJPL-I. The measurements lasted from May 30,2015 to March 16,2016, a total of 290 days, and the total effective measurement time is 6,953 hours. In the main experiment hall equipped with the ventilation system, the average annual radon concentrations were (55+/-7) Bq m^-3, (58+/-10) Bq^m-3and (53+/-9) Bq m^-3 in three locations respectively. In the storage tunnel without any radon reduction facility, the average annual radon concentrations increased to (345+/-15) Bq m^-3, reflecting the original average annual radon concentration levels in the underground laboratory. Compared with the measured results in the period from 2010 to 2011, the average radon concentration in the main experimental hall was reduced from (86+/-25) Bq m^-3 to (58+/-10) Bq m^-3 due to the well-running ventilation system. Compared with other international underground laboratories, radon concentrations in CJPL-I were at an intermediate level and can meet the demand for low background and rare-event physical experiments.

physics.ins-det↗

arXiv2018

Collaboration Diversity and Scientific Impact

The shift from individual effort to collaborative output has benefited science, with scientific work pursued collaboratively having increasingly led to more highly impactful research than that pursued individually. However, understanding of how the diversity of a collaborative team influences the production of knowledge and innovation is sorely lacking. Here, we study this question by breaking down the process of scientific collaboration of 32.9 million papers over the last five decades. We find that the probability of producing a top-cited publication increases as a function of the diversity of a team of collaborators---namely, the distinct number of institutions represented by the team. We discover striking phenomena where a smaller, yet more diverse team is more likely to generate highly innovative work than a relatively larger team within one institution. We demonstrate that the synergy of collaboration diversity is universal across different generations, research fields, and tiers of institutions and individual authors. Our findings suggest that collaboration diversity strongly and positively correlates with the production of scientific innovation, giving rise to the potential revolution of the policies used by funding agencies and authorities to fund research projects, and broadly the principles used to organize teams, organizations, and societies.

cs.DL↗

arXiv2018

A Web-scale system for scientific knowledge exploration

To enable efficient exploration of Web-scale scientific knowledge, it is necessary to organize scientific publications into a hierarchical concept structure. In this work, we present a large-scale system to (1) identify hundreds of thousands of scientific concepts, (2) tag these identified concepts to hundreds of millions of scientific publications by leveraging both text and graph structure, and (3) build a six-level concept hierarchy with a subsumption-based model. The system builds the most comprehensive cross-domain scientific concept ontology published to date, with more than 200 thousand concepts and over one million relationships.

cs.CL↗

arXiv2018

GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs

We propose a new network architecture, Gated Attention Networks (GaAN), for learning on graphs. Unlike the traditional multi-head attention mechanism, which equally consumes all attention heads, GaAN uses a convolutional sub-network to control each attention head's importance. We demonstrate the effectiveness of GaAN on the inductive node classification problem. Moreover, with GaAN as a building block, we construct the Graph Gated Recurrent Unit (GGRU) to address the traffic speed forecasting problem. Extensive experiments on three real-world datasets show that our GaAN framework achieves state-of-the-art results on both tasks.

cs.LG↗

arXiv2018

Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec

Since the invention of word2vec, the skip-gram model has significantly advanced the research of network embedding, such as the recent emergence of the DeepWalk, LINE, PTE, and node2vec approaches. In this work, we show that all of the aforementioned models with negative sampling can be unified into the matrix factorization framework with closed forms. Our analysis and proofs reveal that: (1) DeepWalk empirically produces a low-rank transformation of a network's normalized Laplacian matrix; (2) LINE, in theory, is a special case of DeepWalk when the size of vertices' context is set to one; (3) As an extension of LINE, PTE can be viewed as the joint factorization of multiple networks' Laplacians; (4) node2vec is factorizing a matrix related to the stationary distribution and transition probability tensor of a 2nd-order random walk. We further provide the theoretical connections between skip-gram based network embedding algorithms and the theory of graph Laplacian. Finally, we present the NetMF method as well as its approximation algorithm for computing network embedding. Our method offers significant improvements over DeepWalk and LINE for conventional network mining tasks. This work lays the theoretical foundation for skip-gram based network embedding methods, leading to a better understanding of latent network representation learning.

cs.SI↗

arXiv2017

A Neural Comprehensive Ranker (NCR) for Open-Domain Question Answering

This paper proposes a novel neural machine reading model for open-domain question answering at scale. Existing machine comprehension models typically assume that a short piece of relevant text containing answers is already identified and given to the models, from which the models are designed to extract answers. This assumption, however, is not realistic for building a large-scale open-domain question answering system which requires both deep text understanding and identifying relevant text from corpus simultaneously. In this paper, we introduce Neural Comprehensive Ranker (NCR) that integrates both passage ranking and answer extraction in one single framework. A Q&A system based on this framework allows users to issue an open-domain question without needing to provide a piece of text that must contain the answer. Experiments show that the unified NCR model is able to outperform the states-of-the-art in both retrieval of relevant text and answer extraction.

cs.CL↗

arXiv2017

A Century of Science: Globalization of Scientific Collaborations, Citations, and Innovations

Progress in science has advanced the development of human society across history, with dramatic revolutions shaped by information theory, genetic cloning, and artificial intelligence, among the many scientific achievements produced in the 20th century. However, the way that science advances itself is much less well-understood. In this work, we study the evolution of scientific development over the past century by presenting an anatomy of 89 million digitalized papers published between 1900 and 2015. We find that science has benefited from the shift from individual work to collaborative effort, with over 90% of the world-leading innovations generated by collaborations in this century, nearly four times higher than they were in the 1900s. We discover that rather than the frequent myopic- and self-referencing that was common in the early 20th century, modern scientists instead tend to look for literature further back and farther around. Finally, we also observe the globalization of scientific development from 1900 to 2015, including 25-fold and 7-fold increases in international collaborations and citations, respectively, as well as a dramatic decline in the dominant accumulation of citations by the US, the UK, and Germany, from ~95% to ~50% over the same period. Our discoveries are meant to serve as a starter for exploring the visionary ways in which science has developed throughout the past century, generating insight into and an impact upon the current scientific innovations and funding policies.

cs.DL↗

arXiv2017

KeyVec: Key-semantics Preserving Document Representations

Previous studies have demonstrated the empirical success of word embeddings in various applications. In this paper, we investigate the problem of learning distributed representations for text documents which many machine learning algorithms take as input for a number of NLP tasks. We propose a neural network model, KeyVec, which learns document representations with the goal of preserving key semantics of the input text. It enables the learned low-dimensional vectors to retain the topics and important information from the documents that will flow to downstream tasks. Our empirical evaluations show the superior quality of KeyVec representations in two different document understanding tasks.

cs.CL↗