arXiv subjects

Chen Qu

Publications and source records attributed to Chen Qu.

At least 37 records · Page 2Linked to original sources

arXiv2022

Exploring Dual Encoder Architectures for Question Answering

Dual encoders have been used for question-answering (QA) and information retrieval (IR) tasks with good results. Previous research focuses on two major types of dual encoders, Siamese Dual Encoder (SDE), with parameters shared across two encoders, and Asymmetric Dual Encoder (ADE), with two distinctly parameterized encoders. In this work, we explore different ways in which the dual encoder can be structured, and evaluate how these differences can affect their efficacy in terms of QA retrieval tasks. By evaluating on MS MARCO, open domain NQ and the MultiReQA benchmarks, we show that SDE performs significantly better than ADE. We further propose three different improved versions of ADEs by sharing or freezing parts of the architectures between two encoder towers. We find that sharing parameters in projection layers would enable ADEs to perform competitively with or outperform SDEs. We further explore and explain why parameter sharing in projection layer significantly improves the efficacy of the dual encoders, by directly probing the embedding spaces of the two encoder towers with t-SNE algorithm.

cs.CL↗

arXiv2022

Quantum calculations on a new CCSD(T) machine-learned PES reveal the leaky nature of gas-phase $trans$ and $gauche$ ethanol conformers

Ethanol is a molecule of fundamental interest in combustion, astrochemistry, and condensed phase as a solvent. It is characterized by two methyl rotors and $trans$ ($anti$) and $gauche$ conformers, which are known to be very close in energy. Here we show that based on rigorous quantum calculations of the vibrational zero-point state, using a new ab initio potential energy surface (PES), the ground state resembles the $trans$ conformer but substantial delocalization to the $gauche$ conformer is present. This explains experimental issues about the identification and isolation of the two conformers. This "leak" effect is partially quenched when deuterating the OH group, which further demonstrates the need for a quantum mechanical approach. Diffusion Monte Carlo (DMC) and full-dimensional semiclassical dynamics calculations are employed. The new PES is obtained by means of a $Δ$-Machine learning approach starting from a pre-existing low level (LL) density functional theory (DFT) surface. This surface is brought to the CCSD(T) level of theory using a relatively small number of $ab$ $initio$ CCSD(T) energies. Agreement between the corrected PES and direct $ab$ $initio$ results for standard fidelity tests is excellent. One- and two-dimensional discrete variable representation calculations focusing on the $trans$-$gauche$ torsional motion are also reported, in reasonable agreement with the experiment.

physics.chem-ph↗

arXiv2022

A $Δ$-Machine Learning Approach for Force Fields, Illustrated by a CCSD(T) 4-body Correction to the MB-pol Water Potential

$Δ$-Machine Learning ($Δ$-ML) has been shown to effectively and efficiently bring a low-level ML potential energy surface to CCSD(T) quality. Here we propose extending this approach to general force fields, which implicitly or explicitly contain many-body effects. After describing this general approach, we illustrate it for the MB-pol water potential which contains CCSD(T) 2-body and 3-body interactions but relies on the TTM4-F 4-body and higher body interactions. The 4-body MB-pol (TTM4-F) interaction fails at a very short range and for the water hexamer errors up to 0.84 kcal/mol are seen for some isomers, owing mainly to 4-body errors. We apply $Δ$-ML for the 4-body interaction, using a recent dataset of CCSD(T) 4-body energies that we used to develop a new water potential, q-AQUA. This 4-body correction is shown to improve the accuracy of the MB-pol potential for the relative energies of 8 isomers of the water hexamer as well as the harmonic frequencies. The new potential is robust in the very short range and so should be reliable for simulations at high pressure and/or high temperature.

physics.chem-ph↗

arXiv2022

q-AQUA: a many-body CCSD(T) water potential, including 4-body interactions, demonstrates the quantum nature of water from clusters to the liquid phase

Many model potential energy surfaces (PESs) have been reported for water; however, none are strictly from "first principles". Here we report such a potential, based on a many-body representation at the CCSD(T) level of theory up to the ultimate 4-body interaction. The new PES is benchmarked for the isomers of the water hexamer for dissociation energies, harmonic frequencies, and unrestricted diffusion Monte Carlo (DMC) calculations of the zero-point energies of the Prism, Book, and Cage isomers. Dissociation energies of several isomers of the 20-mer agree well with recent benchmark energies. Exploratory DMC calculations on this cluster verify the robustness of the new PES for quantum simulations. The accuracy and speed of the new PES are demonstrated for standard condensed phase properties, i.e., the radial distribution function and the self-diffusion constant. Quantum effects are shown to be substantial for these observables and also needed to bring theory into an excellent agreement with experiment.

physics.chem-ph↗

arXiv2021

Large Dual Encoders Are Generalizable Retrievers

It has been shown that dual encoders trained on one domain often fail to generalize to other domains for retrieval tasks. One widespread belief is that the bottleneck layer of a dual encoder, where the final score is simply a dot-product between a query vector and a passage vector, is too limited to make dual encoders an effective retrieval model for out-of-domain generalization. In this paper, we challenge this belief by scaling up the size of the dual encoder model {\em while keeping the bottleneck embedding size fixed.} With multi-stage training, surprisingly, scaling up the model size brings significant improvement on a variety of retrieval tasks, especially for out-of-domain generalization. Experimental results show that our dual encoders, \textbf{G}eneralizable \textbf{T}5-based dense \textbf{R}etrievers (GTR), outperform %ColBERT~\cite{khattab2020colbert} and existing sparse and dense retrievers on the BEIR dataset~\cite{thakur2021beir} significantly. Most surprisingly, our ablation study finds that GTR is very data efficient, as it only needs 10\% of MS Marco supervised data to achieve the best out-of-domain performance. All the GTR models are released at https://tfhub.dev/google/collections/gtr/1.

cs.IR↗

arXiv2021

Permutationally invariant polynomial regression for energies and gradients, using reverse differentiation, achieves orders of magnitude speed-up with high precision compared to other machine learning methods

Permutationally invariant polynomial (PIP) regression has been used to obtain machine-learned (ML) potential energy surfaces, including analytical gradients, for many molecules and chemical reactions. Recently, the approach has been extended to moderate size molecules and applied to systems up to 15 atoms. The algorithm, including "purification of the basis", is computationally efficient for energies; however, we found that the recent extension to obtain analytical gradients, despite being a remarkable advance over previous methods, could be further improved. Here we report developments to compact further a purified basis and, more significantly, to use the reverse gradient approach to greatly speed up gradient evaluation. We demonstrate this for our recent 4-body water interaction potential. Comparisons of training and testing precision on the MD17 database of energies and gradients (forces) for ethanol against GP-SOAP, ANI, sGDML, PhysNet, pKREG, KRR, and other methods, which were recently assessed by Dral and co-workers, are given. The PIP fits are as precise as those using these methods, but the PIP computation time for energy and force evaluation is shown to be 10 to 1000 times faster. Finally, a new PIP PES is reported for ethanol based on a more extensive dataset of energies and gradients than in the MD17 database. Diffusion Monte Carlo calculations which fail on MD17-based PESs are successful using the new PES.

physics.chem-ph↗

arXiv2021

Natural Language Understanding with Privacy-Preserving BERT

Privacy preservation remains a key challenge in data mining and Natural Language Understanding (NLU). Previous research shows that the input text or even text embeddings can leak private information. This concern motivates our research on effective privacy preservation approaches for pretrained Language Models (LMs). We investigate the privacy and utility implications of applying dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU applications. More importantly, we further propose privacy-adaptive LM pretraining methods and show that our approach can boost the utility of BERT dramatically while retaining the same level of privacy protection. We also quantify the level of privacy preservation and provide guidance on privacy configuration. Our experiments and findings lay the groundwork for future explorations of privacy-preserving NLU with pretrained LMs.

cs.CL↗

arXiv2021

A CCSD(T)-based permutationally invariant polynomial 4-body potential for water

We report a permutationally invariant polynomial (PIP) potential energy surface for the water 4-body interaction. This 12-atom PES is a fit to 2119, symmetry-unique, CCSD(T)-F12a/haTZ (aug-cc-pVTZ basis for 'O' atom and cc-pVTZ basis for 'H' atom) 4-b interaction energies. These come from low-level, direct-dynamics calculations, tetramer fragments from an MD water simulation at 300 K, and from the water hexamer, heptamer, decamer, and 13-mer clusters. The PIP basis is purified to ensure that the 4-b potential goes rigorously to zero in monomer+trimer and dimer+dimer dissociations for all possible such fragments. The 4-b energies of isomers of the hexamer calculated with the new surface are shown to be in better agreement with benchmark CCSD(T) results than those from the MB-pol potential. Other tests validate the high-fidelity of the PES.

physics.chem-ph↗

arXiv2021

Breaking the Coupled Cluster Barrier for Machine Learned Potentials of Large Molecules: The Case of 15-atom Acetylacetone

Machine-learned potential energy surfaces (PESs) for molecules with more than 10 atoms are typically forced to use lower-level electronic structure methods such as density functional theory and second-order Moller-Plesset perturbation theory (MP2). While these are efficient and realistic, they fall short of the accuracy of the ``gold standard'' coupled-cluster method, especially with respect to reaction and isomerization barriers. We report a major step forward in applying a $Δ$-machine learning method to the challenging case of acetylacetone, whose MP2 barrier height for H-atom transfer is low by roughly 1.5 kcal/mol relative to the benchmark CCSD(T) barrier of 3.2 kcal/mol. From a database of 2151 local CCSD(T) energies, and training with as few as 430 energies, we obtain a new PES with a barrier of 3.49 kcal/mol in agreement with the LCCSD(T) one of 3.54 kcal/mol and close to the benchmark value. Tunneling splittings due to H-atom transfer are calculated using this new PES, providing improved estimates over previous ones obtained using an MP2-based PES.

physics.chem-ph↗

arXiv2021

$Δ$-Machine Learning for Potential Energy Surfaces: A PIP approach to bring a DFT-based PES to CCSD(T) Level of Theory

``$Δ$-machine learning" refers to a machine learning approach to bring a property such as a potential energy surface (PES) based on low-level (LL) density functional theory (DFT) energies and gradients to close to a coupled cluster (CC) level of accuracy. Here we present such an approach that uses the permutationally invariant polynomial (PIP) method to fit high-dimensional PESs. The approach is represented by a simple equation, in obvious notation $V_{LL{\rightarrow}CC}=V_{LL}+Δ{V_{CC-LL}}$, and demonstrated for \ce{CH4}, \ce{H3O+}, and $trans$ and $cis$-$N$-methyl acetamide (NMA), \ce{CH3CONHCH3}. For these molecules, the LL PES, $V_{LL}$, is a PIP fit to DFT/B3LYP/6-31+G(d) energies and gradients, and $Δ{V_{CC-LL}}$ is a precise PIP fit obtained using a low-order PIP basis set and based on a relatively small number of CCSD(T) energies. For \ce{CH4} these are new calculations adopting an aug-cc-pVDZ basis, for \ce{H3O+} previous CCSD(T)-F12/aug-cc-pVQZ energies are used, while for NMA new CCSD(T)-F12/aug-cc-pVDZ calculations are performed. With as few as 200 CCSD(T) energies, the new PESs are in excellent agreement with benchmark CCSD(T) results for the small molecules, and for 12-atom NMA training is done with 4696 CCSD(T) energies.

physics.chem-ph↗

arXiv2021

Passage Retrieval for Outside-Knowledge Visual Question Answering

In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowledge, which in our case we define to be a large unstructured passage collection. We first conduct sparse retrieval with BM25 and study expanding the question with object names and image captions. We verify that visual clues play an important role and captions tend to be more informative than object names in sparse retrieval. We then construct a dual-encoder dense retriever, with the query encoder being LXMERT, a multi-modal pre-trained transformer. We further show that dense retrieval significantly outperforms sparse retrieval that uses object expansion. Moreover, dense retrieval matches the performance of sparse retrieval that leverages human-generated captions.

cs.IR↗

arXiv2021

Weakly-Supervised Open-Retrieval Conversational Question Answering

Recent studies on Question Answering (QA) and Conversational QA (ConvQA) emphasize the role of retrieval: a system first retrieves evidence from a large collection and then extracts answers. This open-retrieval ConvQA setting typically assumes that each question is answerable by a single span of text within a particular passage (a span answer). The supervision signal is thus derived from whether or not the system can recover an exact match of this ground-truth answer span from the retrieved passages. This method is referred to as span-match weak supervision. However, information-seeking conversations are challenging for this span-match method since long answers, especially freeform answers, are not necessarily strict spans of any passage. Therefore, we introduce a learned weak supervision approach that can identify a paraphrased span of the known answer in a passage. Our experiments on QuAC and CoQA datasets show that the span-match weak supervisor can only handle conversations with span answers, and has less satisfactory results for freeform answers generated by people. Our method is more flexible as it can handle both span answers and freeform answers. Moreover, our method can be more powerful when combined with the span-match method which shows it is complementary to the span-match method. We also conduct in-depth analyses to show more insights on open-retrieval ConvQA under a weak supervision setting.

cs.IR↗

arXiv2020

Predicting Kovats Retention Indices Using Graph Neural Networks

The \kovats retention index is a dimensionless quantity that characterizes the rate at which a compound is processed through a gas chromatography column. This quantity is independent of many experimental variables and, as such, is considered a near-universal descriptor of retention time on a chromatography column. The \kovats retention indices of a large number of molecules have been determined experimentally. The "NIST 20: GC Method\slash Retention Index Library" database has collected and, more importantly, curated retention indices of a subset of these compounds resulting in a highly valued reference database. The experimental data in the library form an ideal data set for training machine learning models for the prediction of retention indices of unknown compounds. In this article, we describe the training of a graph neural network model to predict the \kovats retention index for compounds in the NIST library and compare this approach with previous work \cite{2019Matyushin}. We predict the \kovats retention index with a mean unsigned error of 28 index units as compared to 44, the putative best result using a convolutional neural network \cite{2019Matyushin}. The NIST library also incorporates an estimation scheme based on a group contribution approach that achieves a mean unsigned error of 114 compared to the experimental data. Our method uses the same input data source as the group contribution approach, making its application straightforward and convenient to apply to existing libraries. Our results convincingly demonstrate the predictive powers of systematic, data-driven approaches leveraging deep learning methodologies applied to chemical data and for the data in the NIST 20 library outperform previous models.

physics.chem-ph↗

arXiv2020

Open-Retrieval Conversational Question Answering

Conversational search is one of the ultimate goals of information retrieval. Recent research approaches conversational search by simplified settings of response ranking and conversational question answering, where an answer is either selected from a given candidate set or extracted from a given passage. These simplifications neglect the fundamental role of retrieval in conversational search. To address this limitation, we introduce an open-retrieval conversational question answering (ORConvQA) setting, where we learn to retrieve evidence from a large collection before extracting answers, as a further step towards building functional conversational search systems. We create a dataset, OR-QuAC, to facilitate research on ORConvQA. We build an end-to-end system for ORConvQA, featuring a retriever, a reranker, and a reader that are all based on Transformers. Our extensive experiments on OR-QuAC demonstrate that a learnable retriever is crucial for ORConvQA. We further show that our system can make a substantial improvement when we enable history modeling in all system components. Moreover, we show that the reranker component contributes to the model performance by providing a regularization effect. Finally, further in-depth analyses are performed to provide new insights into ORConvQA.

cs.IR↗

arXiv2020

IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems

Personal assistant systems, such as Apple Siri, Google Assistant, Amazon Alexa, and Microsoft Cortana, are becoming ever more widely used. Understanding user intent such as clarification questions, potential answers and user feedback in information-seeking conversations is critical for retrieving good responses. In this paper, we analyze user intent patterns in information-seeking conversations and propose an intent-aware neural response ranking model "IART", which refers to "Intent-Aware Ranking with Transformers". IART is built on top of the integration of user intent modeling and language representation learning with the Transformer architecture, which relies entirely on a self-attention mechanism instead of recurrent nets. It incorporates intent-aware utterance attention to derive an importance weighting scheme of utterances in conversation context with the aim of better conversation history understanding. We conduct extensive experiments with three information-seeking conversation data sets including both standard benchmarks and commercial data. Our proposed model outperforms all baseline methods with respect to a variety of metrics. We also perform case studies and analysis of learned user intent and its impact on response ranking in information-seeking conversations to provide interpretation of results.

cs.IR↗

arXiv2019

BERT with History Answer Embedding for Conversational Question Answering

Conversational search is an emerging topic in the information retrieval community. One of the major challenges to multi-turn conversational search is to model the conversation history to answer the current question. Existing methods either prepend history turns to the current question or use complicated attention mechanisms to model the history. We propose a conceptually simple yet highly effective approach referred to as history answer embedding. It enables seamless integration of conversation history into a conversational question answering (ConvQA) model built on BERT (Bidirectional Encoder Representations from Transformers). We first explain our view that ConvQA is a simplified but concrete setting of conversational search, and then we provide a general framework to solve ConvQA. We further demonstrate the effectiveness of our approach under this framework. Finally, we analyze the impact of different numbers of history turns under different settings to provide new insights into conversation history modeling in ConvQA.

cs.IR↗

arXiv2019

Attentive History Selection for Conversational Question Answering

Conversational question answering (ConvQA) is a simplified but concrete setting of conversational search. One of its major challenges is to leverage the conversation history to understand and answer the current question. In this work, we propose a novel solution for ConvQA that involves three aspects. First, we propose a positional history answer embedding method to encode conversation history with position information using BERT in a natural way. BERT is a powerful technique for text representation. Second, we design a history attention mechanism (HAM) to conduct a "soft selection" for conversation histories. This method attends to history turns with different weights based on how helpful they are on answering the current question. Third, in addition to handling conversation history, we take advantage of multi-task learning (MTL) to do answer prediction along with another essential conversation task (dialog act prediction) using a uniform model architecture. MTL is able to learn more expressive and generic representations to improve the performance of ConvQA. We demonstrate the effectiveness of our model with extensive experimental evaluations on QuAC, a large-scale ConvQA dataset. We show that position information plays an important role in conversation history modeling. We also visualize the history attention and provide new insights into conversation history understanding.

cs.IR↗

arXiv2019

A Hybrid Retrieval-Generation Neural Conversation Model

Intelligent personal assistant systems that are able to have multi-turn conversations with human users are becoming increasingly popular. Most previous research has been focused on using either retrieval-based or generation-based methods to develop such systems. Retrieval-based methods have the advantage of returning fluent and informative responses with great diversity. However, the performance of the methods is limited by the size of the response repository. On the other hand, generation-based methods can produce highly coherent responses on any topics. But the generated responses are often generic and not informative due to the lack of grounding knowledge. In this paper, we propose a hybrid neural conversation model that combines the merits of both response retrieval and generation methods. Experimental results on Twitter and Foursquare data show that the proposed model outperforms both retrieval-based methods and generation-based methods (including a recently proposed knowledge-grounded neural conversation model) under both automatic evaluation metrics and human evaluation. We hope that the findings in this study provide new insights on how to integrate text retrieval and text generation models for building conversation systems.

cs.IR↗