arXiv subjects

Yi Mao

Publications and source records attributed to Yi Mao.

At least 55 records · Page 3Linked to original sources

arXiv2022

DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation

Dialog response generation in open domain is an important research topic where the main challenge is to generate relevant and diverse responses. In this paper, we propose a new dialog pre-training framework called DialogVED, which introduces continuous latent variables into the enhanced encoder-decoder pre-training framework to increase the relevance and diversity of responses. With the help of a large dialog corpus (Reddit), we pre-train the model using the following 4 tasks adopted in language models (LMs) and variational autoencoders (VAEs): 1) masked language model; 2) response generation; 3) bag-of-words prediction; and 4) KL divergence reduction. We also add additional parameters to model the turn structure in dialogs to improve the performance of the pre-trained model. We conduct experiments on PersonaChat, DailyDialog, and DSTC7-AVSD benchmarks for response generation. Experimental results show that our model achieves the new state-of-the-art results on all these datasets.

cs.CL↗

arXiv2022

Explanations from Large Language Models Make Small Reasoners Better

Integrating free-text explanations to in-context learning of large language models (LLM) is shown to elicit strong reasoning capabilities along with reasonable explanations. In this paper, we consider the problem of leveraging the explanations generated by LLM to improve the training of small reasoners, which are more favorable in real-production deployment due to their low cost. We systematically explore three explanation generation approaches from LLM and utilize a multi-task learning framework to facilitate small models to acquire strong reasoning power together with explanation generation capabilities. Experiments on multiple reasoning tasks show that our method can consistently and significantly outperform finetuning baselines across different settings, and even perform better than finetuning/prompting a 60x larger GPT-3 (175B) model by up to 9.5% in accuracy. As a side benefit, human evaluation further shows that our method can generate high-quality explanations to justify its predictions, moving towards the goal of explainable AI.

cs.CL↗

arXiv2022

Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation

Knowledge-grounded dialogue systems are challenging to build due to the lack of training data and heterogeneous knowledge sources. Existing systems perform poorly on unseen topics due to limited topics covered in the training data. In addition, heterogeneous knowledge sources make it challenging for systems to generalize to other tasks because knowledge sources in different knowledge representations require different knowledge encoders. To address these challenges, we present PLUG, a language model that homogenizes different knowledge sources to a unified knowledge representation for knowledge-grounded dialogue generation tasks. PLUG is pre-trained on a dialogue generation task conditioned on a unified essential knowledge representation. It can generalize to different downstream knowledge-grounded dialogue generation tasks with a few training examples. The empirical evaluation on two benchmarks shows that our model generalizes well across different knowledge-grounded tasks. It can achieve comparable performance with state-of-the-art methods under a fully-supervised setting and significantly outperforms other methods in zero-shot and few-shot settings.

cs.CL↗

arXiv2022

Theoretical Models of the Atomic Hydrogen Content in Dark Matter Halos

Atomic hydrogen (H I) gas, mostly residing in dark matter halos after cosmic reionization, is the fuel for star formation. Its relation with properties of host halo is the key to understand the cosmic H I distribution. In this work, we propose a flexible, empirical model of H I-halo relation. In this model, while the H I mass depends primarily on the mass of host halo, there is also secondary dependence on other halo properties. We apply our model to the observation data of the Arecibo Fast Legacy ALFA Survey (ALFALFA), and find it can successfully fit to the cosmic H I abundance ($Ω_{\rm HI}$), average H I-halo mass relation $\langle M_{\rm HI}|M_{\rm h}\rangle$, and the H I clustering. The bestfit of the ALFALFA data rejects with high confidence level the model with no secondary halo dependence of H I mass and the model with secondary dependence on halo spin parameter ($λ$), and shows strong dependence on halo formation time ($a_{1/2}$) and halo concentration ($c_{\rm vir}$). In attempt to explain these findings from the perspective of hydrodynamical simulations, the IllustrisTNG simulation confirms the dependence of H I mass on secondary halo parameters. However, the IllustrisTNG results show strong dependence on $λ$ and weak dependence on $c_{\rm vir}$ and $a_{1/2}$, and also predict a much larger value of H I clustering on large scales than observations. This discrepancy between the simulation and observation calls for improvements in understanding the H I-halo relation from both theoretical and observational sides.

astro-ph.GA↗

arXiv2022

OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering

The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value. The intrinsic complexity of handling tables often adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both alignment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an omnivorous pretraining approach that consumes both natural and synthetic data to endow models with these respective abilities. Specifically, given freely available tables, we leverage retrieval to pair them with relevant natural sentences for mask-based pretraining, and synthesize NL questions by converting SQL sampled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the results clearly demonstrate the superiority of our model OmniTab, with the best multitasking approach achieving an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining. Code, pretraining data, and pretrained models are available at https://github.com/jzbjyb/OmniTab.

cs.CL↗

arXiv2022

Implicit Likelihood Inference of Reionization Parameters from the 21 cm Power Spectrum

The first measurements of the 21 cm brightness temperature power spectrum from the epoch of reionization will very likely be achieved in the near future by radio interferometric array experiments such as the Hydrogen Epoch of Reionization Array (HERA) and the Square Kilometre Array (SKA). Standard MCMC analyses use an explicit likelihood approximation to infer the reionization parameters from the 21 cm power spectrum. In this paper, we present a new Bayesian inference of the reionization parameters where the likelihood is implicitly defined through forward simulations using density estimation likelihood-free inference (DELFI). Realistic effects including thermal noise and foreground avoidance are also applied to the mock observations from the HERA and SKA. We demonstrate that this method recovers accurate posterior distributions for the reionization parameters, and outperforms the standard MCMC analysis in terms of the location and size of credible parameter regions. With the minutes-level processing time once the network is trained, this technique is a promising approach for the scientific interpretation of future 21 cm power spectrum observation data. Our code 21cmDELFI-PS is publicly available at this link.

astro-ph.CO↗

arXiv2022

An End-to-End Dialogue Summarization System for Sales Calls

Summarizing sales calls is a routine task performed manually by salespeople. We present a production system which combines generative models fine-tuned for customer-agent setting, with a human-in-the-loop user experience for an interactive summary curation process. We address challenging aspects of dialogue summarization task in a real-world setting including long input dialogues, content validation, lack of labeled data and quality evaluation. We show how GPT-3 can be leveraged as an offline data labeler to handle training data scarcity and accommodate privacy constraints in an industrial setting. Experiments show significant improvements by our models in tackling the summarization and content validation tasks on public datasets.

cs.CL↗

arXiv2022

A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation

Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentence- or document-level detection may fail to provide the fine-grained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes (HAllucination DEtection dataSet). To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models.

cs.CL↗

arXiv2022

Estimation of HII Bubble Size Distribution from 21cm Power Spectrum with Artificial Neural Networks

The bubble size distribution of ionized hydrogen regions probes the information about the morphology of \HII\ bubbles during the reionization. Conventionally, the \HII\ bubble size distribution can be derived from the tomographic imaging data of the redshifted 21~cm signal from the epoch of reionization, which, however, is observationally challenging even for the upcoming large radio interferometer arrays. Given that these interferometers promise to measure the 21~cm power spectrum accurately, we propose a new method, which is based on the artificial neural networks (ANN), to reconstruct the \HII\ bubble size distribution from the 21~cm power spectrum. We demonstrate that the reconstruction from the 21~cm power spectrum can be almost as accurate as directly measured from the imaging data with the fractional error $\lesssim 10\%$, even with thermal noise at the sensitivity level of the Square Kilometre Array. Nevertheless, the reconstruction implicitly exploits the modelling in reionization simulations, and hence the recovered \HII\ bubble size distribution is not an independent summary statistic from the power spectrum, and should be used only as the indicator for understanding \HII\ bubble morphology and its evolution.

astro-ph.CO↗

arXiv2021

Simulation-Based Inference of Reionization Parameters From 3D Tomographic 21 cm Lightcone Images

Tomographic three-dimensional 21 cm images from the epoch of reionization contain a wealth of information about the reionization of the intergalactic medium by astrophysical sources. Conventional power spectrum analysis cannot exploit the full information in the 21 cm data because the 21 cm signal is highly non-Gaussian due to reionization patchiness. We perform a Bayesian inference of the reionization parameters where the likelihood is implicitly defined through forward simulations using density estimation likelihood-free inference (DELFI). We adopt a trained 3D Convolutional Neural Network (CNN) to compress the 3D image data into informative summaries (DELFI-3D CNN). We show that this method recovers accurate posterior distributions for the reionization parameters. Our approach outperforms earlier analysis based on two-dimensional 21 cm images. In contrast, an MCMC analysis of the 3D lightcone-based 21 cm power spectrum alone and using a standard explicit likelihood approximation results in less accurate credible parameter regions than inferred by the DELFI-3D CNN, both in terms of the location and shape of the contours. Our proof-of-concept study implies that the DELFI-3D CNN can effectively exploit more information in the 3D 21 cm images than a 2D CNN or power spectrum analysis. This technique can be readily extended to include realistic effects and is therefore a promising approach for the scientific interpretation of future 21 cm observation data.

astro-ph.CO↗

arXiv2021

Finetuning Pretrained Transformers into RNNs

Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's complexity scales quadratically with sequence length. Efficient transformer variants have received increasing interest in recent works. Among them, a linear-complexity recurrent variant has proven well suited for autoregressive generation. It approximates the softmax attention with randomized or heuristic feature maps, but can be difficult to train and may yield suboptimal accuracy. This work aims to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy. Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune. With a learned feature map, our approach provides an improved tradeoff between efficiency and accuracy over the standard transformer and other recurrent variants. We also show that the finetuning process has lower training cost relative to training these recurrent variants from scratch. As many models for natural language tasks are increasingly dependent on large-scale pretrained transformers, this work presents a viable approach to improving inference efficiency without repeating the expensive pretraining process.

cs.CL↗

arXiv2021

Extracting the astrophysics of reionization from the Ly$α$ forest power spectrum: a first forecast

The impact of cosmic reionization on the Ly$α$ forest power spectrum has recently been shown to be significant even at low redshifts ($z \sim 2$). This memory of reionization survives cosmological time scales because high-entropy mean-density gas is heated to $\sim 3\times10^4$ K by reionization, which is inhomogeneous, and subsequent shocks from denser regions. In the near future, the first measurements of the Ly$α$ forest 3D power spectrum will be very likely achieved by upcoming observational efforts such as the Dark Energy Spectroscopic Instrument (DESI). In addition to abundant cosmological information, these observations have the potential to extract the astrophysics of reionization from the Ly$α$ forest. We forecast, for the first time, the accuracy with which the measurements of Ly$α$ forest 3D power spectrum can place constraints on the reionization parameters with DESI. Specifically, we demonstrate that the constraints on the ionization efficiency, $ζ$, and the threshold mass for haloes that host ionizing sources, $m_{\rm turn}$, will have the $1σ$ error at the level of $ζ= 25.0 \pm 11.6$ and $\log_{10} (m_{\rm turn}/{\rm M}_\odot) = 8.7^{+0.36}_{-0.70}$, respectively. The Ly$α$ forest 3D power spectrum will thus provide an independent probe of reionization, probably even earlier in detection with DESI, with a sensitivity only slightly worse than the upcoming 21 cm power spectrum measurement with the Hydrogen Epoch of Reionization Array (HERA), i.e.\ $σ_{\rm DESI} / σ_{\rm HERA} \approx 1.5$ for $ζ$ and $σ_{\rm DESI}/σ_{\rm HERA} \approx 2.0$ for $\log_{10}(m_{\rm turn} / $M$_\odot)$. Nevertheless, the Ly$α$ forest constraint will be improved about three times tighter than the current constraint from reionization observations with high-z galaxy priors.

astro-ph.CO↗

arXiv2021

Linear Polarization of the 21 cm Line from the Epoch of Reionization

The 21 cm linear polarization due to Thomson scattering off free electrons can probe the distribution of neutral hydrogen in the intergalactic medium during the epoch of reionization, complementary to the 21 cm temperature fluctuations. Previous study (Babich & Loeb 2005) estimated the strength of polarization with a toy model and claimed that it can be detected with 1-month observation of the Square Kilometre Array (SKA). Here we revisit this investigation with account of nonlinear terms due to inhomogeneous reionization, using seminumerical reionization simulations to provide the realistic estimation of the 21 cm TE and EE angular power spectra ($C^{\rm TE}_\ell$ and $C^{\rm EE}_\ell$). We find that (1) both power spectra are enhanced on sub-bubble scales but suppressed on super-bubble scales, compared with previous results; (2) $C^{\rm TE}_\ell$ displays a zero-crossing at $\ell<100$, and its angular scale is sensitive to the scale-dependence of H I bias on large scales; (3) the ratios of the power spectrum to its maximum value during reionization at a given $\ell$, i.e. $C^{\rm TE}_\ell / C^{\rm TE}_{\ell,{\rm max}} $ and $C^{\rm EE}_{\ell}/C^{\rm EE}_{\ell,{\rm max}}$, show robust correlations with the global ionized fraction. However, measurement of this signal will be very challenging not only because the overall strength is weaker than the sensitivity of SKA, but also because of the polarized foregrounds from diffuse synchrotron emission, and Faraday rotation which modifies the observed polarization. Nevertheless, the 21 cm linear polarization signal may still likely be detectable through other approaches, e.g. its cross-correlation with other probes.

astro-ph.CO↗

arXiv2021

The impact of inhomogeneous subgrid clumping on cosmic reionization II: modelling stochasticity

Small-scale density fluctuations can significantly affect reionization but are typically modelled quite crudely. Unresolved fluctuations in numerical simulations and analytical calculations are included using a gas clumping factor, typically assumed to be independent of the local environment. In Paper I, we presented an improved, local density-dependent model for the sub-grid gas clumping. Here we extend this using an empirical stochastic model based on the results from high-resolution numerical simulations which fully resolve all relevant fluctuations. Our model reproduces well both the mean density-clumping relation and its scatter. We applied our stochastic model, along with the mean clumping one and the Paper I deterministic model, to create a large-volume realisation of the clumping field, and used these in radiative transfer simulations of cosmic reionization. Our results show that the simplistic mean clumping model delays reionization compared to local density-dependent models, despite producing fewer recombinations overall. This is due to the very different spatial distribution of clumping, resulting in much higher photoionization rates in the latter cases. The mean clumping model produces smaller HII regions throughout most of the reionization, but those percolate faster at late times. It also causes a significant delay in the 21-cm fluctuations peak and yields lower non-Gaussianity and many fewer bright pixels in the PDF distribution. The stochastic density-dependent model shows relatively minor differences from the deterministic one, mostly concentrated around overlap, where it significantly suppresses the 21-cm fluctuations, and at the bright tail of the 21-cm PDFs, where it produces noticeably more bright pixels.

astro-ph.CO↗

arXiv2021

Quantifying the Need for Attorney Pro Bono Services in Connection with the Social Determinants of Health

The paper estimates the need for additional attorney hours annually to address the legal needs of indigent clients throughout the United States in matters that comprise the so-called social determinants of health (SDoH). The result will inform stakeholders such as policy makers and private donors so they can allocate resources appropriately and design programs to close the do-called justice gap. As a pilot study, the scope of the project covers only a few major justice problems related to the social determinants of health (standard housing, evictions and foreclosures, guardianships for the incapacitated, and victims of domestic violence) because they significantly impact health outcomes and there are data available. Based on our calculations, we estimate that the total number of attorney hours to address only these five legal issues is over 34 million per year. To put that in perspective, the American Bar Association estimated that in 2018, the year from which much of our data comes, the total number of practicing attorneys in the United States was 1,338,678 (Weiss). Thus, to provide the needed hours, every single practicing attorney in the United States would need to contribute about 26 hours a year. While many lawyers do in fact contribute pro bono hours, they address the full range of legal needs that go well beyond just the five we studied.

cs.CY↗

arXiv2021

Investigating X-ray sources during the epoch of reionization with the 21 cm signal

Heating of neutral gas by energetic sources is crucial for the prediction of the 21 cm signal during the epoch of reionization (EoR). To investigate differences induced on statistics of the 21 cm signal by various source types, we use five radiative transfer simulations which have the same stellar UV emission model and varying combinations of more energetic sources, such as X-ray binaries (XRBs), accreting nuclear black holes (BHs) and hot interstellar medium emission (ISM). We find that the efficient heating from the ISM increases the average global 21~cm signal, while reducing its fluctuations and thus power spectrum. A clear impact is also observed in the bispectrum in terms of scale and timing of the transition between a positive and a negative value. The impact of XRBs is similar to that of the ISM, although it is delayed in time and reduced in intensity because of the less efficient heating. Due to the paucity of nuclear BHs, the behaviour of the 21~cm statistics in their presence is very similar to that of a case when only stars are considered, with the exception of the latest stages of reionization, when the effect of BHs is clearly visible. We find that differences between the source scenarios investigated here are larger than the instrumental noise of SKA1-low at $z \gtrsim 7-8$, suggesting that in the future it might be possible to constrain the spectral energy distribution of the sources contributing to the reionization process.

astro-ph.CO↗

arXiv2021

Robust Intensity Mapping Analysis against Foregrounds for the Epoch of Reionization

astro-ph.CO↗

arXiv2021

Antisymmetric Cross-correlation between H I and CO Line Intensity Maps as a New Probe of Cosmic Reionization

Intensity mapping of the H I 21 cm line and the CO 2.61 mm line from the epoch of reionization has emerged as powerful, complementary, probes of the high-redshift Universe. However, both maps and their cross-correlation are dominated by foregrounds. We propose a new analysis by which the signal is unbiased by foregrounds, i.e. it can be measured without foreground mitigation. We construct the antisymmetric part of two-point cross-correlation between intensity maps of the H I 21 cm line and the CO 2.61 mm line, arising because the statistical fluctuations of two fields have different evolution in time. We show that the sign of this new signal can distinguish model-independently whether inside-out reionization happens during some interval of time. More importantly, within the framework of the excursion set model of reionization, we demonstrate that the slope of the dipole of H I-CO cross-power spectrum at large scales is linear to the rate of change of global neutral fraction of hydrogen in a manner independent of reionization parameters, until the slope levels out near the end of reionization, but this trend might possibly depend on the framework of reionization modelling. The H I-CO dipole may be a smoking-gun probe for the speed of reionization, or "standard speedometer" for cosmic reionization. Observations of this new signal will unveil the global reionization history from the midpoint to near the completion of reionization.

astro-ph.CO↗