arXiv subjects

Lili Yu

Publications and source records attributed to Lili Yu.

30 records · Page 2Linked to original sources

arXiv2023

VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation

We propose a new two-stage pre-training framework for video-to-text generation tasks such as video captioning and video question answering: A generative encoder-decoder model is first jointly pre-trained on massive image-text data to learn fundamental vision-language concepts, and then adapted to video data in an intermediate video-text pre-training stage to learn video-specific skills such as spatio-temporal reasoning. As a result, our VideoOFA model achieves new state-of-the-art performance on four Video Captioning benchmarks, beating prior art by an average of 9.7 points in CIDEr score. It also outperforms existing models on two open-ended Video Question Answering datasets, showcasing its generalization capability as a universal video-to-text model.

cs.CV↗

arXiv2023

Scaling Laws for Generative Mixed-Modal Language Models

Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens from VQ-VAEs, speech tokens from HuBERT, BPE tokens for language or code, and so on). To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens. We report new mixed-modal scaling laws that unify the contributions of individual modalities and the interactions between them. Specifically, we explicitly model the optimal synergy and competition due to data and model size as an additive term to previous uni-modal scaling laws. We also find four empirical phenomena observed during the training, such as emergent coordinate-ascent style training that naturally alternates between modalities, guidelines for selecting critical hyper-parameters, and connections between mixed-modal competition and training stability. Finally, we test our scaling law by training a 30B speech-text model, which significantly outperforms the corresponding unimodal models. Overall, our research provides valuable insights into the design and training of mixed-modal generative models, an important new class of unified models that have unique distributional properties.

cs.CL↗

arXiv2021

Nutribullets Hybrid: Multi-document Health Summarization

We present a method for generating comparative summaries that highlights similarities and contradictions in input documents. The key challenge in creating such summaries is the lack of large parallel training data required for training typical summarization systems. To this end, we introduce a hybrid generation approach inspired by traditional concept-to-text systems. To enable accurate comparison between different sources, the model first learns to extract pertinent relations from input documents. The content planning component uses deterministic operators to aggregate these relations after identifying a subset for inclusion into a summary. The surface realization component lexicalizes this information using a text-infilling language model. By separately modeling content selection and realization, we can effectively train them with limited annotations. We implemented and tested the model in the domain of nutrition and health -- rife with inconsistencies. Compared to conventional methods, our framework leads to more faithful, relevant and aggregation-sensitive summarization -- while being equally fluent.

cs.CL↗

arXiv2021

Nutri-bullets: Summarizing Health Studies by Composing Segments

We introduce \emph{Nutri-bullets}, a multi-document summarization task for health and nutrition. First, we present two datasets of food and health summaries from multiple scientific studies. Furthermore, we propose a novel \emph{extract-compose} model to solve the problem in the regime of limited parallel data. We explicitly select key spans from several abstracts using a policy network, followed by composing the selected spans to present a summary via a task specific language model. Compared to state-of-the-art methods, our approach leads to more faithful, relevant and diverse summarization -- properties imperative to this application. For instance, on the BreastCancer dataset our approach gets a more than 50\% improvement on relevance and faithfulness.\footnote{Our code and data is available at \url{https://github.com/darsh10/Nutribullets.}}

cs.CL↗

arXiv2020

Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transport

Selecting input features of top relevance has become a popular method for building self-explaining models. In this work, we extend this selective rationalization approach to text matching, where the goal is to jointly select and align text pieces, such as tokens or sentences, as a justification for the downstream prediction. Our approach employs optimal transport (OT) to find a minimal cost alignment between the inputs. However, directly applying OT often produces dense and therefore uninterpretable alignments. To overcome this limitation, we introduce novel constrained variants of the OT problem that result in highly sparse alignments with controllable sparsity. Our model is end-to-end differentiable using the Sinkhorn algorithm for OT and can be trained without any alignment annotations. We evaluate our model on the StackExchange, MultiNews, e-SNLI, and MultiRC datasets. Our model achieves very sparse rationale selections with high fidelity while preserving prediction accuracy compared to strong attention baseline models.

cs.LG↗

arXiv2020

Interactive Classification by Asking Informative Questions

We study the potential for interaction in natural language classification. We add a limited form of interaction for intent classification, where users provide an initial query using natural language, and the system asks for additional information using binary or multi-choice questions. At each turn, our system decides between asking the most informative question or making the final classification prediction.The simplicity of the model allows for bootstrapping of the system without interaction data, instead relying on simple crowdsourcing tasks. We evaluate our approach on two domains, showing the benefit of interaction and the advantage of learning to balance between asking additional questions and making the final prediction.

cs.CL↗

arXiv2020

On the evolution of word usage of classical Chinese poetry

The hierarchy of classical Chinese poetry has been broadly acknowledged by a number of studies in Chinese literature. However, quantitative investigations about the evolutionary linkages of classical Chinese poetry are limited. The primary goal of this study is to provide quantitative evidence of the evolutionary linkages, with emphasis on character usage, among different period genres of classical Chinese poetry. Specifically, various statistical analyses are performed to find and compare the patterns of character usage in the poems of nine period genres, including shi jing, chu ci, Han shi , Jin shi, Tang shi, Song shi, Yuan shi, Ming shi, and Qing shi. The result of analysis indicates that each of nine period genres has unique patterns of character usage, with some Chinese characters that are preferably used in the poems of a particular period genre. The analysis on the general pattern of character preference implies a decreasing trend in the use of Chinese characters that rarely occur in modern Chinese literature along the timeline of dynastic types of classical Chinese poetry. The phylogenetic analysis based on the distance matrix suggests that the evolutionary linkages of different types of classical Chinese poetry are congruent with their chronological order, suggesting that character frequencies contain phylogenetic information that is useful for inferring evolutionary linkages among various types of classical Chinese poetry. The estimated phylogenetic tree identifies four groups (shi jing, chu ci), (Han shi, Jin shi), (Tang shi, Song shi, Yuan shi), and (Ming shi, Qing shi). The statistical analyses conducted in this study can be generalized to analyze the data sets of general Chinese literature. Such analyses can provide quantitative insights about the evolutionary linkages of general Chinese literature.

physics.soc-ph↗

arXiv2019

Building a Production Model for Retrieval-Based Chatbots

Response suggestion is an important task for building human-computer conversation systems. Recent approaches to conversation modeling have introduced new model architectures with impressive results, but relatively little attention has been paid to whether these models would be practical in a production setting. In this paper, we describe the unique challenges of building a production retrieval-based conversation system, which selects outputs from a whitelist of candidate responses. To address these challenges, we propose a dual encoder architecture which performs rapid inference and scales well with the size of the whitelist. We also introduce and compare two methods for generating whitelists, and we carry out a comprehensive analysis of the model and whitelists. Experimental results on a large, proprietary help desk chat dataset, including both offline metrics and a human evaluation, indicate production-quality performance and illustrate key lessons about conversation modeling in practice.

cs.CL↗

arXiv2015

Parallel Stitching of Two-Dimensional Materials

Diverse parallel stitched two-dimensional heterostructures are synthesized, including metal-semiconductor (graphene-MoS2), semiconductor-semiconductor (WS2-MoS2), and insulator-semiconductor (hBN-MoS2), directly through selective sowing of aromatic molecules as the seeds in chemical vapor deposition (CVD) method. Our methodology enables the large-scale fabrication of lateral heterostructures with arbitrary patterns, and clean and precisely aligned interfaces, which offers tremendous potential for its application in integrated circuits.

cond-mat.mtrl-sci↗

arXiv2013

Large-scale 2D Electronics based on Single-layer MoS2 Grown by Chemical Vapor Deposition

2D nanoelectronics based on single-layer MoS2 offers great advantages for both conventional and ubiquitous applications. This paper discusses the large-scale CVD growth of single-layer MoS2 and fabrication of devices and circuits for the first time. Both digital and analog circuits are fabricated to demonstrate its capability for mixed-signal applications.

cond-mat.mtrl-sci↗

arXiv2012

Integrated Circuits Based on Bilayer MoS2 Transistors

Two-dimensional (2D) materials, such as molybdenum disulfide (MoS2), have been shown to exhibit excellent electrical and optical properties. The semiconducting nature of MoS2 allows it to overcome the shortcomings of zero-bandgap graphene, while still sharing many of graphene's advantages for electronic and optoelectronic applications. Discrete electronic and optoelectronic components, such as field-effect transistors, sensors and photodetectors made from few-layer MoS2 show promising performance as potential substitute of Si in conventional electronics and of organic and amorphous Si semiconductors in ubiquitous systems and display applications. An important next step is the fabrication of fully integrated multi-stage circuits and logic building blocks on MoS2 to demonstrate its capability for complex digital logic and high-frequency ac applications. This paper demonstrates an inverter, a NAND gate, a static random access memory, and a five-stage ring oscillator based on a direct-coupled transistor logic technology. The circuits comprise between two to twelve transistors seamlessly integrated side-by-side on a single sheet of bilayer MoS2. Both enhancement-mode and depletion-mode transistors were fabricated thanks to the use of gate metals with different work functions.

cond-mat.mes-hall↗

arXiv2010

Functionalized Graphene for High Performance Two-dimensional Spintronics Devices

Using first-principles calculations, we explore the possibility of functionalized graphene as high performance two-dimensional spintronics device. Graphene functionalized with O on one side and H on the other side in the chair conformation is found to be a ferromagnetic metal with a spin-filter efficiency up to 85% at finite bias. The ground state of graphene semi-functionalized with F in the chair conformation is an antiferromagnetic semiconductor, and we construct a magnetoresistive device from it by introducing a magnetic field to stabilize its ferromagnetic metallic state. The resulting room-temperature magnetoresistance is up to 5400%, which is one order of magnitude larger than the available experimental values.

cond-mat.mtrl-sci↗