arXiv subjects

Mina Lee

Publications and source records attributed to Mina Lee.

27 records · Page 2Linked to original sources

arXiv2022

On the Opportunities and Risks of Foundation Models

AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotics, reasoning, human interaction) and technical principles(e.g., model architectures, training procedures, data, systems, security, evaluation, theory) to their applications (e.g., law, healthcare, education) and societal impact (e.g., inequity, misuse, economic and environmental impact, legal and ethical considerations). Though foundation models are based on standard deep learning and transfer learning, their scale results in new emergent capabilities,and their effectiveness across so many tasks incentivizes homogenization. Homogenization provides powerful leverage but demands caution, as the defects of the foundation model are inherited by all the adapted models downstream. Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration commensurate with their fundamentally sociotechnical nature.

cs.LG↗

arXiv2022

TempLM: Distilling Language Models into Template-Based Generators

While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content. In contrast, classic template-based systems provide strong guarantees of faithfulness at the cost of fluency. We propose TempLM, which achieves the best of both worlds by distilling a PLM into a template-based generator. On the E2E and SynthBio data-to-text datasets, we show that TempLM is more faithful than the original PLM and is more fluent than prior template systems. Notably, on an out-of-domain evaluation, TempLM reduces a finetuned BART model's unfaithfulness rate from 83% to 0%. In a human study, we find that TempLM's templates substantially improve upon human-written ones in BERTScore.

cs.CL↗

arXiv2022

CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities

Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs' generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities, and reveal its contribution as a writing "collaborator" under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs' promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.

cs.HC↗

arXiv2021

Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality

We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context. To assist humans with writing, lexical substitution systems can suggest words that humans cannot easily think of. However, existing benchmarks depend on human recall as the only source of data, and therefore lack coverage of the substitutes that would be most helpful to humans. Furthermore, annotators often provide substitutes of low quality, which are not actually appropriate in the given context. We collect higher-coverage and higher-quality data by framing lexical substitution as a classification problem, guided by the intuition that it is easier for humans to judge the appropriateness of candidate substitutes than conjure them from memory. To this end, we use a context-free thesaurus to produce candidates and rely on human judgement to determine contextual appropriateness. Compared to the previous largest benchmark, our Swords benchmark has 4.1x more substitutes per target word for the same level of quality, and its substitutes are 1.5x more appropriate (based on human judgement) for the same number of substitutes.

cs.CL↗

arXiv2021

Forecasting e-scooter substitution of direct and access trips by mode and distance

An e-scooter trip model is estimated from four U.S. cities: Portland, Austin, Chicago and New York City. A log-log regression model is estimated for e-scooter trips based on user age, population, land area, and the number of scooters. The model predicts 75K daily e-scooter trips in Manhattan for a deployment of 2000 scooters, which translates to 77 million USD in annual revenue. We propose a novel nonlinear, multifactor model to break down the number of daily trips by the alternative modes of transportation that they would likely substitute based on statistical similarity. The model parameters reveal a relationship with direct trips of bike, walk, carpool, automobile and taxi as well as access/egress trips with public transit in Manhattan. Our model estimates that e-scooters could replace 32% of carpool; 13% of bike; and 7.2% of taxi trips. The distance structure of revenue from access/egress trips is found to differ from that of other substituted trips.

econ.GN↗

arXiv2020

A validated multi-agent simulation test bed to evaluate congestion pricing policies on population segments by time-of-day in New York City

Evaluation of the demand for emerging transportation technologies and policies can vary by time of day due to spillbacks on roadways, rescheduling of travelers' activity patterns, and shifting to other modes that affect the level of congestion. These effects are not well-captured with static travel demand models. We calibrate and validate the first open-source multi-agent simulation model for New York City, called MATSim-NYC, to support agencies in evaluating policies such as congestion pricing. The simulation-based virtual test bed is loaded with an 8M+ synthetic 2016 population calibrated in a prior study. The road network is calibrated to INRIX speed data and average annual daily traffic for a screenline along the East River crossings, resulting in average speed differences of 7.2% on freeways and 17.1% on arterials, leading to average difference of +1.8% from the East River screenline. Validation against transit stations shows an 8% difference from observed counts and median difference of 29% for select road link counts. The model is used to evaluate a congestion pricing plan proposed by the Regional Plan Association and suggests a much higher (127K) car trip reduction compared to their report (59K). The pricing policy would impact the population segment making trips within Manhattan differently from the population segment of trips outside Manhattan. The multiagent simulation can show that 37.3% of the Manhattan segment would be negatively impacted by the pricing compared to 39.9% of the non-Manhattan segment, which has implications for redistribution of congestion pricing revenues. The citywide travel consumer surplus decreases when the congestion pricing goes up from $9.18 to $14 both ways even as it increases for the Charging-related population segment. This implies that increasing pricing from $9.18 to $14 benefits Manhattanites at the expense of the rest of the city.

physics.soc-ph↗

arXiv2020

Enabling Language Models to Fill in the Blanks

We present a simple approach for text infilling, the task of predicting missing spans of text at any position in a document. While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling---a special case of infilling where text is predicted at the end of a document. In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine-tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling, can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machine-generated in the domain of short stories.

cs.CL↗

arXiv2019

Learning Autocomplete Systems as a Communication Game

We study textual autocomplete---the task of predicting a full sentence from a partial sentence---as a human-machine communication game. Specifically, we consider three competing goals for effective communication: use as few tokens as possible (efficiency), transmit sentences faithfully (accuracy), and be learnable to humans (interpretability). We propose an unsupervised approach which tackles all three desiderata by constraining the communication scheme to keywords extracted from a source sentence for interpretability and optimizing the efficiency-accuracy tradeoff. Our experiments show that this approach results in an autocomplete system that is 52% more accurate at a given efficiency level compared to baselines, is robust to user variations, and saves time by nearly 50% compared to typing full sentences.

cs.CL↗

arXiv2019

SPoC: Search-based Pseudocode to Code

We consider the task of mapping pseudocode to long programs that are functionally correct. Given test cases as a mechanism to validate programs, we search over the space of possible translations of the pseudocode to find a program that passes the validation. However, without proper credit assignment to localize the sources of program failures, it is difficult to guide search toward more promising programs. We propose to perform credit assignment based on signals from compilation errors, which constitute 88.7% of program failures. Concretely, we treat the translation of each pseudocode line as a discrete portion of the program, and whenever a synthesized program fails to compile, an error localization method tries to identify the portion of the program responsible for the failure. We then focus search over alternative translations of the pseudocode for those portions. For evaluation, we collected the SPoC dataset (Search-based Pseudocode to Code) containing 18,356 programs with human-authored pseudocode and test cases. Under a budget of 100 program compilations, performing search improves the synthesis success rate over using the top-one translation of the pseudocode from 25.6% to 44.7%.

cs.LG↗