Searcharxiv⌕ Search

arXiv subjects

Peter West

Publications and source records attributed to Peter West.

At least 73 records · Page 4Linked to original sources
arXiv2022

Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference

Causal inference studies using textual social media data can provide actionable insights on human behavior. Making accurate causal inferences with text requires controlling for confounding which could otherwise impart bias. Recently, many different methods for adjusting for confounders have been proposed, and we show that these existing methods disagree with one another on two datasets inspired by previous social media studies. Evaluating causal methods is challenging, as ground truth counterfactuals are almost never available. Presently, no empirical evaluation framework for causal methods using text exists, and as such, practitioners must select their methods without guidance. We contribute the first such framework, which consists of five tasks drawn from real world studies. Our framework enables the evaluation of any casual inference method using text. Across 648 experiments and two datasets, we evaluate every commonly used causal inference method and identify their strengths and weaknesses to inform social media researchers seeking to use such methods, and guide future improvements. We make all tasks, data, and models public to inform applications and encourage additional research.

cs.CL↗
arXiv2022

Probing Factually Grounded Content Transfer with Factual Ablation

Despite recent success, large neural models often generate factually incorrect text. Compounding this is the lack of a standard automatic evaluation for factuality--it cannot be meaningfully improved if it cannot be measured. Grounded generation promises a path to solving both of these problems: models draw on a reliable external document (grounding) for factual information, simplifying the challenge of factuality. Measuring factuality is also simplified--to factual consistency, testing whether the generation agrees with the grounding, rather than all facts. Yet, without a standard automatic metric for factual consistency, factually grounded generation remains an open problem. We study this problem for content transfer, in which generations extend a prompt, using information from factual grounding. Particularly, this domain allows us to introduce the notion of factual ablation for automatically measuring factual consistency: this captures the intuition that the model should be less likely to produce an output given a less relevant grounding document. In practice, we measure this by presenting a model with two grounding documents, and the model should prefer to use the more factually relevant one. We contribute two evaluation sets to measure this. Applying our new evaluation, we propose multiple novel methods improving over strong baselines.

cs.CL↗
arXiv2022

Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics

Neural sequence models trained with maximum likelihood estimation have led to breakthroughs in many tasks, where success is defined by the gap between training and test performance. However, their ability to achieve stronger forms of generalization remains unclear. We consider the problem of symbolic mathematical integration, as it requires generalizing systematically beyond the test set. We develop a methodology for evaluating generalization that takes advantage of the problem domain's structure and access to a verifier. Despite promising in-distribution performance of sequence-to-sequence models in this domain, we demonstrate challenges in achieving robustness, compositionality, and out-of-distribution generalization, through both carefully constructed manual test suites and a genetic algorithm that automatically finds large collections of failures in a controllable manner. Our investigation highlights the difficulty of generalizing well with the predominant modeling and learning approach, and the importance of evaluating beyond the test set, across different aspects of generalization.

cs.LG↗
arXiv2022

The string little algebra

We consider the string, like point particles and branes, to be an irreducible representation of the semi-direct product of the Cartan involution invariant subalgebra of E11 and its vector representation. We show that the subalgebra that preserves the string charges, the string little algebra, is essentially the Borel subalgebra of E9. We also show that the known string physical states carry a representation of parts of this algebra.

hep-th↗
arXiv2022

The role of the 1.5 order formalism and the gauging of spacetime groups in the development of gravity and supergravity theories

The 1.5 formalism played a key role in the discovery of supergravity and it has been used to prove the invariance of essentially all supergravity theories under local supersymmetry. It emerged from the gauging of the super Poincare group to find supergravity. We review both of these developments as well as the auxiliary fields for simple supergravity and its most general coupling to matter using the tensor calculus.

hep-th↗
arXiv2021

Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models

Publicly available, large pretrained LanguageModels (LMs) generate text with remarkable quality, but only sequentially from left to right. As a result, they are not immediately applicable to generation tasks that break the unidirectional assumption, such as paraphrasing or text-infilling, necessitating task-specific supervision. In this paper, we present Reflective Decoding, a novel unsupervised algorithm that allows for direct application of unidirectional LMs to non-sequential tasks. Our 2-step approach requires no supervision or even parallel corpora, only two off-the-shelf pretrained LMs in opposite directions: forward and backward. First, in the contextualization step, we use LMs to generate ensembles of past and future contexts which collectively capture the input (e.g. the source sentence for paraphrasing). Second, in the reflection step, we condition on these "context ensembles", generating outputs that are compatible with them. Comprehensive empirical results demonstrate that Reflective Decoding outperforms strong unsupervised baselines on both paraphrasing and abductive text infilling, significantly narrowing the gap between unsupervised and supervised methods. Reflective Decoding surpasses multiple supervised baselines on various metrics including human evaluation.

cs.CL↗
arXiv2021

NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics

The dominant paradigm for neural text generation is left-to-right decoding from autoregressive language models. Constrained or controllable generation under complex lexical constraints, however, requires foresight to plan ahead feasible future paths. Drawing inspiration from the A* search algorithm, we propose NeuroLogic A*esque, a decoding algorithm that incorporates heuristic estimates of future cost. We develop efficient lookahead heuristics that are efficient for large-scale language models, making our method a drop-in replacement for common techniques such as beam search and top-k sampling. To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction. Our approach outperforms competitive baselines on five generation tasks, and achieves new state-of-the-art performance on table-to-text generation, constrained machine translation, and keyword-constrained generation. The improvements are particularly notable on tasks that require complex constraint satisfaction or in few-shot or zero-shot settings. NeuroLogic A*esque illustrates the power of decoding for improving and enabling new capabilities of large-scale language models.

cs.CL↗
arXiv2021

Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning

Abductive and counterfactual reasoning, core abilities of everyday human cognition, require reasoning about what might have happened at time t, while conditioning on multiple contexts from the relative past and future. However, simultaneous incorporation of past and future contexts using generative language models (LMs) can be challenging, as they are trained either to condition only on the past context or to perform narrowly scoped text-infilling. In this paper, we propose DeLorean, a new unsupervised decoding algorithm that can flexibly incorporate both the past and future contexts using only off-the-shelf, left-to-right language models and no supervision. The key intuition of our algorithm is incorporating the future through back-propagation, during which, we only update the internal representation of the output while fixing the model parameters. By alternating between forward and backward propagation, DeLorean can decode the output representation that reflects both the left and right contexts. We demonstrate that our approach is general and applicable to two nonmonotonic reasoning tasks: abductive text generation and counterfactual story revision, where DeLorean outperforms a range of unsupervised and some supervised methods, based on automatic and human evaluation.

cs.CL↗
arXiv2021

NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints

Conditional text generation often requires lexical constraints, i.e., which words should or shouldn't be included in the output text. While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples. We propose NeuroLogic Decoding, a simple yet effective algorithm that enables neural language models -- supervised or not -- to generate fluent text while satisfying complex lexical constraints. Our approach is powerful yet efficient. It handles any set of lexical constraints that is expressible under predicate logic, while its asymptotic runtime is equivalent to conventional beam search. Empirical results on four benchmarks show that NeuroLogic Decoding outperforms previous approaches, including algorithms that handle a subset of our constraints. Moreover, we find that unsupervised models with NeuroLogic Decoding often outperform supervised models with conventional decoding, even when the latter is based on considerably larger networks. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms.

cs.CL↗
arXiv2021

The massless irreducible representation in E theory and how bosons can appear as spinors

We study in detail the irreducible representation of E theory that corresponds to massless particles. This has little algebra Ic(E9) and contains 128 physical states that belong to the spinor representation of SO(16). These are the degrees of freedom of maximal supergravity in eleven dimensions. This smaller number of the degrees of freedom, compared to what might be expected, is due to an infinite number of duality relations which in turn can be traced to the existence of a subaglebra of Ic(E9) which forms an ideal and annihilates the representation. We explain how these features are inherited into the covariant theory. We also comment on the remarkable similarity between how the bosons and fermions arise in E theory.

hep-th↗
arXiv2020

Supersymmetry anomalies and the Wess-Zumino Model in a supergravity background

We briefly recall the procedure for computing the Ward Identities in the presence of a regulator which violates the symmetry being considered. We compute the first non-trivial correction to the supersymmetry Ward Identity of the Wess-Zumino model in the presence of background supergravity using dimensional regularisation. We find that the result can be removed using a finite local counter term and so there is no supersymmetry anomaly.

hep-th↗
arXiv2020

Unsupervised Commonsense Question Answering with Self-Talk

Natural language understanding involves reading between the lines with implicit background knowledge. Current systems either rely on pre-trained language models as the sole implicit source of world knowledge, or resort to external knowledge bases (KBs) to incorporate additional relevant knowledge. We propose an unsupervised framework based on self-talk as a novel alternative to multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning (Bruner, 1961), our approach inquires language models with a number of information seeking questions such as "$\textit{what is the definition of ...}$" to discover additional background knowledge. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines on four out of six commonsense benchmarks, and competes with models that obtain knowledge from external KBs. While our approach improves performance on several benchmarks, the self-talk induced knowledge even when leading to correct answers is not always seen as useful by human judges, raising interesting questions about the inner-workings of pre-trained language models for commonsense reasoning.

cs.CL↗
arXiv2020

E11 and the non-linear dual graviton

The non-linear duality relation between the gravity and dual gravity fields are found in E theory by carrying out $E_{11}$ variations of previously found duality relations. We also find the dual graviton equation of motion up to the addition of some very specific terms whose coefficients are not determined. Using the calculations in this paper this ambiguity was resolved in reference [15] where the full non-linear dual gravity equation was found. As a result the equations of motion in E theory have now been found at the full non-linear level up to, and including, level three, which contains the dual graviton field. When truncated to contain fields at levels three and less, and the spacetime is restricted to be the familiar eleven dimensional space time, the equations are equivalent to those of eleven dimensional supergravity.

hep-th↗
arXiv2020

Kac-Moody algebras and the cosmological constant

We show that the theory of gravity constructed from the non-linear realisation of the semi-direct product of the Kac-Moody algebra A1+++ with its vector representation does not allow a cosmological constant. The existence of a cosmological constant in this theory is related to the breaking of the gravitational duality symmetry.

hep-th↗
arXiv2020

Gravity, Dual Gravity and A1+++

We construct the non-linear realisation of the semi-direct product of the very extended algebra A1+++ and its vector representation. This theory has an infinite number of fields that depend on a spacetime with an infinite number of coordinates. Discarding all except the lowest level field and coordinates the dynamics is just Einstein's equation for the graviton field. We show that the gravity field is related to the dual graviton field by a duality relation and we also derive the equation of motion for the dual gravity field.

hep-th↗
arXiv2019

BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle

The principle of the Information Bottleneck (Tishby et al. 1999) is to produce a summary of information X optimized to predict some other relevant information Y. In this paper, we propose a novel approach to unsupervised sentence summarization by mapping the Information Bottleneck principle to a conditional language modelling objective: given a sentence, our approach seeks a compressed sentence that can best predict the next sentence. Our iterative algorithm under the Information Bottleneck objective searches gradually shorter subsequences of the given sentence while maximizing the probability of the next sentence conditioned on the summary. Using only pretrained language models with no direct supervision, our approach can efficiently perform extractive sentence summarization over a large corpus. Building on our unsupervised extractive summarization (BottleSumEx), we then present a new approach to self-supervised abstractive summarization (BottleSumSelf), where a transformer-based language model is trained on the output summaries of our unsupervised method. Empirical results demonstrate that our extractive method outperforms other unsupervised models on multiple automatic metrics. In addition, we find that our self-supervised abstractive model outperforms unsupervised baselines (including our own) by human evaluation along multiple attributes.

cs.CL↗
arXiv2019

Irreducible representations of E theory

We construct the E theory analogue of the particles that transform under the Poincare group, that is, the irreducible representations of the semi-direct product of the Cartan involution subalgebra of E11 with its vector representation. We show that one such irreducible representation has only the degrees of freedom of eleven dimensional supergravity. This representation is most easily discussed in the light cone formalism and we show that the duality relations found in E theory take a particularly simple form in this formalism. We explain that the mysterious symmetries found recently in the light cone formulation of maximal supergravity theories are part of E11. We also argue that our familiar spacetimes have to be extended by additional coordinates when considering extended objects such as branes.

hep-th↗