arXiv subjects

Shashank Srivastava

Publications and source records attributed to Shashank Srivastava.

At least 55 records · Page 3Linked to original sources

arXiv2023

Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture

Real-world domain experts (e.g., doctors) rarely annotate only a decision label in their day-to-day workflow without providing explanations. Yet, existing low-resource learning techniques, such as Active Learning (AL), that aim to support human annotators mostly focus on the label while neglecting the natural language explanation of a data point. This work proposes a novel AL architecture to support experts' real-world need for label and explanation annotations in low-resource scenarios. Our AL architecture leverages an explanation-generation model to produce explanations guided by human explanations, a prediction model that utilizes generated explanations toward prediction faithfully, and a novel data diversity-based AL sampling strategy that benefits from the explanation annotations. Automated and human evaluations demonstrate the effectiveness of incorporating explanations into AL sampling and the improved human annotation efficiency and trustworthiness with our AL architecture. Additional ablation studies illustrate the potential of our AL architecture for transfer learning, generalizability, and integration with large language models (LLMs). While LLMs exhibit exceptional explanation-generation capabilities for relatively simple tasks, their effectiveness in complex real-world tasks warrants further in-depth study.

cs.CL↗

arXiv2023

MaNtLE: Model-agnostic Natural Language Explainer

Understanding the internal reasoning behind the predictions of machine learning systems is increasingly vital, given their rising adoption and acceptance. While previous approaches, such as LIME, generate algorithmic explanations by attributing importance to input features for individual examples, recent research indicates that practitioners prefer examining language explanations that explain sub-groups of examples. In this paper, we introduce MaNtLE, a model-agnostic natural language explainer that analyzes multiple classifier predictions and generates faithful natural language explanations of classifier rationale for structured classification tasks. MaNtLE uses multi-task training on thousands of synthetic classification tasks to generate faithful explanations. Simulated user studies indicate that, on average, MaNtLE-generated explanations are at least 11% more faithful compared to LIME and Anchors explanations across three tasks. Human evaluations demonstrate that users can better predict model behavior using explanations from MaNtLE compared to other techniques

cs.CL↗

arXiv2022

Identifying and Manipulating the Personality Traits of Language Models

Psychology research has long explored aspects of human personality such as extroversion, agreeableness and emotional stability. Categorizations like the `Big Five' personality traits are commonly used to assess and diagnose personality types. In this work, we explore the question of whether the perceived personality in language models is exhibited consistently in their language generation. For example, is a language model such as GPT2 likely to respond in a consistent way if asked to go out to a party? We also investigate whether such personality traits can be controlled. We show that when provided different types of contexts (such as personality descriptions, or answers to diagnostic questions about personality traits), language models such as BERT and GPT2 can consistently identify and reflect personality markers in those contexts. This behavior illustrates an ability to be manipulated in a highly predictable way, and frames them as tools for identifying personality traits and controlling personas in applications such as dialog systems. We also contribute a crowd-sourced data-set of personality descriptions of human subjects paired with their `Big Five' personality assessment data, and a data-set of personality descriptions collated from Reddit.

cs.AI↗

arXiv2022

LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning

A hallmark of human intelligence is the ability to learn new concepts purely from language. Several recent approaches have explored training machine learning models via natural language supervision. However, these approaches fall short in leveraging linguistic quantifiers (such as 'always' or 'rarely') and mimicking humans in compositionally learning complex tasks. Here, we present LaSQuE, a method that can learn zero-shot classifiers from language explanations by using three new strategies - (1) modeling the semantics of linguistic quantifiers in explanations (including exploiting ordinal strength relationships, such as 'always' > 'likely'), (2) aggregating information from multiple explanations using an attention-based mechanism, and (3) model training via curriculum learning. With these strategies, LaSQuE outperforms prior work, showing an absolute gain of up to 7% in generalizing to unseen real-world classification tasks.

cs.CL↗

arXiv2022

What do Large Language Models Learn beyond Language?

Large language models (LMs) have rapidly become a mainstay in Natural Language Processing. These models are known to acquire rich linguistic knowledge from training on large amounts of text. In this paper, we investigate if pre-training on text also confers these models with helpful `inductive biases' for non-linguistic reasoning. On a set of 19 diverse non-linguistic tasks involving quantitative computations, recognizing regular expressions and reasoning over strings. We find that pretrained models significantly outperform comparable non-pretrained neural models. This remains true also in experiments with training non-pretrained models with fewer parameters to account for model regularization effects. We further explore the effect of text domain on LMs by pretraining models from text from different domains and provenances. Our experiments surprisingly reveal that the positive effects of pre-training persist even when pretraining on multi-lingual text or computer code, and even for text generated from synthetic languages. Our findings suggest a hitherto unexplored deep connection between pre-training and inductive learning abilities of language models.

cs.CL↗

arXiv2022

ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding

While large language models have shown exciting progress on several NLP benchmarks, evaluating their ability for complex analogical reasoning remains under-explored. Here, we introduce a high-quality crowdsourced dataset of narratives for employing proverbs in context as a benchmark for abstract language understanding. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and contains minimal lexical overlaps between narratives and proverbs, ensuring that models need to go beyond surface-level reasoning to succeed. We explore three tasks: (1) proverb recommendation and alignment prediction, (2) narrative generation for a given proverb and topic, and (3) identifying narratives with similar motifs. Our experiments show that neural language models struggle on these tasks compared to humans, and these tasks pose multiple learning challenges.

cs.CL↗

arXiv2022

CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations

Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, humans have the ability to learn new concepts from language. Here, we explore training zero-shot classifiers for structured data purely from language. For this, we introduce CLUES, a benchmark for Classifier Learning Using natural language ExplanationS, consisting of a range of classification tasks over structured data along with natural language supervision in the form of explanations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. To model the influence of explanations in classifying an example, we develop ExEnt, an entailment-based model that learns classifiers using explanations. ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations. We delineate key challenges for automated learning from explanations, addressing which can lead to progress on CLUES in the future. Code and datasets are available at: https://clues-benchmark.github.io.

cs.CL↗

arXiv2021

Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning

Mapping natural language instructions to programs that computers can process is a fundamental challenge. Existing approaches focus on likelihood-based training or using reinforcement learning to fine-tune models based on a single reward. In this paper, we pose program generation from language as Inverse Reinforcement Learning. We introduce several interpretable reward components and jointly learn (1) a reward function that linearly combines them, and (2) a policy for program generation. Fine-tuning with our approach achieves significantly better performance than competitive methods using Reinforcement Learning (RL). On the VirtualHome framework, we get improvements of up to 9.0% on the Longest Common Subsequence metric and 14.7% on recall-based metrics over previous work on this framework (Puig et al., 2018). The approach is data-efficient, showing larger gains in performance in the low-data regime. Generated programs are also preferred by human evaluators over an RL-based approach, and rated higher on relevance, completeness, and human-likeness.

cs.CL↗

arXiv2021

Improving and Simplifying Pattern Exploiting Training

Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few-shot learning without any unlabeled data and introduce ADAPET, which modifies PET's objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data. Our code can be found at https://github.com/rrmenon10/ADAPET.

cs.CL↗

arXiv2021

Adversarial Scrubbing of Demographic Information for Text Classification

Contextual representations learned by language models can often encode undesirable attributes, like demographic associations of the users, while being trained for an unrelated target task. We aim to scrub such undesirable attributes and learn fair representations while maintaining performance on the target task. In this paper, we present an adversarial learning framework "Adversarial Scrubber" (ADS), to debias contextual representations. We perform theoretical analysis to show that our framework converges without leaking demographic information under certain conditions. We extend previous evaluation techniques by evaluating debiasing performance using Minimum Description Length (MDL) probing. Experimental evaluations on 8 datasets show that ADS generates representations with minimal information about demographic attributes while being maximally informative about the target task.

cs.CL↗

arXiv2020

List Decoding of Direct Sum Codes

We consider families of codes obtained by "lifting" a base code $\mathcal{C}$ through operations such as $k$-XOR applied to "local views" of codewords of $\mathcal{C}$, according to a suitable $k$-uniform hypergraph. The $k$-XOR operation yields the direct sum encoding used in works of [Ta-Shma, STOC 2017] and [Dinur and Kaufman, FOCS 2017]. We give a general framework for list decoding such lifted codes, as long as the base code admits a unique decoding algorithm, and the hypergraph used for lifting satisfies certain expansion properties. We show that these properties are satisfied by the collection of length $k$ walks on an expander graph, and by hypergraphs corresponding to high-dimensional expanders. Instantiating our framework, we obtain list decoding algorithms for direct sum liftings on the above hypergraph families. Using known connections between direct sum and direct product, we also recover the recent results of Dinur et al. [SODA 2019] on list decoding for direct product liftings. Our framework relies on relaxations given by the Sum-of-Squares (SOS) SDP hierarchy for solving various constraint satisfaction problems (CSPs). We view the problem of recovering the closest codeword to a given word, as finding the optimal solution of a CSP. Constraints in the instance correspond to edges of the lifting hypergraph, and the solutions are restricted to lie in the base code $\mathcal{C}$. We show that recent algorithms for (approximately) solving CSPs on certain expanding hypergraphs also yield a decoding algorithm for such lifted codes. We extend the framework to list decoding, by requiring the SOS solution to minimize a convex proxy for negative entropy. We show that this ensures a covering property for the SOS solution, and the "condition and round" approach used in several SOS algorithms can then be used to recover the required list of codewords.

cs.DS↗

arXiv2020

Unique Decoding of Explicit $ε$-balanced Codes Near the Gilbert-Varshamov Bound

The Gilbert-Varshamov bound (non-constructively) establishes the existence of binary codes of distance $1/2 -ε$ and rate $Ω(ε^2)$ (where an upper bound of $O(ε^2\log(1/ε))$ is known). Ta-Shma [STOC 2017] gave an explicit construction of $ε$-balanced binary codes, where any two distinct codewords are at a distance between $1/2 -ε/2$ and $1/2+ε/2$, achieving a near optimal rate of $Ω(ε^{2+β})$, where $β\to 0$ as $ε\to 0$. We develop unique and list decoding algorithms for (essentially) the family of codes constructed by Ta-Shma. We prove the following results for $ε$-balanced codes with block length $N$ and rate $Ω(ε^{2+β})$ in this family: - For all $ε, β> 0$ there are explicit codes which can be uniquely decoded up to an error of half the minimum distance in time $N^{O_{ε, β}(1)}$. - For any fixed constant $β$ independent of $ε$, there is an explicit construction of codes which can be uniquely decoded up to an error of half the minimum distance in time $(\log(1/ε))^{O(1)} \cdot N^{O_β(1)}$. - For any $ε> 0$, there are explicit $ε$-balanced codes with rate $Ω(ε^{2+β})$ which can be list decoded up to error $1/2 - ε'$ in time $N^{O_{ε,ε',β}(1)}$, where $ε', β\to 0$ as $ε\to 0$. The starting point of our algorithms is the list decoding framework from Alev et al. [SODA 2020], which uses the Sum-of-Squares SDP hierarchy. The rates obtained there were quasipolynomial in $ε$. Here, we show how to overcome the far from optimal rates of this framework obtaining unique decoding algorithms for explicit binary codes of near optimal rate. These codes are based on simple modifications of Ta-Shma's construction.

cs.DS↗

arXiv2020

PRover: Proof Generation for Interpretable Reasoning over Rules

Recent work by Clark et al. (2020) shows that transformers can act as 'soft theorem provers' by answering questions over explicitly provided knowledge in natural language. In our work, we take a step closer to emulating formal theorem provers, by proposing PROVER, an interpretable transformer-based model that jointly answers binary questions over rule-bases and generates the corresponding proofs. Our model learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm. During inference, a valid proof, satisfying a set of global constraints is generated. We conduct experiments on synthetic, hand-authored, and human-paraphrased rule-bases to show promising results for QA and proof generation, with strong generalization performance. First, PROVER generates proofs with an accuracy of 87%, while retaining or improving performance on the QA task, compared to RuleTakers (up to 6% improvement on zero-shot evaluation). Second, when trained on questions requiring lower depths of reasoning, it generalizes significantly better to higher depths (up to 15% improvement). Third, PROVER obtains near perfect QA accuracy of 98% using only 40% of the training data. However, generating proofs for questions requiring higher depths of reasoning becomes challenging, and the accuracy drops to 65% for 'depth 5', indicating significant scope for future work. Our code and models are publicly available at https://github.com/swarnaHub/PRover

cs.CL↗

arXiv2020

Modular and fractional L-intersecting families of vector spaces

In the first part of this paper, we prove a theorem which is the $q$-analogue of a generalized modular Ray-Chaudhuri-Wilson Theorem shown in [Alon, Babai, Suzuki, J. Combin. Theory Series A, 1991]. It is also a generalization of the main theorem in [Frankl and Graham, European J. Combin. 1985] under certain circumstances. In the second part of this paper, we prove $q$-analogues of results on a recent notion called \emph{fractional $L$-intersecting family} for families of subspaces of a given vector space. We use the above theorem to obtain a general upper bound to the cardinality of such families. We give an improvement to this general upper bound in certain special cases.

math.CO↗

arXiv2019

Models for Predicting Transonic Flutter of a Wing-Section with Sloshing in an Embedded Fuel Tank

The present study focuses on the development, application, and comparison of three computational frameworks of varying fidelities for assessing the effects of fuel sloshing in internal fuel tanks on the aeroelastic characteristics of a wing section. The first approach uses the coupling of compressible flow solver for external aerodynamics integrated with structural solver and incompressible multiphase flow solver for fuel sloshing in the embedded fuel tank As time-domain flutter solution of these coupled solvers is computationally expensive, two approximate surrogate models to emulate sloshing flows are considered. One surrogate model utilizes a linearised approach for sloshing load computations by creating an Equivalent Mechanical System (EMS) with its parameters derived from potential flow theory. The other surrogate model aims to efficiently describe the dominant dynamic characteristics of the underlying system by employing the Radial Basis Function Neural Networks (RBF-NN) using limited CFD-based data to calibrate this model. The flutter boundaries of a wing section with and without the effects of fuel sloshing are compared. The limitation of the EMS surrogate to represent nonlinearities are reflected in this study. The RBF-NN surrogate shows remarkable agreement with the high-fidelity solution for sloshing with significantly low computational cost, thereby motivating extension to three-dimensional problems.

physics.flu-dyn↗

arXiv2019

Machine Learning Surrogates for Predicting Response of an Aero-Structural-Sloshing System

This study demonstrates the feasibility of developing machine learning (ML) surrogates based on Recurrent Neural Networks (RNN) for predicting the unsteady aeroelastic response of transonic pitching and plunging wing-fuel tank sloshing system by considering an approximate simplified model of an airfoil in transonic flow and sloshing loads from a partially filled fuel tank rigidly embedded inside the airfoil and undergoing a free unsteady motion. The ML surrogates are then used to predict the aeroelastic response of the coupled system. The external aerodynamic loads on the airfoil and the two-phase sloshing loads data for training the RNN are generated using open-source computational fluid dynamics (CFD) codes. The aerodynamic force and moment coefficients are predicted from the surrogate model based on its motion history. Similarly, the lateral and vertical forces and moments from fuel sloshing in the fuel tank are predicted using the surrogate model resulting from the motion of the embedded fuel tank. Comparing the free motion of the airfoil without sloshing tank to the free motion of the airfoil with a partially-filled fuel tank shows that the effects of sloshing on the aeroelastic motion of the aero-structural system. The effectiveness of the predictions from RNN are then assessed by comparing with the results from the high-fidelity coupled aero-structural-fuel tank sloshing simulations. It is demonstrated that the surrogate models can accurately and economically predict the coupled aero-structural motion.

cs.CE↗

arXiv2019

Fractional cross intersecting families

Let $\mathcal{A}=\{A_{1},...,A_{p}\}$ and $\mathcal{B}=\{B_{1},...,B_{q}\}$ be two families of subsets of $[n]$ such that for every $i\in [p]$ and $j\in [q]$, $|A_{i}\cap B_{j}|= \frac{c}{d}|B_{j}|$, where $\frac{c}{d}\in [0,1]$ is an irreducible fraction. We call such families "$\frac{c}{d}$-cross intersecting families". In this paper, we find a tight upper bound for the product $|\mathcal{A}||\mathcal{B}|$ and characterize the cases when this bound is achieved for $\frac{c}{d}=\frac{1}{2}$. Also, we find a tight upper bound on $|\mathcal{A}||\mathcal{B}|$ when $\mathcal{B}$ is $k$-uniform and characterize, for all $\frac{c}{d}$, the cases when this bound is achieved.

math.CO↗

arXiv2015

Inferring Interpersonal Relations in Narrative Summaries

Characterizing relationships between people is fundamental for the understanding of narratives. In this work, we address the problem of inferring the polarity of relationships between people in narrative summaries. We formulate the problem as a joint structured prediction for each narrative, and present a model that combines evidence from linguistic and semantic features, as well as features based on the structure of the social community in the text. We also provide a clustering-based approach that can exploit regularities in narrative types. e.g., learn an affinity for love-triangles in romantic stories. On a dataset of movie summaries from Wikipedia, our structured models provide more than a 30% error-reduction over a competitive baseline that considers pairs of characters in isolation.

cs.CL↗