arXiv subjects

Arindam Mitra

Publications and source records attributed to Arindam Mitra.

At least 37 records · Page 2Linked to original sources

arXiv2023

Teaching Language Models to Hallucinate Less with Synthetic Tasks

Large language models (LLMs) frequently hallucinate on abstractive summarization tasks such as document-based question-answering, meeting summarization, and clinical report generation, even though all necessary information is included in context. However, optimizing LLMs to hallucinate less on these tasks is challenging, as hallucination is hard to efficiently evaluate at each optimization step. In this work, we show that reducing hallucination on a synthetic task can also reduce hallucination on real-world downstream tasks. Our method, SynTra, first designs a synthetic task where hallucinations are easy to elicit and measure. It next optimizes the LLM's system message via prefix-tuning on the synthetic task, and finally transfers the system message to realistic, hard-to-optimize tasks. Across three realistic abstractive summarization tasks, SynTra reduces hallucination for two 13B-parameter LLMs using only a synthetic retrieval task for supervision. We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively increase hallucination. Overall, SynTra demonstrates that the extra flexibility of working with synthetic data can help mitigate undesired behaviors in practice.

cs.CL↗

arXiv2023

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

Recent research has focused on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundation models (LFMs). A number of issues impact the quality of these models, ranging from limited imitation signals from shallow LFM outputs; small scale homogeneous training data; and most notably a lack of rigorous evaluation resulting in overestimating the small model's capability as they tend to learn to imitate the style, but not the reasoning process of LFMs. To address these challenges, we develop Orca (We are working with our legal team to publicly release a diff of the model weights in accordance with LLaMA's release policy to be published at https://aka.ms/orca-lm), a 13-billion parameter model that learns to imitate the reasoning process of LFMs. Orca learns from rich signals from GPT-4 including explanation traces; step-by-step thought processes; and other complex instructions, guided by teacher assistance from ChatGPT. To promote this progressive learning, we tap into large-scale and diverse imitation data with judicious sampling and selection. Orca surpasses conventional state-of-the-art instruction-tuned models such as Vicuna-13B by more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard (BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH benchmark and shows competitive performance (4 pts gap with optimized system message) in professional and academic examinations like the SAT, LSAT, GRE, and GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our research indicates that learning from step-by-step explanations, whether these are generated by humans or more advanced AI models, is a promising direction to improve model capabilities and skills.

cs.CL↗

arXiv2023

Instruction Tuned Models are Quick Learners

Instruction tuning of language models has demonstrated the ability to enhance model generalization to unseen tasks via in-context learning using a few examples. However, typical supervised learning still requires a plethora of downstream training data for finetuning. Often in real-world situations, there is a scarcity of data available for finetuning, falling somewhere between few shot inference and fully supervised finetuning. In this work, we demonstrate the sample efficiency of instruction tuned models over various tasks by estimating the minimal downstream training data required by them to perform transfer learning and match the performance of state-of-the-art (SOTA) supervised models. We conduct experiments on 119 tasks from Super Natural Instructions (SuperNI) in both the single task learning (STL) and multi task learning (MTL) settings. Our findings reveal that, in the STL setting, instruction tuned models equipped with 25% of the downstream train data surpass the SOTA performance on the downstream tasks. In the MTL setting, an instruction tuned model trained on only 6% of downstream training data achieve SOTA, while using 100% of the training data results in a 3.69% points improvement (ROUGE-L 74.68) over the previous SOTA. We conduct an analysis on T5 vs Tk-Instruct by developing several baselines to demonstrate that instruction tuning aids in increasing both sample efficiency and transfer learning. Additionally, we observe a consistent ~4% performance increase in both settings when pre-finetuning is performed with instructions. Finally, we conduct a categorical study and find that contrary to previous results, tasks in the question rewriting and title generation categories suffer from instruction tuning.

cs.CL↗

arXiv2023

Improvement in quantum communication using quantum switch

Applications of the quantum switch on quantum channels have recently become a topic of intense discussion. In the present work, we show that some useless (for communication) channels may provide useful communication under the action of quantum switch for several information-theoretic tasks: quantum random access codes, quantum steering, etc. We demonstrate that the quantum switch can also be useful in preventing the loss of coherence in a system when only coherence-breaking channels are the available channels for communication. We also show that if a useless quantum channel does not provide useful communication even after using a quantum switch, concatenating the channel with another suitable quantum channel, and subsequently using the switch, one may achieve useful communication. Finally, we discuss how the introduction of noise in the quantum switch can reduce the advantage that the switch provides.

quant-ph↗

arXiv2022

Characterizing and quantifying the incompatibility of quantum instruments

Incompatibility of quantum devices is one of the cornerstones of quantum theory, and the incompatibility of quantum measurements and channels has been linked to quantum advantage in certain information theoretic tasks. In this work, we focus on the less well-explored question of the incompatibility of quantum instruments, that is, devices that describe the measurement process in its entirety, accounting for both the classical measurement outcome and the quantum post-measurement state. In particular, we focus on the recently introduced notion of parallel compatibility of instruments, which has been argued to be a natural notion of instrument compatibility. We introduce -- similarly to the case of measurements and channels -- the incompatibility robustness of quantum instruments and derive universal bounds on it. We then prove that post-processing of quantum instruments is a free operation for parallel compatibility. Last, we provide families of instruments for which our bounds are tight, and families of compatible indecomposable instruments.

quant-ph↗

arXiv2022

NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks

Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.

cs.CL↗

arXiv2022

On the compatibility of quantum instruments

Incompatibility of quantum devices is a useful resource in various quantum information theoretical tasks, and it is at the heart of some fundamental features of quantum theory. While the incompatibility of measurements and quantum channels is well-studied, the incompatibility of quantum instruments has not been explored in much detail. In this work, we revise a notion of instrument compatibility introduced in the literature that we call traditional compatibility. Then, we introduce the new notion of parallel compatibility, and show that these two notions are inequivalent. Then, we argue that the notion of traditional compatibility is incomplete, and prove that while parallel compatibility captures measurement and channel compatibility, traditional compatibility does not. Hence, we propose parallel compatibility as the conceptually complete definition of compatibility of quantum instruments.

quant-ph↗

arXiv2022

Improving Biomedical Information Retrieval with Neural Retrievers

Information retrieval (IR) is essential in search engines and dialogue systems as well as natural language processing tasks such as open-domain question answering. IR serve an important function in the biomedical domain, where content and sources of scientific knowledge may evolve rapidly. Although neural retrievers have surpassed traditional IR approaches such as TF-IDF and BM25 in standard open-domain question answering tasks, they are still found lacking in the biomedical domain. In this paper, we seek to improve information retrieval (IR) using neural retrievers (NR) in the biomedical domain, and achieve this goal using a three-pronged approach. First, to tackle the relative lack of data in the biomedical domain, we propose a template-based question generation method that can be leveraged to train neural retriever models. Second, we develop two novel pre-training tasks that are closely aligned to the downstream task of information retrieval. Third, we introduce the ``Poly-DPR'' model which encodes each context into multiple context vectors. Extensive experiments and analysis on the BioASQ challenge suggest that our proposed method leads to large gains over existing neural approaches and beats BM25 in the small-corpus setting. We show that BM25 and our method can complement each other, and a simple hybrid model leads to further gains in the large corpus setting.

cs.IR↗

arXiv2022

Quantifying unsharpness of observables in an outcome-independent way

Recently, a very beautiful measure of the unsharpness (fuzziness) of the observables is discussed in the paper [Phys. Rev. A 104, 052227 (2021)]. The measure which is defined in this paper is constructed via uncertainty and does not depend on the values of the outcomes. There exist several properties of a set of observables (e.g., incompatibility, non-disturbance) that do not depend on the values of the outcomes. Therefore, the approach in the above-said paper is consistent with the above-mentioned fact and is able to measure the intrinsic unsharpness of the observables. In this work, we also quantify the unsharpness of observables in an outcome-independent way. But our approach is different than the approach of the above-said paper. In this work, at first, we construct two Luder's instrument-based unsharpness measures and provide the tight upper bounds of those measures. Then we prove the monotonicity of the above-said measures under a class of fuzzifying processes (processes that make the observables more fuzzy). This is consistent with the resource-theoretic framework. Then we relate our approach to the approach of the above-said paper. Next, we try to construct two instrument-independent unsharpness measures. In particular, we define two instrument-independent unsharpness measures and provide the tight upper bounds of those measures and then we derive the condition for the monotonicity of those measures under a class of fuzzifying processes and prove the monotonicity for dichotomic qubit observables. Then we show that for an unknown measurement, the values of all of these measures can be determined experimentally. Finally, we present the idea of the resource theory of the sharpness of the observables.

quant-ph↗

arXiv2021

Characterizing incompatibility of quantum measurements via their Naimark extensions

We obtain a formal characterization of the compatibility or otherwise of a set of positive-operator-valued measures (POVMs) via their Naimark extensions. We show that a set of POVMs is jointly measurable if and only if there exists a single Naimark extension, specified by a fixed ancilla state on the same ancilla Hilbert space, that maps them to a set of commuting projective measurements (PVMs). We use our result to obtain an easily checkable sufficient condition for the compatibility of a pair of dichotomic observables in any dimension. This in turn leads to a characterization of the compatibility regions for some important classes of observables including a pair of unsharp qubit observables. Finally, we also outline as to how our result provides an alternate approach to quantifying the incompatibility of a general set of quantum measurements.

quant-ph↗

arXiv2021

Information leak and incompatibility of physical context: A modified approach

A beautiful idea about the incompatibility of Physical Context(IPC) was introduced in Phys. Rev. A 102, 050201(R) (2020). Here, a context is defined as a set of a quantum state and two sharp rank-one measurements, and the incompatibility of physical context is defined as the leakage of information while implementing those two measurements successively in that quantum state. In this work, we show the limitations in their approach. The three primary limitations are that, (i) their approach is not positive operator-valued measurements and (ii), they restrict information-theoretic agents Alice, Eve and Bob to specific quantum operations and do not consider most general quantum operations i.e., quantum instruments and (iii), their measure of IPC can take negative values in specific cases in a more general scenario which implies the limitation of their information measure. Thereby, we have introduced a generalization and modification to their approach in more general and convenient way, such that this idea is well-defined for generic measurements, without these limitations. We also present a comparison of the measure of the IPC through their and our method. Lastly, we show, how the IPC reduces in the presence of memory using our modification, which further validates our approach.

quant-ph↗

arXiv2021

Layers of classicality in the compatibility of measurements

The term "Layers of classicality" in the context of quantum measurements, was introduced in [T. Heinosaari, Phys. Rev. A 93, 042118 (2016)]. The strongest layer among these consists of the sets of observables that can be broadcast and the weakest layer consists of the sets of compatible observables. There are several other layers in between those two layers. In this work, we study their physical and geometric properties and show the differences and similarities among the layers in these properties. In particular we show that: (i) none of the layers of classicality respect transitivity property, (ii) the concept like degree of broadcasting similar to degree of compatibility does not exist, (iii) there exist informationally incomplete POVMs that are not individually broadcastable, (iv) a set of broadcasting channels can be obtained through concatenation of broadcasting and non-disturbing channels, (v) unlike compatibility, other layers of classicality are not convex, in general. Finally, we discuss the relations among these layers. More specifically, we show that specific type of concatenation relations among broadcasting channels decide the layer in which a pair of observables resides.

quant-ph↗

arXiv2021

On optimal cloning and incompatibility

We investigate the role of symmetric quantum cloning machines (QCMs) in quantifying the mutual incompatibility of quantum observables. Specifically, we identify a cloning-based incompatibility measure whereby the incompatibility of a set of observables maybe quantified in terms of how well a uniform ensemble of their eigenstates can be cloned via a symmetric QCM. We show that this new incompatibility measure $\mathcal{Q}_{c}$ is {\it faithful} since it vanishes only for commuting observables. We prove an upper bound for $\mathcal{Q}_{c}$ for any set of observables in a finite-dimensional system and show that the upper bound is attained if and only if the observables are mutually unbiased. Finally, we use our formalism to obtain the optimal quantum cloner for a pair of qubit observables. Our work marks an important step in formalising the connection between two fundamental concepts in quantum information theory, namely, the no-cloning principle and the existence of incompatible observables in quantum theory.

quant-ph↗

arXiv2020

Can Transformers Reason About Effects of Actions?

A recent work has shown that transformers are able to "reason" with facts and rules in a limited setting where the rules are natural language expressions of conjunctions of conditions implying a conclusion. Since this suggests that transformers may be used for reasoning with knowledge given in natural language, we do a rigorous evaluation of this with respect to a common form of knowledge and its corresponding reasoning -- the reasoning about effects of actions. Reasoning about action and change has been a top focus in the knowledge representation subfield of AI from the early days of AI and more recently it has been a highlight aspect in common sense question answering. We consider four action domains (Blocks World, Logistics, Dock-Worker-Robots and a Generic Domain) in natural language and create QA datasets that involve reasoning about the effects of actions in these domains. We investigate the ability of transformers to (a) learn to reason in these domains and (b) transfer that learning from the generic domains to the other domains.

cs.CL↗

arXiv2020

Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks

Numerical reasoning is often important to accurately understand the world. Recently, several format-specific datasets have been proposed, such as numerical reasoning in the settings of Natural Language Inference (NLI), Reading Comprehension (RC), and Question Answering (QA). Several format-specific models and architectures in response to those datasets have also been proposed. However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain. In pursuit of this goal, we introduce NUMBERGAME, a multifaceted benchmark to evaluate model performance across numerical reasoning tasks of eight diverse formats. We add four existing question types in our compilation. Two of the new types we add are about questions that require external numerical knowledge, commonsense knowledge and domain knowledge. For building a more practical numerical reasoning system, NUMBERGAME demands four capabilities beyond numerical reasoning: (i) detecting question format directly from data (ii) finding intermediate common format to which every format can be converted (iii) incorporating commonsense knowledge (iv) handling data imbalance across formats. We build several baselines, including a new model based on knowledge hunting using a cheatsheet. However, all baselines perform poorly in contrast to the human baselines, indicating the hardness of our benchmark. Our work takes forward the recent progress in generic system development, demonstrating the scope of these under-explored tasks.

cs.CL↗

arXiv2020

How Additional Knowledge can Improve Natural Language Commonsense Question Answering?

Recently several datasets have been proposed to encourage research in Question Answering domains where commonsense knowledge is expected to play an important role. Recent language models such as ROBERTA, BERT and GPT that have been pre-trained on Wikipedia articles and books have shown reasonable performance with little fine-tuning on several such Multiple Choice Question-Answering (MCQ) datasets. Our goal in this work is to develop methods to incorporate additional (commonsense) knowledge into language model-based approaches for better question-answering in such domains. In this work, we first categorize external knowledge sources, and show performance does improve on using such sources. We then explore three different strategies for knowledge incorporation and four different models for question-answering using external commonsense knowledge. We analyze our predictions to explore the scope of further improvements.

cs.CL↗

arXiv2020

Natural Language QA Approaches using Reasoning with External Knowledge

Question answering (QA) in natural language (NL) has been an important aspect of AI from its early days. Winograd's ``councilmen'' example in his 1972 paper and McCarthy's Mr. Hug example of 1976 highlights the role of external knowledge in NL understanding. While Machine Learning has been the go-to approach in NL processing as well as NL question answering (NLQA) for the last 30 years, recently there has been an increasingly emphasized thread on NLQA where external knowledge plays an important role. The challenges inspired by Winograd's councilmen example, and recent developments such as the Rebooting AI book, various NLQA datasets, research on knowledge acquisition in the NLQA context, and their use in various NLQA models have brought the issue of NLQA using ``reasoning'' with external knowledge to the forefront. In this paper, we present a survey of the recent work on them. We believe our survey will help establish a bridge between multiple fields of AI, especially between (a) the traditional fields of knowledge representation and reasoning and (b) the field of NL understanding and NLQA.

cs.CL↗

arXiv2019

A Generate-Validate Approach to Answering Questions about Qualitative Relationships

Qualitative relationships describe how increasing or decreasing one property (e.g. altitude) affects another (e.g. temperature). They are an important aspect of natural language question answering and are crucial for building chatbots or voice agents where one may enquire about qualitative relationships. Recently a dataset about question answering involving qualitative relationships has been proposed, and a few approaches to answer such questions have been explored, in the heart of which lies a semantic parser that converts the natural language input to a suitable logical form. A problem with existing semantic parsers is that they try to directly convert the input sentences to a logical form. Since the output language varies with each application, it forces the semantic parser to learn almost everything from scratch. In this paper, we show that instead of using a semantic parser to produce the logical form, if we apply the generate-validate framework i.e. generate a natural language description of the logical form and validate if the natural language description is followed from the input text, we get a better scope for transfer learning and our method outperforms the state-of-the-art by a large margin of 7.93%.

cs.CL↗