arXiv subjects

Saleh Soltan

Publications and source records attributed to Saleh Soltan.

15 recordsLinked to original sources

arXiv2024

GeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning

The emergence of Large Language Models (LLMs) with capabilities like In-Context Learning (ICL) has ushered in new possibilities for data generation across various domains while minimizing the need for extensive data collection and modeling techniques. Researchers have explored ways to use this generated synthetic data to optimize smaller student models for reduced deployment costs and lower latency in downstream tasks. However, ICL-generated data often suffers from low quality as the task specificity is limited with few examples used in ICL. In this paper, we propose GeMQuAD - a semi-supervised learning approach, extending the WeakDAP framework, applied to a dataset generated through ICL with just one example in the target language using AlexaTM 20B Seq2Seq LLM. Through our approach, we iteratively identify high-quality data to enhance model performance, especially for low-resource multilingual setting in the context of Extractive Question Answering task. Our framework outperforms the machine translation-augmented model by 0.22/1.68 F1/EM (Exact Match) points for Hindi and 0.82/1.37 F1/EM points for Spanish on the MLQA dataset, and it surpasses the performance of model trained on an English-only dataset by 5.05/6.50 F1/EM points for Hindi and 3.81/3.69 points F1/EM for Spanish on the same dataset. Notably, our approach uses a pre-trained LLM for generation with no fine-tuning (FT), utilizing just a single annotated example in ICL to generate data, providing a cost-effective development process.

cs.CL↗

arXiv2023

Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models

Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have advantages, however training both model types from scratch is computationally expensive. We explore recipes to improve pre-training efficiency by initializing one model from the other. (1) Extracting the encoder from a seq2seq model, we show it under-performs a Masked Language Modeling (MLM) encoder, particularly on sequence labeling tasks. Variations of masking during seq2seq training, reducing the decoder size, and continuing with a small amount of MLM training do not close the gap. (2) Conversely, using an encoder to warm-start seq2seq training, we show that by unfreezing the encoder partway through training, we can match task performance of a from-scratch seq2seq model. Overall, this two-stage approach is an efficient recipe to obtain both a multilingual encoder and a seq2seq model, matching the performance of training each model from scratch while reducing the total compute cost by 27%.

cs.CL↗

arXiv2022

CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing

A bottleneck to developing Semantic Parsing (SP) models is the need for a large volume of human-labeled training data. Given the complexity and cost of human annotation for SP, labeled data is often scarce, particularly in multilingual settings. Large Language Models (LLMs) excel at SP given only a few examples, however LLMs are unsuitable for runtime systems which require low latency. In this work, we propose CLASP, a simple method to improve low-resource SP for moderate-sized models: we generate synthetic data from AlexaTM 20B to augment the training set for a model 40x smaller (500M parameters). We evaluate on two datasets in low-resource settings: English PIZZA, containing either 348 or 16 real examples, and mTOP cross-lingual zero-shot, where training data is available only in English, and the model must generalize to four new languages. On both datasets, we show significant improvements over strong baseline methods.

cs.CL↗

arXiv2022

LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging

We present LINGUIST, a method for generating annotated data for Intent Classification and Slot Tagging (IC+ST), via fine-tuning AlexaTM 5B, a 5-billion-parameter multilingual sequence-to-sequence (seq2seq) model, on a flexible instruction prompt. In a 10-shot novel intent setting for the SNIPS dataset, LINGUIST surpasses state-of-the-art approaches (Back-Translation and Example Extrapolation) by a wide margin, showing absolute improvement for the target intents of +1.9 points on IC Recall and +2.5 points on ST F1 Score. In the zero-shot cross-lingual setting of the mATIS++ dataset, LINGUIST out-performs a strong baseline of Machine Translation with Slot Alignment by +4.14 points absolute on ST F1 Score across 6 languages, while matching performance on IC. Finally, we verify our results on an internal large-scale multilingual dataset for conversational agent IC+ST and show significant improvements over a baseline which uses Back-Translation, Paraphrasing and Slot Catalog Resampling. To our knowledge, we are the first to demonstrate instruction fine-tuning of a large-scale seq2seq model to control the outputs of multilingual intent- and slot-labeled data generation.

cs.CL↗

arXiv2022

AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.

cs.CL↗

arXiv2022

Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems

We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system. Though we train using 70% spoken-form data, our teacher models perform comparably to XLM-R and mT5 when evaluated on the written-form Cross-lingual Natural Language Inference (XNLI) corpus. We perform a second stage of pretraining on our teacher models using in-domain data from our system, improving error rates by 3.86% relative for intent classification and 7.01% relative for slot filling. We find that even a 170M-parameter model distilled from our Stage 2 teacher model has 2.88% better intent classification and 7.69% better slot filling error rates when compared to the 2.3B-parameter teacher trained only on public data (Stage 1), emphasizing the importance of in-domain data for pretraining. When evaluated offline using labeled NLU data, our 17M-parameter Stage 2 distilled model outperforms both XLM-R Base (85M params) and DistillBERT (42M params) by 4.23% to 6.14%, respectively. Finally, we present results from a full virtual assistant experimentation platform, where we find that models trained using our pretraining and distillation pipeline outperform models distilled from 85M-parameter teachers by 3.74%-4.91% on an automatic measurement of full-system user dissatisfaction.

cs.CL↗

arXiv2020

Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding

Semantic parsing is one of the key components of natural language understanding systems. A successful parse transforms an input utterance to an action that is easily understood by the system. Many algorithms have been proposed to solve this problem, from conventional rulebased or statistical slot-filling systems to shiftreduce based neural parsers. For complex parsing tasks, the state-of-the-art method is based on autoregressive sequence to sequence models to generate the parse directly. This model is slow at inference time, generating parses in O(n) decoding steps (n is the length of the target sequence). In addition, we demonstrate that this method performs poorly in zero-shot cross-lingual transfer learning settings. In this paper, we propose a non-autoregressive parser which is based on the insertion transformer to overcome these two issues. Our approach 1) speeds up decoding by 3x while outperforming the autoregressive model and 2) significantly improves cross-lingual transfer in the low-resource setting by 37% compared to autoregressive baseline. We test our approach on three well-known monolingual datasets: ATIS, SNIPS and TOP. For cross lingual semantic parsing, we use the MultiATIS++ and the multilingual TOP datasets.

cs.CL↗

arXiv2020

Distribution of blackouts in the power grid and the Motter and Lai model

Carreras, Dobson and colleagues have studied empirical data on the sizes of the blackouts in real grids and modeled them by computer simulations using the direct current approximation. They have found that the resulting blackout sizes are distributed as a power law and suggested that this is because the grids are driven to the self-organized critical state. In contrast, more recent studies found that the distribution of cascades is bimodal as in a first order phase transition, resulting in either a very small blackout or a very large blackout, engulfing a finite fraction of the system. Here we reconcile the two approaches and investigate how the distribution of the blackouts change with model parameters, including the tolerance criteria and the dynamic rules of failure of the overloaded lines during the cascade. In addition, we study the same problem for the Motter and Lai model and find similar results, suggesting that the physical laws of flow on the network are not as important as network topology, overload conditions, and dynamic rules of failure.

physics.soc-ph↗

arXiv2018

Protecting the Grid against IoT Botnets of High-Wattage Devices

We provide methods to prevent line failures in the power grid caused by a newly revealed MAnipulation of Demand (MAD) attacks via an IoT botnet of high-wattage devices. In particular, we develop two algorithms named Securing Additional margin For generators in Economic dispatch (SAFE) Algorithm and Iteratively MiniMize and boUNd Economic dispatch (IMMUNE) Algorithm for finding robust operating points for generators during the economic dispatch such that no lines are overloaded after automatic primary control response to any MAD attacks. In situations that the operating cost of the grid in a robust state is costly (or no robust operating points exist), we provide efficient methods to verify--in advance--if possible line overloads can be cleared during the secondary control after any MAD attacks. We then define the $\alpha D$-robustness notion for the grids indicating that any line failures can be cleared during the secondary control if an adversary can increase/decrease the demands by $\alpha$ fraction. We demonstrate that practical upper and lower bounds on the maximum $\alpha$ for which the grid is $\alpha D$-robust can be found efficiently in polynomial time. Finally, we evaluate the performance of the developed algorithms and methods on realistic power grid test cases. Our work provides the first methods for protecting the grid against potential line failures caused by MAD attacks.

eess.SY↗

arXiv2017

EXPOSE the Line Failures following a Cyber-Physical Attack on the Power Grid

Recent attacks on power grids demonstrated the vulnerability of the grids to cyber and physical attacks. To analyze this vulnerability, we study cyber-physical attacks that affect both the power grid physical infrastructure and its underlying Supervisory Control And Data Acquisition (SCADA) system. We assume that an adversary attacks an area by: (i) disconnecting some lines within that area, and (ii) obstructing the information (e.g., status of the lines and voltage measurements) from within the area to reach the control center. We leverage the algebraic properties of the AC power flows to introduce the efficient EXPOSE Algorithm for detecting line failures and recovering voltages inside that attacked area after such an attack. The EXPOSE Algorithm outperforms the state-of-the-art algorithm for detecting line failures using partial information under the AC power flow model in terms of scalability and accuracy. The main advantages of the EXPOSE Algorithm are that its running time is independent of the size of the grid and number of line failures, and that it provides accurate information recovery under some conditions on the attacked area. Moreover, it approximately recovers the information and provides the confidence of the solution when these conditions do not hold.

eess.SY↗

arXiv2017

REACT to Cyber Attacks on Power Grids

Motivated by the recent cyber attack on the Ukrainian power grid, we study cyber attacks on power grids that affect both the physical infrastructure and the data at the control center. In particular, we assume that an adversary attacks an area by: (i) remotely disconnecting some lines within the attacked area, and (ii) modifying the information received from the attacked area to mask the line failures and hide the attacked area from the control center. For the latter, we consider two types of attacks: (i) data distortion: which distorts the data by adding powerful noise to the actual data, and (ii) data replay: which replays a locally consistent old data instead of the actual data. We use the DC power flow model and prove that the problem of finding the set of line failures given the phase angles of the nodes outside of the attacked area is strongly NP-hard, even when the attacked area is known. However, we introduce the polynomial time REcurrent Attack Containment and deTection (REACT) Algorithm to approximately detect the attacked area and line failures after a cyber attack. We numerically show that it performs very well in detecting the attacked area, and detecting single, double, and triple line failures in small and large attacked areas.

eess.SY↗

arXiv2016

A study of cascading failures in real and synthetic power grid topologies using DC power flows

Using the linearized DC power flow model, we study cascading failures and their spatial and temporal properties in the US Western Interconnect (USWI) power grid. We also introduce the preferential Degree And Distance Attachment (DADA) model, with similar degree distributions, resistances, and currents to the USWI. We investigate the behavior of both grids resulting from the failure of a single line. We find that the DADA model and the USWI model react very similarly to that failure, and that their blackout characteristics resemble each other. In many cases, the failure of a single line can cause cascading failures, which impact the entire grid. We characterize the resilience of the grid by three parameters, the most important of which is tolerance $α$, which is the ratio of the maximal load a line can carry to its initial load. We characterize a blackout by its yield, which we define as the ratio of the final to the initial consumed currents. We find that if $α\leq2$, the probability of a large blackout occurring is very small. By contrast, in a broad range of $1 < α < 2$, the initial failure of a single line can result, with a high probability, in cascading failures leading to a massive blackout with final yield less than 80%. The yield has a bimodal distribution typical of a first-order transition, i.e., the failure of a randomly selected line leads either to an insignificant current reduction or to a major blackout. We find that there is a latent period in the development of major blackouts during which few lines are overloaded, and the yield remains high. The duration of this latent period is proportional to the tolerance. The existence of the latent period suggests that intervention during early time steps of a cascade can significantly reduce the risk of a major blackout.

physics.soc-ph↗

arXiv2016

Doubly Balanced Connected Graph Partitioning

We introduce and study the Doubly Balanced Connected graph Partitioning (DBCP) problem: Let $G=(V,E)$ be a connected graph with a weight (supply/demand) function $p:V\rightarrow \{-1,+1\}$ satisfying $p(V)=\sum_{j\in V} p(j)=0$. The objective is to partition $G$ into $(V_1,V_2)$ such that $G[V_1]$ and $G[V_2]$ are connected, $|p(V_1)|,|p(V_2)|\leq c_p$, and $\max\{\frac{|V_1|}{|V_2|},\frac{|V_2|}{|V_1|}\}\leq c_s$, for some constants $c_p$ and $c_s$. When $G$ is 2-connected, we show that a solution with $c_p=1$ and $c_s=3$ always exists and can be found in polynomial time. Moreover, when $G$ is 3-connected, we show that there is always a `perfect' solution (a partition with $p(V_1)=p(V_2)=0$ and $|V_1|=|V_2|$, if $|V|\equiv 0 (\mathrm{mod}~4)$), and it can be found in polynomial time. Our techniques can be extended, with similar results, to the case in which the weights are arbitrary (not necessarily $\pm 1$), and to the case that $p(V)\neq 0$ and the excess supply/demand should be split evenly. They also apply to the problem of partitioning a graph with two types of nodes into two large connected subgraphs that preserve approximately the proportion of the two types.

math.CO↗

arXiv2015

Generation of Synthetic Spatially Embedded Power Grid Networks

The development of algorithms for enhancing the resilience and efficiency of the power grid requires performance evaluation with real topologies of power transmission networks. However, due to security reasons, such topologies and particularly the locations of the substations and the lines are usually not publicly available. Therefore, we study the structural properties of the North American grids and present an algorithm for generating synthetic spatially embedded networks with similar properties to a given grid. The algorithm uses the Gaussian Mixture Model (GMM) for density estimation of the node positions and generates a set of nodes with similar spatial distribution to the nodes in a given network. Then, it uses two procedures, which are inspired by the historical evolution of the grids, to connect the nodes. The algorithm has several tunable parameters that allow generating grids similar to any given grid. Particularly, we apply it to the Western Interconnection (WI) and to grids that operate under the SERC Reliability Corporation (SERC) and the Florida Reliability Coordinating Council (FRCC), and show that it generates grids with similar structural and spatial properties to these grids. To the best of our knowledge, this is the first attempt to consider the spatial distribution of the nodes and lines and its importance in generating synthetic power grids.

eess.SY↗

arXiv2014

Cascading Failures in Power Grids - Analysis and Algorithms

This paper focuses on cascading line failures in the transmission system of the power grid. Recent large-scale power outages demonstrated the limitations of percolation- and epid- emic-based tools in modeling cascades. Hence, we study cascades by using computational tools and a linearized power flow model. We first obtain results regarding the Moore-Penrose pseudo-inverse of the power grid admittance matrix. Based on these results, we study the impact of a single line failure on the flows on other lines. We also illustrate via simulation the impact of the distance and resistance distance on the flow increase following a failure, and discuss the difference from the epidemic models. We then study the cascade properties, considering metrics such as the distance between failures and the fraction of demand (load) satisfied after the cascade (yield). We use the pseudo-inverse of admittance matrix to develop an efficient algorithm to identify the cascading failure evolution, which can be a building block for cascade mitigation. Finally, we show that finding the set of lines whose removal has the most significant impact (under various metrics) is NP-Hard and introduce a simple heuristic for the minimum yield problem. Overall, the results demonstrate that using the resistance distance and the pseudo-inverse of admittance matrix provides important insights and can support the development of efficient algorithms.

eess.SY↗