arXiv subjects

Rahul Goel

Publications and source records attributed to Rahul Goel.

At least 37 records · Page 2Linked to original sources

arXiv2021

Linear Predictive Coding for Acute Stress Prediction from Computer Mouse Movements

Prior work demonstrated the potential of using the Linear Predictive Coding (LPC) filter to approximate muscle stiffness and damping from computer mouse movements to predict acute stress levels of users. Theoretically, muscle stiffness and damping in the arm can be estimated using a mass-spring-damper (MSD) biomechanical model. However, the damping frequency (i.e., stiffness) and damping ratio values derived using LPC were not yet compared with those from a theoretical MSD model. This work demonstrates that the damping frequency and damping ratio from LPC are significantly correlated with those from an MSD model, thus confirming the validity of using LPC to infer muscle stiffness and damping. We also compare the stress level binary classification performance using the values from LPC and MSD with each other and with neural network-based baselines. We found comparable performance across all conditions demonstrating LPC and MSD model-based stress prediction efficacy, especially for longer mouse trajectories. Clinical relevance: This work demonstrates the validity of the LPC filter to approximate muscle stiffness and damping and predict acute stress from computer mouse movements.

eess.SP↗

arXiv2021

Do Facial Trait Correlates with Roll Call Voting in Parliament? Using fWHR to Study Performance in Politics

Research has shown that people recognize and select leaders based on their facial appearance. However, considering the correlation between the performance of leaders and their facial traits, empirical findings are mixed. This paper adds to the debate by focusing on two previously understudied aspects of facial traits among political leaders: (i) previous studies have focused on electoral success and achievement drive of politicians omitting their actual daily performance after elections; (ii) previous research has analyzed individual politicians omitting the context of social circumstances which potentially influence their performance. We address these issues by analyzing Ukrainian members of parliament (MPs) who voted for bills in six consecutive Verkhovna Rada starting from Rada 4 (2002-06) to Rada 9 (2019-Present) to study politicians' performance, which is defined as co-voting or cooperation between MPs in voting on the same bill. In simple words, we analyze whether politicians tend to follow leaders when voting. This ability to summon the votes of others is interpreted as better performance. To measure performance, we proposed a generic methodology named Feature Importance For Measuring Performance (FIMP) that can be used in various scenarios. Using FIMP, our data suggest that MPs vote has no impact from their colleagues with higher or lower facial width-to-height ratio (fWHR), a popular measure of the facial trait.

cs.SI↗

arXiv2021

Misinformation Detection on YouTube Using Video Captions

Millions of people use platforms such as YouTube, Facebook, Twitter, and other mass media. Due to the accessibility of these platforms, they are often used to establish a narrative, conduct propaganda, and disseminate misinformation. This work proposes an approach that uses state-of-the-art NLP techniques to extract features from video captions (subtitles). To evaluate our approach, we utilize a publicly accessible and labeled dataset for classifying videos as misinformation or not. The motivation behind exploring video captions stems from our analysis of videos metadata. Attributes such as the number of views, likes, dislikes, and comments are ineffective as videos are hard to differentiate using this information. Using caption dataset, the proposed models can classify videos among three classes (Misinformation, Debunking Misinformation, and Neutral) with 0.85 to 0.90 F1-score. To emphasize the relevance of the misinformation class, we re-formulate our classification problem as a two-class classification - Misinformation vs. others (Debunking Misinformation and Neutral). In our experiments, the proposed models can classify videos with 0.92 to 0.95 F1-score and 0.78 to 0.90 AUC ROC.

cs.LG↗

arXiv2021

Overall Behavioural Index (OBI) For Measuring Segregation

Segregation, defined as the degree of separation between two or more population groups, helps to understand a complex social environment and subsequently provides a basis for public policy intervention. To measure segregation, past works often propose indexes that are criticized for being over-simplified and over-reduced. In other words, these indexes use the highly aggregated information to measure segregation. In this paper, we propose three novel indexes to measure segregation, namely: (i) Individual Segregation Index (ISI), (ii) Individual Inclination Index (III), and (iii) Overall Behavioural Index (OBI). The ISI index measures individuals' segregation, and the III index reports the individuals' inclination towards other population groups. The OBI index, calculated using both III and ISI index, is non-simplified and not only recognizes individuals' connectivity behaviour but group's connectivity behavioural distribution as well. By considering commonly used Freeman's segregation and homophily index as baseline indexes, we compare the OBI index on real call data records (CDR) dataset of Estonia to show the effectiveness of the proposed indexes.

cs.SI↗

arXiv2021

Studying Leaders & Their Concerns Using Online Social Media During The Times Of Crisis -- A COVID Case Study

Online social media (OSM) has emerged as a prominent platform for debate on a wide range of issues. Even celebrities and public figures often share their opinions on a variety of topics through OSM platforms. One such subject that has gained a lot of coverage on Twitter is the Novel Coronavirus, officially known as COVID-19, which has become a pandemic and has sparked a crisis in human history. In this study, we examine 29 million tweets over three months to study highly influential users, whom we refer to as leaders. We recognize these leaders through social network techniques and analyze their tweets using text analysis. Using a community detection algorithm, we categorize these leaders into four clusters: research, news, health, and politics, with each cluster containing Twitter handles (accounts) of individual users or organizations. E.g., the health cluster includes the World Health Organization (@WHO), the Director-General of WHO (@DrTedros), and so on. The emotion analysis reveals that (i) all clusters show an equal amount of fear in their tweets, (ii) research and news clusters display more sadness than others, and (iii) health and politics clusters are attempting to win public trust. According to the text analysis, the (i) research cluster is more concerned with recognizing symptoms and the development of vaccination; (ii) news and politics clusters are mostly concerned with travel. We then show that we can use our findings to classify tweets into clusters with a score of 96% AUC ROC.

cs.SI↗

arXiv2021

Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems

Traditional goal-oriented dialogue systems rely on various components such as natural language understanding, dialogue state tracking, policy learning and response generation. Training each component requires annotations which are hard to obtain for every new domain, limiting scalability of such systems. Similarly, rule-based dialogue systems require extensive writing and maintenance of rules and do not scale either. End-to-End dialogue systems, on the other hand, do not require module-specific annotations but need a large amount of data for training. To overcome these problems, in this demo, we present Alexa Conversations, a new approach for building goal-oriented dialogue systems that is scalable, extensible as well as data efficient. The components of this system are trained in a data-driven manner, but instead of collecting annotated conversations for training, we generate them using a novel dialogue simulator based on a few seed dialogues and specifications of APIs and entities provided by the developer. Our approach provides out-of-the-box support for natural conversational phenomena like entity sharing across turns or users changing their mind during conversation without requiring developers to provide any such dialogue flows. We exemplify our approach using a simple pizza ordering task and showcase its value in reducing the developer burden for creating a robust experience. Finally, we evaluate our system using a typical movie ticket booking task and show that the dialogue simulator is an essential component of the system that leads to over $50\%$ improvement in turn-level action signature prediction accuracy.

cs.CL↗

arXiv2021

Update Frequently, Update Fast: Retraining Semantic Parsing Systems in a Fraction of Time

Currently used semantic parsing systems deployed in voice assistants can require weeks to train. Datasets for these models often receive small and frequent updates, data patches. Each patch requires training a new model. To reduce training time, one can fine-tune the previously trained model on each patch, but naive fine-tuning exhibits catastrophic forgetting - degradation of the model performance on the data not represented in the data patch. In this work, we propose a simple method that alleviates catastrophic forgetting and show that it is possible to match the performance of a model trained from scratch in less than 10% of a time via fine-tuning. The key to achieving this is supersampling and EWC regularization. We demonstrate the effectiveness of our method on multiple splits of the Facebook TOP and SNIPS datasets.

cs.CL↗

arXiv2020

COVID-19 and the stock market: evidence from Twitter

COVID-19 has had a much larger impact on the financial markets compared to previous epidemics because the news information is transferred over the social networks at a speed of light. Using Twitter's API, we compiled a unique dataset with more than 26 million COVID-19 related Tweets collected from February 2nd until May 1st, 2020. We find that more frequent use of the word "stock" in daily Tweets is associated with a substantial decline in log returns of three key US indices - Dow Jones Industrial Average, S&P500, and NASDAQ. The results remain virtually unchanged in multiple robustness checks.

econ.GN↗

arXiv2020

Mobility Based SIR Model For Pandemics -- With Case Study Of COVID-19

In the last decade, humanity has faced many different pandemics such as SARS, H1N1, and presently novel coronavirus (COVID-19). On one side, scientists are focusing on vaccinations, and on the other side, there is a need to propose models that can help us in understanding the spread of these pandemics as it can help governmental and other concerned agencies to be well prepared, especially from pandemics, which spreads faster like COVID-19. The main reason for some epidemic turning into pandemics is the connectivity among different regions of the world, which makes it easier to affect a wider geographical area, often worldwide. In addition, the population distribution and social coherence in the different regions of the world is non-uniform. Thus, once the epidemic enters a region, then the local population distribution plays an important role. Inspired by these ideas, we proposed a mobility-based SIR model for epidemics, which especially takes into account pandemic situations. To the best of our knowledge, this model is first of its kind, which takes into account the population distribution and connectivity of different geographic locations across the globe. In addition to presenting the mathematical proof of our model, we have performed extensive simulations using synthetic data to demonstrate our model's generalizability. To demonstrate the wider scope of our model, we used our model to forecast the COVID-19 cases for Estonia.

cs.SI↗

arXiv2019

MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines

MultiWOZ 2.0 (Budzianowski et al., 2018) is a recently released multi-domain dialogue dataset spanning 7 distinct domains and containing over 10,000 dialogues. Though immensely useful and one of the largest resources of its kind to-date, MultiWOZ 2.0 has a few shortcomings. Firstly, there is substantial noise in the dialogue state annotations and dialogue utterances which negatively impact the performance of state-tracking models. Secondly, follow-up work (Lee et al., 2019) has augmented the original dataset with user dialogue acts. This leads to multiple co-existent versions of the same dataset with minor modifications. In this work we tackle the aforementioned issues by introducing MultiWOZ 2.1. To fix the noisy state annotations, we use crowdsourced workers to re-annotate state and utterances based on the original utterances in the dataset. This correction process results in changes to over 32% of state annotations across 40% of the dialogue turns. In addition, we fix 146 dialogue utterances by canonicalizing slot values in the utterances to the values in the dataset ontology. To address the second problem, we combined the contributions of the follow-up works into MultiWOZ 2.1. Hence, our dataset also includes user dialogue acts as well as multiple slot descriptions per dialogue state slot. We then benchmark a number of state-of-the-art dialogue state tracking models on the MultiWOZ 2.1 dataset and show the joint state tracking performance on the corrected state annotations. We are publicly releasing MultiWOZ 2.1 to the community, hoping that this dataset resource will allow for more effective models across various dialogue subproblems to be built in the future.

cs.CL↗

arXiv2019

Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators

Encoder-decoder based neural architectures serve as the basis of state-of-the-art approaches in end-to-end open domain dialog systems. Since most of such systems are trained with a maximum likelihood~(MLE) objective they suffer from issues such as lack of generalizability and the generic response problem, i.e., a system response that can be an answer to a large number of user utterances, e.g., "Maybe, I don't know." Having explicit feedback on the relevance and interestingness of a system response at each turn can be a useful signal for mitigating such issues and improving system quality by selecting responses from different approaches. Towards this goal, we present a system that evaluates chatbot responses at each dialog turn for coherence and engagement. Our system provides explicit turn-level dialog quality feedback, which we show to be highly correlated with human evaluation. To show that incorporating this feedback in the neural response generation models improves dialog quality, we present two different and complementary mechanisms to incorporate explicit feedback into a neural response generation model: reranking and direct modification of the loss function during training. Our studies show that a response generation model that incorporates these combined feedback mechanisms produce more engaging and coherent responses in an open-domain spoken dialog setting, significantly improving the response quality using both automatic and human evaluation.

cs.CL↗

arXiv2019

Natural Language Generation at Scale: A Case Study for Open Domain Question Answering

Current approaches to Natural Language Generation (NLG) for dialog mainly focus on domain-specific, task-oriented applications (e.g. restaurant booking) using limited ontologies (up to 20 slot types), usually without considering the previous conversation context. Furthermore, these approaches require large amounts of data for each domain, and do not benefit from examples that may be available for other domains. This work explores the feasibility of applying statistical NLG to scenarios requiring larger ontologies, such as multi-domain dialog applications or open-domain question answering (QA) based on knowledge graphs. We model NLG through an Encoder-Decoder framework using a large dataset of interactions between real-world users and a conversational agent for open-domain QA. First, we investigate the impact of increasing the number of slot types on the generation quality and experiment with different partitions of the QA data with progressively larger ontologies (up to 369 slot types). Second, we perform multi-task learning experiments between open-domain QA and task-oriented dialog, and benchmark our model on a popular NLG dataset. Moreover, we experiment with using the conversational context as an additional input to improve response generation quality. Our experiments show the feasibility of learning statistical NLG models for open-domain QA with larger ontologies.

cs.CL↗

arXiv2019

Towards Universal Dialogue Act Tagging for Task-Oriented Dialogues

Machine learning approaches for building task-oriented dialogue systems require large conversational datasets with labels to train on. We are interested in building task-oriented dialogue systems from human-human conversations, which may be available in ample amounts in existing customer care center logs or can be collected from crowd workers. Annotating these datasets can be prohibitively expensive. Recently multiple annotated task-oriented human-machine dialogue datasets have been released, however their annotation schema varies across different collections, even for well-defined categories such as dialogue acts (DAs). We propose a Universal DA schema for task-oriented dialogues and align existing annotated datasets with our schema. Our aim is to train a Universal DA tagger (U-DAT) for task-oriented dialogues and use it for tagging human-human conversations. We investigate multiple datasets, propose manual and automated approaches for aligning the different schema, and present results on a target corpus of human-human dialogues. In unsupervised learning experiments we achieve an F1 score of 54.1% on system turns in human-human dialogues. In a semi-supervised setup, the F1 score increases to 57.7% which would otherwise require at least 1.7K manually annotated turns. For new domains, we show further improvements when unlabeled or labeled target domain data is available.

cs.CL↗

arXiv2019

HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking

Recent works on end-to-end trainable neural network based approaches have demonstrated state-of-the-art results on dialogue state tracking. The best performing approaches estimate a probability distribution over all possible slot values. However, these approaches do not scale for large value sets commonly present in real-life applications and are not ideal for tracking slot values that were not observed in the training set. To tackle these issues, candidate-generation-based approaches have been proposed. These approaches estimate a set of values that are possible at each turn based on the conversation history and/or language understanding outputs, and hence enable state tracking over unseen values and large value sets however, they fall short in terms of performance in comparison to the first group. In this work, we analyze the performance of these two alternative dialogue state tracking methods, and present a hybrid approach (HyST) which learns the appropriate method for each slot type. To demonstrate the effectiveness of HyST on a rich-set of slot types, we experiment with the recently released MultiWOZ-2.0 multi-domain, task-oriented dialogue-dataset. Our experiments show that HyST scales to multi-domain applications. Our best performing model results in a relative improvement of 24% and 10% over the previous SOTA and our best baseline respectively.

cs.CL↗

arXiv2019

A guide to Value of Information methods for prioritising research in health impact modelling

Health impact simulation models are used to predict how a proposed intervention or scenario will affect public health outcomes, based on available data and knowledge of the process. The outputs of these models are uncertain due to uncertainty in the structure and inputs to the model. In order to assess the extent of uncertainty in the outcome we must quantify all potentially relevant uncertainties. Then to reduce uncertainty we should obtain and analyse new data, but it may be unclear which parts of the model would benefit from such extra research. This paper presents methods for uncertainty quantification and research prioritisation in health impact models based on Value of Information (VoI) analysis. Specifically, we 1. discuss statistical methods for quantifying uncertainty in this type of model, given the typical kinds of data that are available, which are often weaker than the ideal data that are desired; 2. show how the expected value of partial perfect information (EVPPI) can be calculated to compare how uncertainty in each model parameter influences uncertainty in the output; 3. show how research time can be prioritised efficiently, in the light of which components contribute most to outcome uncertainty. The same methods can be used whether the purpose of the model is to estimate quantities of interest to a policy maker, or to explicitly decide between policies. We demonstrate how these methods might be used in a model of the impact of air pollution on health outcomes.

stat.ME↗

arXiv2019

Practical Semantic Parsing for Spoken Language Understanding

Executable semantic parsing is the task of converting natural language utterances into logical forms that can be directly used as queries to get a response. We build a transfer learning framework for executable semantic parsing. We show that the framework is effective for Question Answering (Q&A) as well as for Spoken Language Understanding (SLU). We further investigate the case where a parser on a new domain can be learned by exploiting data on other domains, either via multi-task learning between the target domain and an auxiliary domain or via pre-training on the auxiliary domain and fine-tuning on the target domain. With either flavor of transfer learning, we are able to improve performance on most domains; we experiment with public data sets such as Overnight and NLmaps as well as with commercial SLU data. The experiments carried out on data sets that are different in nature show how executable semantic parsing can unify different areas of NLP such as Q&A and SLU.

cs.CL↗

arXiv2018

On Evaluating and Comparing Open Domain Dialog Systems

Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliver the best social conversational experience. Alexa Prize provided the academic community with the unique opportunity to perform research with a live system used by millions of users. The subjectivity associated with evaluating conversations is key element underlying the challenge of building non-goal oriented dialogue systems. In this paper, we propose a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement. The proposed metrics provide granular analysis of the conversational agents, which is not captured in human ratings. We show that these metrics can be used as a reasonable proxy for human judgment. We provide a mechanism to unify the metrics for selecting the top performing agents, which has also been applied throughout the Alexa Prize competition. To our knowledge, to date it is the largest setting for evaluating agents with millions of conversations and hundreds of thousands of ratings from users. We believe that this work is a step towards an automatic evaluation process for conversational AIs.

cs.CL↗

arXiv2018

Flexible and Scalable State Tracking Framework for Goal-Oriented Dialogue Systems

Goal-oriented dialogue systems typically rely on components specifically developed for a single task or domain. This limits such systems in two different ways: If there is an update in the task domain, the dialogue system usually needs to be updated or completely re-trained. It is also harder to extend such dialogue systems to different and multiple domains. The dialogue state tracker in conventional dialogue systems is one such component - it is usually designed to fit a well-defined application domain. For example, it is common for a state variable to be a categorical distribution over a manually-predefined set of entities (Henderson et al., 2013), resulting in an inflexible and hard-to-extend dialogue system. In this paper, we propose a new approach for dialogue state tracking that can generalize well over multiple domains without incorporating any domain-specific knowledge. Under this framework, discrete dialogue state variables are learned independently and the information of a predefined set of possible values for dialogue state variables is not required. Furthermore, it enables adding arbitrary dialogue context as features and allows for multiple values to be associated with a single state variable. These characteristics make it much easier to expand the dialogue state space. We evaluate our framework using the widely used dialogue state tracking challenge data set (DSTC2) and show that our framework yields competitive results with other state-of-the-art results despite incorporating little domain knowledge. We also show that this framework can benefit from widely available external resources such as pre-trained word embeddings.

cs.AI↗