arXiv subjects

Ivan Kobyzev

Publications and source records attributed to Ivan Kobyzev.

26 records · Page 2Linked to original sources

arXiv2021

A Short Study on Compressing Decoder-Based Language Models

Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks. The state-of-the-art of PLMs, however, are extremely large to be used on edge devices. As a result, the topic of model compression has attracted increasing attention in the NLP community. Most of the existing works focus on compressing encoder-based models (tiny-BERT, distilBERT, distilRoBERTa, etc), however, to the best of our knowledge, the compression of decoder-based models (such as GPT-2) has not been investigated much. Our paper aims to fill this gap. Specifically, we explore two directions: 1) we employ current state-of-the-art knowledge distillation techniques to improve fine-tuning of DistilGPT-2. 2) we pre-train a compressed GPT-2 model using layer truncation and compare it against the distillation-based method (DistilGPT2). The training time of our compressed model is significantly less than DistilGPT-2, but it can achieve better performance when fine-tuned on downstream tasks. We also demonstrate the impact of data cleaning on model performance.

cs.CL↗

arXiv2021

Polarized-VAE: Proximity Based Disentangled Representation Learning for Text Generation

Learning disentangled representations of real-world data is a challenging open problem. Most previous methods have focused on either supervised approaches which use attribute labels or unsupervised approaches that manipulate the factorization in the latent space of models such as the variational autoencoder (VAE) by training with task-specific losses. In this work, we propose polarized-VAE, an approach that disentangles select attributes in the latent space based on proximity measures reflecting the similarity between data points with respect to these attributes. We apply our method to disentangle the semantics and syntax of sentences and carry out transfer experiments. Polarized-VAE outperforms the VAE baseline and is competitive with state-of-the-art approaches, while being more a general framework that is applicable to other attribute disentanglement tasks.

cs.CL↗

arXiv2020

Tails of Lipschitz Triangular Flows

We investigate the ability of popular flow based methods to capture tail-properties of a target density by studying the increasing triangular maps used in these flow methods acting on a tractable source density. We show that the density quantile functions of the source and target density provide a precise characterization of the slope of transformation required to capture tails in a target density. We further show that any Lipschitz-continuous transport map acting on a source density will result in a density with similar tail properties as the source, highlighting the trade-off between a complex source density and a sufficiently expressive transformation to capture desirable properties of a target density. Subsequently, we illustrate that flow models like Real-NVP, MAF, and Glow as implemented originally lack the ability to capture a distribution with non-Gaussian tails. We circumvent this problem by proposing tail-adaptive flows consisting of a source distribution that can be learned simultaneously with the triangular map to capture tail-properties of a target density. We perform several synthetic and real-world experiments to compliment our theoretical findings.

math.ST↗

arXiv2020

Normalizing Flows: An Introduction and Review of Current Methods

Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact. The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning. We aim to provide context and explanation of the models, review current state-of-the-art literature, and identify open questions and promising future directions.

stat.ML↗

arXiv2020

Representation Learning for Dynamic Graphs: A Survey

Graphs arise naturally in many real-world applications including social networks, recommender systems, ontologies, biology, and computational finance. Traditionally, machine learning models for graphs have been mostly designed for static graphs. However, many applications involve evolving graphs. This introduces important challenges for learning and inference since nodes, attributes, and edges change over time. In this survey, we review the recent advances in representation learning for dynamic graphs, including dynamic knowledge graphs. We describe existing models from an encoder-decoder perspective, categorize these encoders and decoders based on the techniques they employ, and analyze the approaches in each category. We also review several prominent applications and widely used datasets and highlight directions for future research.

cs.LG↗

arXiv2020

Generating Emotionally Aligned Responses in Dialogues using Affect Control Theory

State-of-the-art neural dialogue systems excel at syntactic and semantic modelling of language, but often have a hard time establishing emotional alignment with the human interactant during a conversation. In this work, we bring Affect Control Theory (ACT), a socio-mathematical model of emotions for human-human interactions, to the neural dialogue generation setting. ACT makes predictions about how humans respond to emotional stimuli in social situations. Due to this property, ACT and its derivative probabilistic models have been successfully deployed in several applications of Human-Computer Interaction, including empathetic tutoring systems, assistive healthcare devices and two-person social dilemma games. We investigate how ACT can be used to develop affect-aware neural conversational agents, which produce emotionally aligned responses to prompts and take into consideration the affective identities of the interactants.

cs.CL↗

arXiv2018

A categorical approach to cyclic cohomology of quasi-Hopf algebras and Hopf algebroids

We apply categorical machinery to the problem of defining cyclic cohomology with coefficients in two particular cases, namely quasi-Hopf algebras and Hopf algebroids. In the case of the former, no definition was thus far available in the literature, and while a definition exists for the latter, we feel that our approach demystifies the seemingly arbitrary formulas present there. This paper emphasizes the importance of working with a biclosed monoidal category in order to obtain natural coefficients for a cyclic theory that are analogous to the stable anti-Yetter-Drinfeld contramodules for Hopf algebras.

math.KT↗

arXiv2018

Anti-Yetter-Drinfeld Modules for Quasi-Hopf Algebras

We apply categorical machinery to the problem of defining anti-Yetter-Drinfeld modules for quasi-Hopf algebras. While a definition of Yetter-Drinfeld modules in this setting, extracted from their categorical interpretation as the center of the monoidal category of modules has been given, none was available for the anti-Yetter-Drinfeld modules that serve as coefficients for a Hopf cyclic type cohomology theory for quasi-Hopf algebras. This is a followup paper to the authors' previous effort that addressed the somewhat different case of anti-Yetter-Drinfeld contramodule coefficients in this and in the Hopf algebroid setting.

math.KT↗