arXiv subjects

Rumen Dangovski

Publications and source records attributed to Rumen Dangovski.

30 records · Page 2Linked to original sources

arXiv2022

DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other "harmful" types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE by 2.3 absolute points on semantic textual similarity tasks.

cs.CL↗

arXiv2022

Equivariant Contrastive Learning

In state-of-the-art self-supervised learning (SSL) pre-training produces semantically good representations by encouraging them to be invariant under meaningful transformations prescribed from human knowledge. In fact, the property of invariance is a trivial instance of a broader class called equivariance, which can be intuitively understood as the property that representations transform according to the way the inputs transform. Here, we show that rather than using only invariance, pre-training that encourages non-trivial equivariance to some transformations, while maintaining invariance to other transformations, can be used to improve the semantic quality of representations. Specifically, we extend popular SSL methods to a more general framework which we name Equivariant Self-Supervised Learning (E-SSL). In E-SSL, a simple additional pre-training objective encourages equivariance by predicting the transformations applied to the input. We demonstrate E-SSL's effectiveness empirically on several popular computer vision benchmarks, e.g. improving SimCLR to 72.5% linear probe accuracy on ImageNet. Furthermore, we demonstrate usefulness of E-SSL for applications beyond computer vision; in particular, we show its utility on regression problems in photonics science. Our code, datasets and pre-trained models are available at https://github.com/rdangovs/essl to aid further research in E-SSL.

cs.CV↗

arXiv2021

Meta-Learning and Self-Supervised Pretraining for Real World Image Translation

Recent advances in deep learning, in particular enabled by hardware advances and big data, have provided impressive results across a wide range of computational problems such as computer vision, natural language, or reinforcement learning. Many of these improvements are however constrained to problems with large-scale curated data-sets which require a lot of human labor to gather. Additionally, these models tend to generalize poorly under both slight distributional shifts and low-data regimes. In recent years, emerging fields such as meta-learning or self-supervised learning have been closing the gap between proof-of-concept results and real-life applications of machine learning by extending deep-learning to the semi-supervised and few-shot domains. We follow this line of work and explore spatio-temporal structure in a recently introduced image-to-image translation problem in order to: i) formulate a novel multi-task few-shot image generation benchmark and ii) explore data augmentations in contrastive pre-training for image translation downstream tasks. We present several baselines for the few-shot problem and discuss trade-offs between different approaches. Our code is available at https://github.com/irugina/meta-image-translation.

cs.CV↗

arXiv2021

Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science

Deep learning techniques have been increasingly applied to the natural sciences, e.g., for property prediction and optimization or material discovery. A fundamental ingredient of such approaches is the vast quantity of labelled data needed to train the model; this poses severe challenges in data-scarce settings where obtaining labels requires substantial computational or labor resources. Here, we introduce surrogate- and invariance-boosted contrastive learning (SIB-CL), a deep learning framework which incorporates three ``inexpensive'' and easily obtainable auxiliary information sources to overcome data scarcity. Specifically, these are: 1)~abundant unlabeled data, 2)~prior knowledge of symmetries or invariances and 3)~surrogate data obtained at near-zero cost. We demonstrate SIB-CL's effectiveness and generality on various scientific problems, e.g., predicting the density-of-states of 2D photonic crystals and solving the 3D time-independent Schrodinger equation. SIB-CL consistently results in orders of magnitude reduction in the number of labels needed to achieve the same network accuracies.

cs.LG↗

arXiv2020

Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications

Deep neural networks have become the standard approach to building reliable Natural Language Processing (NLP) applications, ranging from Neural Machine Translation (NMT) to dialogue systems. However, improving accuracy by increasing the model size requires a large number of hardware computations, which can slow down NLP applications significantly at inference time. To address this issue, we propose a novel vector-vector-matrix architecture (VVMA), which greatly reduces the latency at inference time for NMT. This architecture takes advantage of specialized hardware that has low-latency vector-vector operations and higher-latency vector-matrix operations. It also reduces the number of parameters and FLOPs for virtually all models that rely on efficient matrix multipliers without significantly impacting accuracy. We present empirical results suggesting that our framework can reduce the latency of sequence-to-sequence and Transformer models used for NMT by a factor of four. Finally, we show evidence suggesting that our VVMA extends to other domains, and we discuss novel hardware for its efficient use.

cs.CL↗

arXiv2020

Contextualizing Enhances Gradient Based Meta Learning

Meta learning methods have found success when applied to few shot classification problems, in which they quickly adapt to a small number of labeled examples. Prototypical representations, each representing a particular class, have been of particular importance in this setting, as they provide a compact form to convey information learned from the labeled examples. However, these prototypes are just one method of representing this information, and they are narrow in their scope and ability to classify unseen examples. We propose the implementation of contextualizers, which are generalizable prototypes that adapt to given examples and play a larger role in classification for gradient-based models. We demonstrate how to equip meta learning methods with contextualizers and show that their use can significantly boost performance on a range of few shot learning datasets. We also present figures of merit demonstrating the potential benefits of contextualizers, along with analysis of how models make use of them. Our approach is particularly apt for low-data environments where it is difficult to update parameters without overfitting. Our implementation and instructions to reproduce the experiments are available at https://github.com/naveace/proto-context.

cs.LG↗

arXiv2018

WaveletNet: Logarithmic Scale Efficient Convolutional Neural Networks for Edge Devices

We present a logarithmic-scale efficient convolutional neural network architecture for edge devices, named WaveletNet. Our model is based on the well-known depthwise convolution, and on two new layers, which we introduce in this work: a wavelet convolution and a depthwise fast wavelet transform. By breaking the symmetry in channel dimensions and applying a fast algorithm, WaveletNet shrinks the complexity of convolutional blocks by an O(logD/D) factor, where D is the number of channels. Experiments on CIFAR-10 and ImageNet classification show superior and comparable performances of WaveletNet compared to state-of-the-art models such as MobileNetV2.

cs.LG↗

arXiv2017

Shaping Long-lived Electron Wavepackets for Customizable Optical Spectra

Electrons in atoms and molecules are versatile physical systems covering a vast range of light-matter interactions, enabling the physics of Rydberg states, photon-photon bound states, simulation of condensed matter Hamiltonians, and quantum sources of light. A limitation on the versatility of such electronic systems for optical interactions would appear to arise from the discrete nature of the electronic transitions and from the limited ionization energy, constraining the energy scale through which the rich physics of bound electrons can be accessed. In this work, we propose the concept of shaping spatially confined electron wavepackets as superpositions of extended states in the ionization continuum. These wavepackets enable customizable optical emission spectra transitions in the eV-keV range. We find that the specific shaping lengthens the diffraction lifetime of the wavepackets in exchange for increasing their spatial spreads. Finally, we study the spontaneous radiative transitions of these excited states, examining the influence of the wavepacket shape and size on the radiation rate of the excited state. We observe that the rate of radiative capture is primarily limited by the diffraction lifetime of the wavepacket. The approach proposed in this work could have applications towards developing designer emitters at tunable energy and length scales, potentially bringing the physics of Rydberg states to new energy scales and extending the range and versatility of emitters that can be developed using shaped electrons.

quant-ph↗

arXiv2017

Rotational Unit of Memory

The concepts of unitary evolution matrices and associative memory have boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art performance in a variety of sequential tasks. However, RNN still have a limited capacity to manipulate long-term memory. To bypass this weakness the most successful applications of RNN use external techniques such as attention mechanisms. In this paper we propose a novel RNN model that unifies the state-of-the-art approaches: Rotational Unit of Memory (RUM). The core of RUM is its rotational operation, which is, naturally, a unitary matrix, providing architectures with the power to learn long-term dependencies by overcoming the vanishing and exploding gradients problem. Moreover, the rotational unit also serves as associative memory. We evaluate our model on synthetic memorization, question answering and language modeling tasks. RUM learns the Copying Memory task completely and improves the state-of-the-art result in the Recall task. RUM's performance in the bAbI Question Answering task is comparable to that of models with attention mechanism. We also improve the state-of-the-art result to 1.189 bits-per-character (BPC) loss in the Character Level Penn Treebank (PTB) task, which is to signify the applications of RUM to real-world sequential data. The universality of our construction, at the core of RNN, establishes RUM as a promising approach to language modeling, speech recognition and machine translation.

cs.LG↗

arXiv2017

Self-avoiding Walks of Lattice Strips

We study self-avoiding walks on restricted square lattices, more precisely on the lattice strips $\mathbb{Z} \times \{-1,0,1\}$ and $\mathbb{Z}\times \{-1,0,1,2\}$. We obtain the value of the connective constant for the $\mathbb{Z} \times \{-1,0,1\}$ lattice in a new shorter way and deduce close bounds for the connective constant for the $\mathbb{Z}\times \{-1,0,1,2\}$ lattice. Moreover, for both lattice strips we find close lower and upper bounds for the number of SAWs of length $n$ by using the connective constant.

math.CO↗

arXiv2015

Weitzenboeck derivations of free metabelian associative algebras

By the classical theorem of Weitzenboeck the algebra of constants (i.e., the kernel) of a nonzero locally nilpotent linear derivation of the polynomial algebra K[X] in d variables over a field K of characteristic 0 is finitely generated. As a noncommutative generalization one considers the algebra of constants of a locally nilpotent linear derivation of a d-generated relatively free algebra F(V) in a variety V of unitary associative algebras over K. It is known that the algebra of constants of F(V) is finitely generated if and only if V satisfies a polynomial identity which does not hold for the algebra of 2 x 2 upper triangular matrices. Hence the free metabelian associative algebra F(M) is a crucial object to study. We show that the vector space of the constants in the commutator ideal F'(M) is a finitely generated module of the algebra of constants of the polynomial algebra K[U,V] in 2d variables, where the derivation acts on U and V in the same way as on X. For small d, we calculate the Hilbert series of the constants in F'(M) and find the generators of the related module. This gives also an (infinite) set of generators of the algebra of constants in F(M).

math.RA↗

arXiv2013

Weitzenboeck derivations of free metabelian Lie algebras

A nonzero locally nilpotent linear derivation of the polynomial algebra K[X] in d variables over a field K of characteristic 0 is called a Weitzenboeck derivation. The classical theorem of Weitzenboeck states that the algebra of constants (which coincides with the algebra of invariants of a single unipotent transformation) is finitely generated. Similarly one may consider the algebra of constants of a locally nilpotent linear derivation of a finitely generated (not necessarily commutative or associative) algebra which is relatively free in a variety of algebras over K. Now the algebra of constants is usually not finitely generated. Except for some trivial cases this holds for the algebra of constants of the free metabelian Lie algebra L/L" with d generators. We show that the vector space of the constants in the commutator ideal L'/L" is a finitely generated module over the algebra of constants in K[X]. For small d, we calculate the Hilbert series of the algebra of constants in L/L" and find the generators of the module of the constants in L'/L". This gives also an (infinite) set of generators of the Lie algebra of constants in L/L".

math.RA↗