arXiv subjects

Eli Shlizerman

Publications and source records attributed to Eli Shlizerman.

54 records · Page 3Linked to original sources

arXiv2020

Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements

We propose a novel system that takes as an input body movements of a musician playing a musical instrument and generates music in an unsupervised setting. Learning to generate multi-instrumental music from videos without labeling the instruments is a challenging problem. To achieve the transformation, we built a pipeline named 'Multi-instrumentalistNet' (MI Net). At its base, the pipeline learns a discrete latent representation of various instruments music from log-spectrogram using a Vector Quantized Variational Autoencoder (VQ-VAE) with multi-band residual blocks. The pipeline is then trained along with an autoregressive prior conditioned on the musician's body keypoints movements encoded by a recurrent neural network. Joint training of the prior with the body movements encoder succeeds in the disentanglement of the music into latent features indicating the musical components and the instrumental features. The latent space results in distributions that are clustered into distinct instruments from which new music can be generated. Furthermore, the VQ-VAE architecture supports detailed music generation with additional conditioning. We show that a Midi can further condition the latent space such that the pipeline will generate the exact content of the music being played by the instrument in the video. We evaluate MI Net on two datasets containing videos of 13 instruments and obtain generated music of reasonable audio quality, easily associated with the corresponding instrument, and consistent with the music audio content.

cs.SD↗

arXiv2020

Neuro-PC: Causal Functional Connectivity from Neural Dynamics

Functional connectome extends the anatomical connectome by capturing the relations between neurons according to their activity and interactions. When these relations are causal, the functional connectome maps how neural activity flows within neural circuits and provides the possibility for inference of functional neural pathways, such as sensory-motor-behavioral pathways. While there exist various information approaches for non-causal estimations of the functional connectome, approaches that characterize the causal functional connectivity - the causal relationships between neuronal time series, are scarce. In this work, we develop the Neuro-PC algorithm which is a novel methodology for inferring the causal functional connectivity between neurons from multi-dimensional time series, such as neuronal recordings. The core of our methodology relies on a novel adaptation of the PC algorithm, a state-of-the-art method for statistical causal inference, to the multi-dimensional time-series of neural dynamics. We validate the performance of the method on network motifs with various interactions between their neurons simulated using continuous-time artificial network of neurons. We then consider the application of the method to obtain the causal functional connectome for recent multi-array electrophysiological recordings from the mouse visual cortex in the presence of different stimuli. We show how features of the mapping can be used for quantification of the similarities between neural responses subject to different stimuli.

q-bio.NC↗

arXiv2020

On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of RNN training dynamics because forward propagation of information is related to the backward propagation of error gradients. LEs measure the asymptotic rates of expansion and contraction of nonlinear system trajectories, and generalize stability analysis to the time-varying attractors structuring the non-autonomous dynamics of data-driven RNNs. As a tool to understand and exploit stability of training dynamics, the Lyapunov spectrum fills an existing gap between prescriptive mathematical approaches of limited scope and computationally-expensive empirical approaches. To leverage this tool, we implement an efficient way to compute LEs for RNNs during training, discuss the aspects specific to standard RNN architectures driven by typical sequential datasets, and show that the Lyapunov spectrum can serve as a robust readout of training stability across hyperparameters. With this exposition-oriented contribution, we hope to draw attention to this understudied, but theoretically grounded tool for understanding training stability in RNNs.

cs.LG↗

arXiv2020

Audeo: Audio Generation for a Silent Performance Video

We present a novel system that gets as an input video frames of a musician playing the piano and generates the music for that video. Generation of music from visual cues is a challenging problem and it is not clear whether it is an attainable goal at all. Our main aim in this work is to explore the plausibility of such a transformation and to identify cues and components able to carry the association of sounds with visual events. To achieve the transformation we built a full pipeline named `\textit{Audeo}' containing three components. We first translate the video frames of the keyboard and the musician hand movements into raw mechanical musical symbolic representation Piano-Roll (Roll) for each video frame which represents the keys pressed at each time step. We then adapt the Roll to be amenable for audio synthesis by including temporal correlations. This step turns out to be critical for meaningful audio generation. As a last step, we implement Midi synthesizers to generate realistic music. \textit{Audeo} converts video to audio smoothly and clearly with only a few setup constraints. We evaluate \textit{Audeo} on `in the wild' piano performance videos and obtain that their generated music is of reasonable audio quality and can be successfully recognized with high precision by popular music identification software.

cs.CV↗

arXiv2020

Iterate & Cluster: Iterative Semi-Supervised Action Recognition

We propose a novel system for active semi-supervised feature-based action recognition. Given time sequences of features tracked during movements our system clusters the sequences into actions. Our system is based on encoder-decoder unsupervised methods shown to perform clustering by self-organization of their latent representation through the auto-regression task. These methods were tested on human action recognition benchmarks and outperformed non-feature based unsupervised methods and achieved comparable accuracy to skeleton-based supervised methods. However, such methods rely on K-Nearest Neighbours (KNN) associating sequences to actions, and general features with no annotated data would correspond to approximate clusters which could be further enhanced. Our system proposes an iterative semi-supervised method to address this challenge and to actively learn the association of clusters and actions. The method utilizes latent space embedding and clustering of the unsupervised encoder-decoder to guide the selection of sequences to be annotated in each iteration. Each iteration, the selection aims to enhance action recognition accuracy while choosing a small number of sequences for annotation. We test the approach on human skeleton-based action recognition benchmarks assuming that only annotations chosen by our method are available and on mouse movements videos recorded in lab experiments. We show that our system can boost recognition performance with only a small percentage of annotations. The system can be used as an interactive annotation tool to guide labeling efforts for 'in the wild' videos of various objects and actions to reach robust recognition.

cs.CV↗

arXiv2020

Deep Reinforcement Learning for Neural Control

We present a novel methodology for control of neural circuits based on deep reinforcement learning. Our approach achieves aimed behavior by generating external continuous stimulation of existing neural circuits (neuromodulation control) or modulations of neural circuits architecture (connectome control). Both forms of control are challenging due to nonlinear and recurrent complexity of neural activity. To infer candidate control policies, our approach maps neural circuits and their connectome into a grid-world like setting and infers the actions needed to achieve aimed behavior. The actions are inferred by adaptation of deep Q-learning methods known for their robust performance in navigating grid-worlds. We apply our approach to the model of \textit{C. elegans} which simulates the full somatic nervous system with muscles and body. Our framework successfully infers neuropeptidic currents and synaptic architectures for control of chemotaxis. Our findings are consistent with in vivo measurements and provide additional insights into neural control of chemotaxis. We further demonstrate the generality and scalability of our methods by inferring chemotactic neural circuits from scratch.

q-bio.NC↗

arXiv2020

BI-MAML: Balanced Incremental Approach for Meta Learning

We present a novel Balanced Incremental Model Agnostic Meta Learning system (BI-MAML) for learning multiple tasks. Our method implements a meta-update rule to incrementally adapt its model to new tasks without forgetting old tasks. Such a capability is not possible in current state-of-the-art MAML approaches. These methods effectively adapt to new tasks, however, suffer from 'catastrophic forgetting' phenomena, in which new tasks that are streamed into the model degrade the performance of the model on previously learned tasks. Our system performs the meta-updates with only a few-shots and can successfully accomplish them. Our key idea for achieving this is the design of balanced learning strategy for the baseline model. The strategy sets the baseline model to perform equally well on various tasks and incorporates time efficiency. The balanced learning strategy enables BI-MAML to both outperform other state-of-the-art models in terms of classification accuracy for existing tasks and also accomplish efficient adaption to similar new tasks with less required shots. We evaluate BI-MAML by conducting comparisons on two common benchmark datasets with multiple number of image classification tasks. BI-MAML performance demonstrates advantages in both accuracy and efficiency.

cs.LG↗

arXiv2020

Clustering and Recognition of Spatiotemporal Features through Interpretable Embedding of Sequence to Sequence Recurrent Neural Networks

Encoder-decoder recurrent neural network models (RNN Seq2Seq) have achieved great success in ubiquitous areas of computation and applications. It was shown to be successful in modeling data with both temporal and spatial dependencies for translation or prediction tasks. In this study, we propose an embedding approach to visualize and interpret the representation of data by these models. Furthermore, we show that the embedding is an effective method for unsupervised learning and can be utilized to estimate the optimality of model training. In particular, we demonstrate that embedding space projections of the decoder states of RNN Seq2Seq model trained on sequences prediction are organized in clusters capturing similarities and differences in the dynamics of these sequences. Such performance corresponds to an unsupervised clustering of any spatio-temporal features and can be employed for time-dependent problems such as temporal segmentation, clustering of dynamic activity, self-supervised classification, action recognition, failure prediction, etc. We test and demonstrate the application of the embedding methodology to time-sequences of 3D human body poses. We show that the methodology provides a high-quality unsupervised categorization of movements.

cs.LG↗

arXiv2019

PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition

We propose a novel system for unsupervised skeleton-based action recognition. Given inputs of body keypoints sequences obtained during various movements, our system associates the sequences with actions. Our system is based on an encoder-decoder recurrent neural network, where the encoder learns a separable feature representation within its hidden states formed by training the model to perform prediction task. We show that according to such unsupervised training the decoder and the encoder self-organize their hidden states into a feature space which clusters similar movements into the same cluster and distinct movements into distant clusters. Current state-of-the-art methods for action recognition are strongly supervised, i.e., rely on providing labels for training. Unsupervised methods have been proposed, however, they require camera and depth inputs (RGB+D) at each time step. In contrast, our system is fully unsupervised, does not require labels of actions at any stage, and can operate with body keypoints input only. Furthermore, the method can perform on various dimensions of body keypoints (2D or 3D) and include additional cues describing movements. We evaluate our system on three extensive action recognition benchmarks with different number of actions and examples. Our results outperform prior unsupervised skeleton-based methods, unsupervised RGB+D based methods on cross-view tests and while being unsupervised have similar performance to supervised skeleton-based action recognition.

cs.CV↗

arXiv2019

An Optical Frontend for a Convolutional Neural Network

The parallelism of optics and the miniaturization of optical components using nanophotonic structures, such as metasurfaces present a compelling alternative to electronic implementations of convolutional neural networks. The lack of a low-power optical nonlinearity, however, requires slow and energy-inefficient conversions between the electronic and optical domains. Here, we design an architecture which utilizes a single electrical to optical conversion by designing a free-space optical frontend unit that implements the linear operations of the first layer with the subsequent layers realized electronically. Speed and power analysis of the architecture indicates that the hybrid photonic-electronic architecture outperforms sole electronic architecture for large image sizes and kernels. Benchmarking of the photonic-electronic architecture on a modified version of AlexNet achieves a classification accuracy of 87% on images from the Kaggle Cats and Dogs challenge database.

cs.CV↗

arXiv2017

Audio to Body Dynamics

We present a method that gets as input an audio of violin or piano playing, and outputs a video of skeleton predictions which are further used to animate an avatar. The key idea is to create an animation of an avatar that moves their hands similarly to how a pianist or violinist would do, just from audio. Aiming for a fully detailed correct arms and fingers motion is a goal, however, it's not clear if body movement can be predicted from music at all. In this paper, we present the first result that shows that natural body dynamics can be predicted at all. We built an LSTM network that is trained on violin and piano recital videos uploaded to the Internet. The predicted points are applied onto a rigged avatar to create the animation.

eess.AS↗

arXiv2017

Functional Connectomics from Data: Probabilistic Graphical Models for Neuronal Network of C. elegans

We propose a data-driven approach to represent neuronal network dynamics as a Probabilistic Graphical Model (PGM). Our approach learns the PGM structure by employing dimension reduction to network response dynamics evoked by stimuli applied to each neuron separately. The outcome model captures how stimuli propagate through the network and thus represents functional dependencies between neurons, i.e., functional connectome. The benefit of using a PGM as the functional connectome is that posterior inference can be done efficiently and circumvent the complexities in direct inference of response pathways in dynamic neuronal networks. In particular, posterior inference reveals the relations between known stimuli and downstream neurons or allows to query which stimuli are associated with downstream neurons. For validation and as an example for our approach we apply our methodology to a model of Caenorhabiditis elegans nervous system which structure and dynamics are well-studied. From its dynamical model we collect time series of the network response and use singular value decomposition to obtain a low-dimensional projection of the time series data. We then extract dominant patterns in each data matrix to get pairwise dependency information and create a graphical model for the full somatic nervous system. The PGM enables us to obtain and verify underlying neuronal pathways dominant for known behavioral scenarios and to detect possible pathways for novel scenarios.

q-bio.NC↗

arXiv2017

Fokas's Uniform Transform Method for linear systems

We demonstrate the use of the Unified Transform Method or Method of Fokas for boundary value problems for systems of constant-coefficient linear partial differential equations. We discuss how the apparent branch singularities typically appearing in the global relation are removable, allowing the method to proceed, in essence, as for scalar problems. We illustrate the use of the method with boundary value problems for the Klein-Gordon equation and the linearized Fitzhugh-Nagumo system. The case of wave equations is treated separately in an appendix.

math.AP↗

arXiv2017

Classification of Fixed Point Network Dynamics From Multiple Node Timeseries Data

Fixed point networks are dynamic networks that encode stimuli via distinct output patterns. Although such networks are omnipresent in neural systems, their structures are typically unknown or poorly characterized. It is therefore valuable to use a supervised approach for resolving how a network encodes distinct inputs of interest, and the superposition of those inputs from sampled multiple node time series. In this paper we show that accomplishing such a task involves finding a low-dimensional state space from supervised recordings. We demonstrate that standard methods for dimension reduction are unable to provide the desired functionality of optimal separation of the fixed points and transient trajectories to them. However, the combination of dimension reduction with selection and optimization can successfully provide such functionality. Specifically, we propose two methods: Exclusive Threshold Reduction (ETR) and Optimal Exclusive Threshold Reduction (OETR) for finding a basis for the classification state space. We show that the classification space constructed upon combination of dimension reduction optimal separation can directly facilitate recognition of stimuli, and classify complex inputs (mixtures) into similarity classes. We test our methodology and compare it to standard state-of-the-art methods on a benchmark dataset - an experimental neuronal network (the olfactory system) that we recorded from to test these methods. We show that our methods are capable of providing a basis for the classification space in such network, and to perform recognition at a significantly better rate than previously proposed approaches.

q-bio.NC↗

arXiv2016

Symmetries constrain dynamics in a family of balanced neural networks

We examine a family of random firing-rate neural networks in which we enforce the neurobiological constraint of Dale's Law --- each neuron makes either excitatory or inhibitory connections onto its post-synaptic targets. We find that this constrained system may be described as a perturbation from a system with non-trivial symmetries. We analyze the symmetric system using the tools of equivariant bifurcation theory, and demonstrate that the symmetry-implied structures remain evident in the perturbed system. In comparison, spectral characteristics of the network coupling matrix are relatively uninformative about the behavior of the constrained system.

q-bio.NC↗

arXiv2015

The role of multistability and transient trajectories in networked dynamical systems: Connectomic dynamics of C. elegans and behavioral assays

The neural dynamics of the nematode C. elegans are experimentally low-dimensional and correspond to discrete behavioral states, where previous modeling work has found neural proxies for some of these states. Experimental results further suggest that dynamics may be understood as long-timescale transitions between multiple low-dimensional attractors. To identify multistable regimes of our model, we develop a method for systematic generation of bifurcation diagrams and their analysis in an interpretable low-dimensional subspace, showing the existence and nature of multistable input responses at a glance. Stimulation of the PLM neuron pair, experimentally associated with forward movement and shown in simulation to drive a limit cycle, defines our low-dimensional projection space. We then obtain bifurcation diagrams for single-neuron excitation over a range of amplitudes and which classify whether the dynamics in this projection space are associated with a limit cycle, fixed point, or multiple states. In the specific case of compound input into both the PLM pair and ASK pair we discover bistability of a limit cycle and a fixed point, with transitional timescales between different states being much longer than other timescales in the system. This suggests consistency of our model with the characterization of dynamics in neural systems as long-timescale transitions between discrete, low-dimensional attractors corresponding to behavioral states. Our methodology thus prescribes a method for identifying these states and transitions in response to arbitrary input.

q-bio.NC↗

arXiv2014

Low-dimensional functionality of complex network dynamics: Neuro-sensory integration in the Caenorhabditis elegans connectome

We develop a biophysical model of neuro-sensory integration in the model organism Caenorhabditis elegans. Building on recent experimental findings of the neuron conductances and their resolved connectome, we posit the first full dynamic model of the neural voltage excitations that allows for a characterization of input stimuli to behavioral responses. Thus a clear connection between receptory cell inputs to downstream motor-responses is illustrated, showing that robust, low-dimensional bifurcation structures dominate neural pathways of activity. The underlying bifurcation structures discovered, i.e. an induced Hopf bifurcation, are critical in explaining behavioral responses such as swimming and crawling. More broadly, we demonstrate that complex dynamical networks can produce robust functionality from underlying low-dimensional bifurcations.

q-bio.NC↗

arXiv2013

Data-driven modeling of the olfactory neural codes and their dynamics in the insect antennal lobe

Recordings from neurons in the insects' olfactory primary processing center, the antennal lobe (AL), reveal that the AL is able to process the input from chemical receptors into distinct neural activity patterns, called olfactory neural codes. These exciting results show the importance of neural codes and their relation to perception. The next challenge is to \emph{model the dynamics} of neural codes. In our study, we perform multichannel recordings from the projection neurons in the AL driven by different odorants. We then derive a neural network from the electrophysiological data. The network consists of lateral-inhibitory neurons and excitatory neurons, and is capable of producing unique olfactory neural codes for the tested odorants. Specifically, we (i) design a projection, an odor space, for the neural recording from the AL, which discriminates between distinct odorants trajectories (ii) characterize scent recognition, i.e., decision-making based on olfactory signals and (iii) infer the wiring of the neural circuit, the connectome of the AL. We show that the constructed model is consistent with biological observations, such as contrast enhancement and robustness to noise. The study answers a key biological question in identifying how lateral inhibitory neurons can be wired to excitatory neurons to permit robust activity patterns.

q-bio.NC↗