arXiv subjects

David Cox

Publications and source records attributed to David Cox.

At least 55 records · Page 3Linked to original sources

arXiv2019

ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization

The adaptive momentum method (AdaMM), which uses past gradients to update descent directions and learning rates simultaneously, has become one of the most popular first-order optimization methods for solving machine learning problems. However, AdaMM is not suited for solving black-box optimization problems, where explicit gradient forms are difficult or infeasible to obtain. In this paper, we propose a zeroth-order AdaMM (ZO-AdaMM) algorithm, that generalizes AdaMM to the gradient-free regime. We show that the convergence rate of ZO-AdaMM for both convex and nonconvex optimization is roughly a factor of $O(\sqrt{d})$ worse than that of the first-order AdaMM algorithm, where $d$ is problem size. In particular, we provide a deep understanding on why Mahalanobis distance matters in convergence of ZO-AdaMM and other AdaMM-type methods. As a byproduct, our analysis makes the first step toward understanding adaptive learning rate methods for nonconvex constrained optimization. Furthermore, we demonstrate two applications, designing per-image and universal adversarial attacks from black-box neural networks, respectively. We perform extensive experiments on ImageNet and empirically show that ZO-AdaMM converges much faster to a solution of high accuracy compared with $6$ state-of-the-art ZO optimization methods.

cs.LG↗

arXiv2019

Learning Nonlinear Brain Dynamics: van der Pol Meets LSTM

Many real-world data sets, especially in biology, are produced by complex nonlinear dynamical systems. In this paper, we focus on brain calcium imaging (CaI) of different organisms (zebrafish and rat), aiming to build a model of joint activation dynamics in large neuronal populations, including the whole brain of zebrafish. We propose a new approach for capturing dynamics of temporal SVD components that uses the coupled (multivariate) van der Pol (VDP) oscillator, a nonlinear ordinary differential equation (ODE) model describing neural activity, with a new parameter estimation technique that combines variable projection optimization and stochastic search. We show that the approach successfully handles nonlinearities and hidden state variables in the coupled VDP. The approach is accurate, achieving 0.82 to 0.94 correlation between the actual and model-generated components, and interpretable, as VDP's coupling matrix reveals anatomically meaningful positive (excitatory) and negative (inhibitory) interactions across different brain subsystems corresponding to spatial SVD components. Moreover, VDP is comparable to (or sometimes better than) recurrent neural networks (LSTM) for (short-term) prediction of future brain activity; VDP needs less parameters to train, which was a plus on our small training data. Finally, the overall best predictive method, greatly outperforming both VDP and LSTM in short- and long-term predictive settings on both datasets, was the new hybrid VDP-LSTM approach that used VDP to simulate large domain-specific dataset for LSTM pretraining; note that simple LSTM data-augmentation via noisy versions of training data was much less effective.

stat.ML↗

arXiv2019

The Rees Algebra of Parametric Curves via liftings

We study the defining equations of the Rees algebra of ideals arising from curve parametrizations in the plane and in rational normal scrolls, inspired by the work of Madsen and Kustin, Polini and Ulrich. The curves are related by work of Bernardi, Gimigliano and Ida, and we use this framework to relate the defining equations.

math.AC↗

arXiv2018

Conditional Infilling GANs for Data Augmentation in Mammogram Classification

Deep learning approaches to breast cancer detection in mammograms have recently shown promising results. However, such models are constrained by the limited size of publicly available mammography datasets, in large part due to privacy concerns and the high cost of generating expert annotations. Limited dataset size is further exacerbated by substantial class imbalance since "normal" images dramatically outnumber those with findings. Given the rapid progress of generative models in synthesizing realistic images, and the known effectiveness of simple data augmentation techniques (e.g. horizontal flipping), we ask if it is possible to synthetically augment mammogram datasets using generative adversarial networks (GANs). We train a class-conditional GAN to perform contextual in-filling, which we then use to synthesize lesions onto healthy screening mammograms. First, we show that GANs are capable of generating high-resolution synthetic mammogram patches. Next, we experimentally evaluate using the augmented dataset to improve breast cancer classification performance. We observe that a ResNet-50 classifier trained with GAN-augmented training data produces a higher AUROC compared to the same model trained only on traditionally augmented data, demonstrating the potential of our approach.

cs.CV↗

arXiv2018

A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception

While deep neural networks take loose inspiration from neuroscience, it is an open question how seriously to take the analogies between artificial deep networks and biological neuronal systems. Interestingly, recent work has shown that deep convolutional neural networks (CNNs) trained on large-scale image recognition tasks can serve as strikingly good models for predicting the responses of neurons in visual cortex to visual stimuli, suggesting that analogies between artificial and biological neural networks may be more than superficial. However, while CNNs capture key properties of the average responses of cortical neurons, they fail to explain other properties of these neurons. For one, CNNs typically require large quantities of labeled input data for training. Our own brains, in contrast, rarely have access to this kind of supervision, so to the extent that representations are similar between CNNs and brains, this similarity must arise via different training paths. In addition, neurons in visual cortex produce complex time-varying responses even to static inputs, and they dynamically tune themselves to temporal regularities in the visual environment. We argue that these differences are clues to fundamental differences between the computations performed in the brain and in deep networks. To begin to close the gap, here we study the emergent properties of a previously-described recurrent generative network that is trained to predict future video frames in a self-supervised manner. Remarkably, the model is able to capture a wide variety of seemingly disparate phenomena observed in visual cortex, ranging from single unit response dynamics to complex perceptual motion illusions. These results suggest potentially deep connections between recurrent predictive neural network models and the brain, providing new leads that can enrich both fields.

q-bio.NC↗

arXiv2018

Magnetic Force Microscopy Imaging Using Geometrically Constrained Nano-Domain Walls

Domain wall probes (DW-probes) were custom-made by modifying standard commercial magnetic force microscopy (MFM) probes using focused ion beam lithography. Excess of magnetic coating from the probes was milled out, leaving a V-shaped nanostructure on one face of the probe apex. Owing to the nanostructure's shape anisotropy, such probe has four possible magnetic states depending on the direction of the magnetization along each arm of the V-shape. Two states of opposite polarity are characterised by the presence of a geometrically constrained DW, pinned at the corner of the V-shape nanostructure. In the other two states, the magnetization curls around the corner with opposite chirality. Electron holography studies, supported by numerical simulations, demonstrate that a strong stray field emanates from the pinned DW, whilst a much weaker stray field is generated by the curling configurations. Using in situ MFM, we show that the magnetization states of the DW-probe can be easily controlled by applying an external magnetic field, thereby demonstrating that this type of probe can be used as a switchable tool with a low or high stray field intensity. We demonstrate that DW-probes enable acquiring magnetic images with a negligible interference with the sample magnetization, similar to that of commercial low moment probes, but with a higher magnetic contrast. In addition, the DW-probe in the curl state provides complementary information about the in-plane component of the sample's magnetization, which is not achievable by standard methods and provides additional information about the stray fields, e.g. as when imaging DWs.

physics.ins-det↗

arXiv2018

Recurrent computations for visual pattern completion

Making inferences from partial information constitutes a critical aspect of cognition. During visual perception, pattern completion enables recognition of poorly visible or occluded objects. We combined psychophysics, physiology and computational models to test the hypothesis that pattern completion is implemented by recurrent computations and present three pieces of evidence that are consistent with this hypothesis. First, subjects robustly recognized objects even when rendered <15% visible, but recognition was largely impaired when processing was interrupted by backward masking. Second, invasive physiological responses along the human ventral cortex exhibited visually selective responses to partially visible objects that were delayed compared to whole objects, suggesting the need for additional computations. These physiological delays were correlated with the effects of backward masking. Third, state-of-the-art feed-forward computational architectures were not robust to partial visibility. However, recognition performance was recovered when the model was augmented with attractor-based recurrent connectivity. These results provide a strong argument of plausibility for the role of recurrent computations in making visual inferences from partial information.

q-bio.NC↗

arXiv2018

Input-Aware Auto-Tuning of Compute-Bound HPC Kernels

Efficient implementations of HPC applications for parallel architectures generally rely on external software packages (e.g., BLAS, LAPACK, CUDNN). While these libraries provide highly optimized routines for certain characteristics of inputs (e.g., square matrices), they generally do not retain optimal performance across the wide range of problems encountered in practice. In this paper, we present an input-aware auto-tuning framework for matrix multiplications and convolutions, ISAAC, which uses predictive modeling techniques to drive highly parameterized PTX code templates towards not only hardware-, but also application-specific kernels. Numerical experiments on the NVIDIA Maxwell and Pascal architectures show up to 3x performance gains over both cuBLAS and cuDNN after only a few hours of auto-tuning.

cs.DC↗

arXiv2017

Using Human Brain Activity to Guide Machine Learning

Machine learning is a field of computer science that builds algorithms that learn. In many cases, machine learning algorithms are used to recreate a human ability like adding a caption to a photo, driving a car, or playing a game. While the human brain has long served as a source of inspiration for machine learning, little effort has been made to directly use data collected from working brains as a guide for machine learning algorithms. Here we demonstrate a new paradigm of "neurally-weighted" machine learning, which takes fMRI measurements of human brain activity from subjects viewing images, and infuses these data into the training process of an object recognition learning algorithm to make it more consistent with the human brain. After training, these neurally-weighted classifiers are able to classify images without requiring any additional neural data. We show that our neural-weighting approach can lead to large performance gains when used with traditional machine vision features, as well as to significant improvements with already high-performing convolutional neural network features. The effectiveness of this approach points to a path forward for a new class of hybrid machine learning algorithms which take both inspiration and direct constraints from neuronal data.

cs.CV↗

arXiv2017

A Multi-Scale CNN and Curriculum Learning Strategy for Mammogram Classification

Screening mammography is an important front-line tool for the early detection of breast cancer, and some 39 million exams are conducted each year in the United States alone. Here, we describe a multi-scale convolutional neural network (CNN) trained with a curriculum learning strategy that achieves high levels of accuracy in classifying mammograms. Specifically, we first train CNN-based patch classifiers on segmentation masks of lesions in mammograms, and then use the learned features to initialize a scanning-based model that renders a decision on the whole image, trained end-to-end on outcome data. We demonstrate that our approach effectively handles the "needle in a haystack" nature of full-image mammogram classification, achieving 0.92 AUROC on the DDSM dataset.

cs.CV↗

arXiv2017

Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning

While great strides have been made in using deep learning algorithms to solve supervised learning tasks, the problem of unsupervised learning - leveraging unlabeled examples to learn about the structure of a domain - remains a difficult unsolved challenge. Here, we explore prediction of future frames in a video sequence as an unsupervised learning rule for learning about the structure of the visual world. We describe a predictive neural network ("PredNet") architecture that is inspired by the concept of "predictive coding" from the neuroscience literature. These networks learn to predict future frames in a video sequence, with each layer in the network making local predictions and only forwarding deviations from those predictions to subsequent network layers. We show that these networks are able to robustly learn to predict the movement of synthetic (rendered) objects, and that in doing so, the networks learn internal representations that are useful for decoding latent object parameters (e.g. pose) that support object recognition with fewer training views. We also show that these networks can scale to complex natural image streams (car-mounted camera videos), capturing key aspects of both egocentric movement and the movement of objects in the visual scene, and the representation learned in this setting is useful for estimating the steering angle. Altogether, these results suggest that prediction represents a powerful framework for unsupervised learning, allowing for implicit learning of object and scene structure.

cs.LG↗

arXiv2016

Tensor Switching Networks

We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The TS network copies its entire input vector to different locations in an expanded representation, with the location determined by its hidden unit activity. In this way, even a simple linear readout from the TS representation can implement a highly expressive deep-network-like function. The TS network hence avoids the vanishing gradient problem by construction, at the cost of larger representation size. We develop several methods to train the TS network, including equivalent kernels for infinitely wide and deep TS networks, a one-pass linear learning algorithm, and two backpropagation-inspired representation learning algorithms. Our experimental results demonstrate that the TS network is indeed more expressive and consistently learns faster than standard ReLU networks.

cs.NE↗

arXiv2016

Syntactically Informed Text Compression with Recurrent Neural Networks

We present a self-contained system for constructing natural language models for use in text compression. Our system improves upon previous neural network based models by utilizing recent advances in syntactic parsing -- Google's SyntaxNet -- to augment character-level recurrent neural networks. RNNs have proven exceptional in modeling sequence data such as text, as their architecture allows for modeling of long-term contextual information.

cs.LG↗

arXiv2016

Delta Epsilon Alpha Star: A PAC-Admissible Search Algorithm

Delta Epsilon Alpha Star is a minimal coverage, real-time robotic search algorithm that yields a moderately aggressive search path with minimal backtracking. Search performance is bounded by a placing a combinatorial bound, epsilon and delta, on the maximum deviation from the theoretical shortest path and the probability at which further deviations can occur. Additionally, we formally define the notion of PAC-admissibility -- a relaxed admissibility criteria for algorithms, and show that PAC-admissible algorithms are better suited to robotic search situations than epsilon-admissible or strict algorithms.

cs.AI↗

arXiv2016

Clique Topology Reveals Intrinsic Geometric Structure in Neural Correlations: An Overview

This publication serves as an overview of clique topology -- a novel matrix analysis technique used to extract structural features from neural activity data that contains hidden nonlinearities. We highlight work done by Gusti et al. which introduces clique topology and verifies its applicability to neural feature extraction by showing that neural correlations in the rat hippocampus are determined by geometric structure of hippocampal circuits, rather than being a consequence of positional coding.

q-bio.NC↗

arXiv2016

Unsupervised Learning of Visual Structure using Predictive Generative Networks

The ability to predict future states of the environment is a central pillar of intelligence. At its core, effective prediction requires an internal model of the world and an understanding of the rules by which the world changes. Here, we explore the internal models developed by deep neural networks trained using a loss based on predicting future frames in synthetic video sequences, using a CNN-LSTM-deCNN framework. We first show that this architecture can achieve excellent performance in visual sequence prediction tasks, including state-of-the-art performance in a standard 'bouncing balls' dataset (Sutskever et al., 2009). Using a weighted mean-squared error and adversarial loss (Goodfellow et al., 2014), the same architecture successfully extrapolates out-of-the-plane rotations of computer-generated faces. Furthermore, despite being trained end-to-end to predict only pixel-level information, our Predictive Generative Networks learn a representation of the latent structure of the underlying three-dimensional objects themselves. Importantly, we find that this representation is naturally tolerant to object transformations, and generalizes well to new tasks, such as classification of static images. Similar models trained solely with a reconstruction loss fail to generalize as effectively. We argue that prediction can serve as a powerful unsupervised loss for learning rich internal representations of high-level object features.

cs.LG↗

arXiv2016

Microdatabases for the Industrial Internet

The Industrial Internet market is targeted to grow by trillions of US dollars by the year 2030, driven by adoption, deployment and integration of billions of intelligent devices and their associated data. This digital expansion faces a number of significant challenges, including reliable data management, security and privacy. Realizing the benefits from this evolution is made more difficult because a typical industrial plant includes multiple vendors and legacy technology stacks. Aggregating all the raw data to a single data center before performing analysis increases response times, raising performance concerns in traditional markets and requiring a compromise between data duplication and data access performance. Similar to the way microservices can integrate disparate information technologies without imposing monolithic cross-cutting architecture impacts, we propose microdatabases to manage the data heterogeneity of the Industrial Internet while allowing records to be captured and secured close to the industrial processes, but also be made available near the applications that can benefit from the data. A microdatabase is an abstraction of a data store that standardizes and protects the interactions between distributed data sources, providers and consumers. It integrates an information model with discoverable object types that can be browsed interactively and programmatically, and supports repository instances that evolve with their own lifecycles. The microdatabase abstraction is independent of technology choice and was designed based on solicitation and review of industry stakeholder concerns.

cs.DB↗

arXiv2012

A study of singularities on rational curves via syzygies

Consider a rational projective curve C of degree d over an algebraically closed field k. There are n homogeneous forms g_1,...,g_n of degree d in B=k[x,y] which parameterize C in a birational, base point free, manner. We study the singularities of C by studying a Hilbert-Burch matrix phi for the row vector [g_1,...,g_n]. In the "General Lemma" we use the generalized row ideals of phi to identify the singular points on C, their multiplicities, the number of branches at each singular point, and the multiplicity of each branch. Let p be a singular point on the parameterized planar curve C which corresponds to a generalized zero of phi. In the "Triple Lemma" we give a matrix phi' whose maximal minors parameterize the closure, in projective 2-space, of the blow-up at p of C in a neighborhood of p. We apply the General Lemma to phi' in order to learn about the singularities of C in the first neighborhood of p. If C has even degree d=2c and the multiplicity of C at p is equal to c, then we apply the Triple Lemma again to learn about the singularities of C in the second neighborhood of p. Consider rational plane curves C of even degree d=2c. We classify curves according to the configuration of multiplicity c singularities on or infinitely near C. There are 7 possible configurations of such singularities. We classify the Hilbert-Burch matrix which corresponds to each configuration. The study of multiplicity c singularities on, or infinitely near, a fixed rational plane curve C of degree 2c is equivalent to the study of the scheme of generalized zeros of the fixed balanced Hilbert-Burch matrix phi for a parameterization of C.

math.AC↗