arXiv subjects

Mark D. McDonnell

Publications and source records attributed to Mark D. McDonnell.

At least 19 recordsLinked to original sources

arXiv2024

Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning

Continual learning requires a model to adapt to ongoing changes in the data distribution, and often to the set of tasks to be performed. It is rare, however, that the data and task changes are completely unpredictable. Given a description of an overarching goal or data theme, which we call a realm, humans can often guess what concepts are associated with it. We show here that the combination of a large language model and an image generation model can similarly provide useful premonitions as to how a continual learning challenge might develop over time. We use the large language model to generate text descriptions of semantically related classes that might potentially appear in the data stream in future. These descriptions are then rendered using Stable Diffusion to generate new labelled image samples. The resulting synthetic dataset is employed for supervised pre-training, but is discarded prior to commencing continual learning, along with the pre-training classification head. We find that the backbone of our pre-trained networks can learn representations useful for the downstream continual learning problem, thus becoming a valuable input to any existing continual learning method. Although there are complexities arising from the domain gap between real and synthetic images, we show that pre-training models in this manner improves multiple Class Incremenal Learning (CIL) methods on fine-grained image classification benchmarks. Supporting code can be found at https://github.com/cl-premonition/premonition.

cs.CV↗

arXiv2024

RanPAC: Random Projections and Pre-trained Models for Continual Learning

Continual learning (CL) aims to incrementally learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. Most CL works focus on tackling catastrophic forgetting under a learning-from-scratch paradigm. However, with the increasing prominence of foundation models, pre-trained models equipped with informative representations have become available for various downstream requirements. Several CL methods based on pre-trained models have been explored, either utilizing pre-extracted features directly (which makes bridging distribution gaps challenging) or incorporating adaptors (which may be subject to forgetting). In this paper, we propose a concise and effective approach for CL with pre-trained models. Given that forgetting occurs during parameter updating, we contemplate an alternative approach that exploits training-free random projectors and class-prototype accumulation, which thus bypasses the issue. Specifically, we inject a frozen Random Projection layer with nonlinear activation between the pre-trained model's feature representations and output head, which captures interactions between features with expanded dimensionality, providing enhanced linear separability for class-prototype-based CL. We also demonstrate the importance of decorrelating the class-prototypes to reduce the distribution disparity when using pre-trained representations. These techniques prove to be effective and circumvent the problem of forgetting for both class- and domain-incremental continual learning. Compared to previous methods applied to pre-trained ViT-B/16 models, we reduce final error rates by between 20% and 62% on seven class-incremental benchmarks, despite not using any rehearsal memory. We conclude that the full potential of pre-trained models for simple, effective, and fast CL has not hitherto been fully tapped. Code is at github.com/RanPAC/RanPAC.

cs.LG↗

arXiv2022

Planet Four: A Neural Network's Search For Polar Spring-time Fans On Mars

Dark deposits visible from orbit appear in the Martian south polar region during the springtime. These are thought to form from explosive jets of carbon dioxide gas breaking through the thawing seasonal ice cap, carrying dust and dirt which is then deposited onto the ice as dark 'blotches', or blown by the surface winds into streaks or 'fans'. We investigate machine learning (ML) methods for automatically identifying these seasonal features in High Resolution Imaging Science Experiment (HiRISE) satellite imagery. We designed deep Convolutional Neural Networks (CNNs) that were trained and tested using the catalog generated by Planet Four, an online citizen science project mapping the south polar seasonal deposits. We validated the CNNs by comparing their results with those of ISODATA (Iterative Self-Organizing Data Analysis Technique) clustering and as expected, the CNNs were significantly better at predicting the results found by Planet Four, in both the area of predicted seasonal deposits and in delineating their boundaries. We found neither the CNNs or ISODATA were suited to predicting the source point and directions of seasonal fans, which is a strength of the citizen science approach. The CNNs showed good agreement with Planet Four in cross-validation metrics and detected some seasonal deposits in the HiRISE images missed in the Planet Four catalog; the total area of seasonal deposits predicted by the CNNs was 27% larger than that of the Planet Four catalog, but this aspect varied considerably on a per-image basis.

astro-ph.EP↗

arXiv2019

Single-bit-per-weight deep convolutional neural networks without batch-normalization layers for embedded systems

Batch-normalization (BN) layers are thought to be an integrally important layer type in today's state-of-the-art deep convolutional neural networks for computer vision tasks such as classification and detection. However, BN layers introduce complexity and computational overheads that are highly undesirable for training and/or inference on low-power custom hardware implementations of real-time embedded vision systems such as UAVs, robots and Internet of Things (IoT) devices. They are also problematic when batch sizes need to be very small during training, and innovations such as residual connections introduced more recently than BN layers could potentially have lessened their impact. In this paper we aim to quantify the benefits BN layers offer in image classification networks, in comparison with alternative choices. In particular, we study networks that use shifted-ReLU layers instead of BN layers. We found, following experiments with wide residual networks applied to the ImageNet, CIFAR 10 and CIFAR 100 image classification datasets, that BN layers do not consistently offer a significant advantage. We found that the accuracy margin offered by BN layers depends on the data set, the network size, and the bit-depth of weights. We conclude that in situations where BN layers are undesirable due to speed, memory or complexity costs, that using shifted-ReLU layers instead should be considered; we found they can offer advantages in all these areas, and often do not impose a significant accuracy cost.

cs.LG↗

arXiv2018

Diagnosing Convolutional Neural Networks using their Spectral Response

Convolutional Neural Networks (CNNs) are a class of artificial neural networks whose computational blocks use convolution, together with other linear and non-linear operations, to perform classification or regression. This paper explores the spectral response of CNNs and its potential use in diagnosing problems with their training. We measure the gain of CNNs trained for image classification on ImageNet and observe that the best models are also the most sensitive to perturbations of their input. Further, we perform experiments on MNIST and CIFAR-10 to find that the gain rises as the network learns and then saturates as the network converges. Moreover, we find that strong gain fluctuations can point to overfitting and learning problems caused by a poor choice of learning rate. We argue that the gain of CNNs can act as a diagnostic tool and potential replacement for the validation loss when hold-out validation data are not available.

cs.CV↗

arXiv2018

Training wide residual networks for deployment using a single bit for each weight

For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit. Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight. Using wide residual networks as our main baseline, our approach simplifies existing methods that binarize weights by applying the sign function in training; we apply scaling factors for each layer with constant unlearned values equal to the layer-specific standard deviations used for initialization. For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bit-per-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively. We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1% respectively. For CIFAR, our error rates halve previously reported values, and are within about 1% of our error-rates for the same network with full-precision weights. For networks that overfit, we also show significant improvements in error rate by not learning batch normalization scale and offset parameters. This applies to both full precision and 1-bit-per-weight networks. Using a warm-restart learning-rate schedule, we found that training for 1-bit-per-weight is just as fast as full-precision networks, with better accuracy than standard schedules, and achieved about 98%-99% of peak performance in just 62 training epochs for CIFAR-10/100. For full training code and trained models in MATLAB, Keras and PyTorch see https://github.com/McDonnell-Lab/1-bit-per-weight/ .

cs.LG↗

arXiv2017

Track Everything: Limiting Prior Knowledge in Online Multi-Object Recognition

This paper addresses the problem of online tracking and classification of multiple objects in an image sequence. Our proposed solution is to first track all objects in the scene without relying on object-specific prior knowledge, which in other systems can take the form of hand-crafted features or user-based track initialization. We then classify the tracked objects with a fast-learning image classifier that is based on a shallow convolutional neural network architecture and demonstrate that object recognition improves when this is combined with object state information from the tracking algorithm. We argue that by transferring the use of prior knowledge from the detection and tracking stages to the classification stage we can design a robust, general purpose object recognition system with the ability to detect and track a variety of object types. We describe our biologically inspired implementation, which adaptively learns the shape and motion of tracked objects, and apply it to the Neovision2 Tower benchmark data set, which contains multiple object types. An experimental evaluation demonstrates that our approach is competitive with state-of-the-art video object recognition systems that do make use of object-specific prior knowledge in detection and tracking, while providing additional practical advantages by virtue of its generality.

cs.CV↗

arXiv2016

Understanding data augmentation for classification: when to warp?

In this paper we investigate the benefit of augmenting data with synthetically created samples when training a machine learning classifier. Two approaches for creating additional training samples are data warping, which generates additional samples through transformations applied in the data-space, and synthetic over-sampling, which creates additional samples in feature-space. We experimentally evaluate the benefits of data augmentation for a convolutional backpropagation-trained neural network, a convolutional support vector machine and a convolutional extreme learning machine classifier, using the standard MNIST handwritten digit dataset. We found that while it is possible to perform generic augmentation in feature-space, if plausible transforms for the data are known then augmentation in data-space provides a greater benefit for improving performance and reducing overfitting.

cs.CV↗

arXiv2015

Enhanced Image Classification With a Fast-Learning Shallow Convolutional Neural Network

We present a neural network architecture and training method designed to enable very rapid training and low implementation complexity. Due to its training speed and very few tunable parameters, the method has strong potential for applications requiring frequent retraining or online training. The approach is characterized by (a) convolutional filters based on biologically inspired visual processing filters, (b) randomly-valued classifier-stage input weights, (c) use of least squares regression to train the classifier output weights in a single batch, and (d) linear classifier-stage output units. We demonstrate the efficacy of the method by applying it to image classification. Our results match existing state-of-the-art results on the MNIST (0.37% error) and NORB-small (2.2% error) image classification databases, but with very fast training times compared to standard deep network approaches. The network's performance on the Google Street View House Number (SVHN) (4% error) database is also competitive with state-of-the art methods.

cs.NE↗

arXiv2015

Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the 'extreme learning machine' algorithm

Recent advances in training deep (multi-layer) architectures have inspired a renaissance in neural network use. For example, deep convolutional networks are becoming the default option for difficult tasks on large datasets, such as image and speech recognition. However, here we show that error rates below 1% on the MNIST handwritten digit benchmark can be replicated with shallow non-convolutional neural networks. This is achieved by training such networks using the 'Extreme Learning Machine' (ELM) approach, which also enables a very rapid training time (~10 minutes). Adding distortions, as is common practise for MNIST, reduces error rates even further. Our methods are also shown to be capable of achieving less than 5.5% error rates on the NORB image database. To achieve these results, we introduce several enhancements to the standard ELM algorithm, which individually and in combination can significantly improve performance. The main innovation is to ensure each hidden-unit operates only on a randomly sized and positioned patch of each image. This form of random `receptive field' sampling of the input ensures the input weight matrix is sparse, with about 90% of weights equal to zero. Furthermore, combining our methods with a small number of iterations of a single-batch backpropagation method can significantly reduce the number of hidden-units required to achieve a particular performance. Our close to state-of-the-art results for MNIST and NORB suggest that the ease of use and accuracy of the ELM algorithm for designing a single-hidden-layer neural network classifier should cause it to be given greater consideration either as a standalone method for simpler problems, or as the final classification stage in deep neural networks applied to more difficult problems.

cs.NE↗

arXiv2014

Distance Distributions for Real Cellular Networks

This paper presents the general distribution for the distance between a mobile user and any base station (BS). We show that a random variable proportional to the distance squared is Gamma distributed. In the case of the nearest BS, it can be reduced to the well established result of the distance being Rayleigh distributed. We validate our results using a random node simulation and real Vodafone 3G network data, and go on to show how the distribution is tractable by deriving the average aggregate interference power.

cs.NI↗

arXiv2014

Transmit Pulse Shaping for Molecular Communications

This paper presents a method for shaping the transmit pulse of a molecular signal such that the diffusion channel's response is a sharp pulse. The impulse response of a diffusion channel is typically characterised as having an infinitely long transient response. This can cause severe inter-symbol-interference, and reduce the achievable reliable bit rate. We achieve the desired chemical channel response by poisoning the channel with a secondary compound, such that it chemically cancels aspects of the primary information signal. We use two independent methods to show that the chemical concentration of the \emph{information signal} should be $\propto δ(t)$ and that of the \emph{poison signal} should be $\propto t^{-3/2}$.

cs.ET↗

arXiv2014

Downlink Interference Estimation without Feedback for Heterogeneous Network Interference Avoidance

In this paper, we present a novel method for a base station (BS) to estimate the total downlink interference power received by any given mobile receiver, without information feedback from the user or information exchange between neighbouring BSs. The prediction method is deterministic and can be computed rapidly. This is achieved by first abstracting the cellular network into a mathematical model, and then inferring the interference power received at any location based on the power spectrum measurements taken at the observing BS. The analysis expands the methodology to a $\mathsf{K}$-tier heterogeneous network and demonstrates the accuracy of the technique for a variety of sampling densities. The paper demonstrates the methodology by applying it to an opportunistic transmission technique that avoids transmissions to channels which are overwhelmed by interference. The simulation results show that the proposed technique performs closely or better than existing interference avoidance techniques that require information exchange, and yields a 30% throughput improvement over baseline configurations.

cs.NI↗

arXiv2014

Performance of Macro-Scale Molecular Communications with Sensor Cleanse Time

In this paper, we consider a molecular diffusion based communications link that conveys information on the macro-scale (several metres). The motivation is to apply molecular-based communications to challenging electromagnetic environments. We first derive a novel capture probability expression of a finite sized receiver. The paper then introduces the concept of time-aggregated molecular noise at the receiver as a function of the rate at which the sensor can self-cleanse. The resulting inter-symbol-interference is expressed as a function of the sensor cleanse time, and the performance metrics of bit error rate, throughput and round-trip-time are derived. The results show that the performance is very sensitive to the sensor cleanse time and the drift velocity. The paper concludes with recommendations on the design of a real communication link based on these findings and applies the concepts to a test-bed.

cs.NI↗

arXiv2013

Channel noise induced stochastic facilitation in an auditory brainstem neuron model

Neuronal membrane potentials fluctuate stochastically due to conductance changes caused by random transitions between the open and close states of ion channels. Although it has previously been shown that channel noise can nontrivially affect neuronal dynamics, it is unknown whether ion-channel noise is strong enough to act as a noise source for hypothesised noise-enhanced information processing in real neuronal systems, i.e. 'stochastic facilitation.' Here, we demonstrate that biophysical models of channel noise can give rise to two kinds of recently discovered stochastic facilitation effects in a Hodgkin-Huxley-like model of auditory brainstem neurons. The first, known as slope-based stochastic resonance (SBSR), enables phasic neurons to emit action potentials that can encode the slope of inputs that vary slowly relative to key time-constants in the model. The second, known as inverse stochastic resonance (ISR), occurs in tonically firing neurons when small levels of noise inhibit tonic firing and replace it with burst-like dynamics. Consistent with previous work, we conclude that channel noise can provide significant variability in firing dynamics, even for large numbers of channels. Moreover, our results show that possible associated computational benefits may occur due to channel noise in neurons of the auditory brainstem. This holds whether the firing dynamics in the model are phasic (SBSR can occur due to channel noise) or tonic (ISR can occur due to channel noise).

q-bio.NC↗

arXiv2011

An Introductory Review of Information Theory in the Context of Computational Neuroscience

This paper introduces several fundamental concepts in information theory from the perspective of their origins in engineering. Understanding such concepts is important in neuroscience for two reasons. Simply applying formulae from information theory without understanding the assumptions behind their definitions can lead to erroneous results and conclusions. Furthermore, this century will see a convergence of information theory and neuroscience; information theory will expand its foundations to incorporate more comprehensively biological processes thereby helping reveal how neuronal networks achieve their remarkable information processing abilities.

cs.IT↗

arXiv2009

Signal acquisition via polarization modulation in single photon sources

A simple model system is introduced for demonstrating how a single photon source might be used to transduce classical analog information. The theoretical scheme results in measurements of analog source samples that are (i) quantized in the sense of analog-to-digital conversion and (ii) corrupted by random noise that is solely due to the quantum uncertainty in detecting the polarization state of each photon. This noise is unavoidable if more than one bit per sample is to be transmitted, and we show how it may be exploited in a manner inspired by suprathreshold stochastic resonance. The system is analyzed information theoretically, as it can be modeled as a noisy optical communication channel, although unlike classical Poisson channels, the detector's photon statistics are binomial. Previous results on binomial channels are adapted to demonstrate numerically that the classical information capacity, and thus the accuracy of the transduction, increases logarithmically with the square root of the number of photons, N. Although the capacity is shown to be reduced when an additional detector nonideality is present, the logarithmic increase with N remains.

quant-ph↗

arXiv2009

Neural Population Coding is Optimized by Discrete Tuning Curves

The sigmoidal tuning curve that maximizes the mutual information for a Poisson neuron, or population of Poisson neurons, is obtained. The optimal tuning curve is found to have a discrete structure that results in a quantization of the input signal. The number of quantization levels undergoes a hierarchy of phase transitions as the length of the coding window is varied. We postulate, using the mammalian auditory system as an example, that the presence of a subpopulation structure within a neural population is consistent with an optimal neural code.

q-bio.NC↗