arXiv subjects

Bob Pepin

Publications and source records attributed to Bob Pepin.

8 recordsLinked to original sources

arXiv2025

CoDeQ: End-to-End Joint Model Compression with Dead-Zone Quantizer for High-Sparsity and Low-Precision Networks

While joint pruning--quantization is theoretically superior to sequential application, current joint methods rely on auxiliary procedures outside the training loop for finding compression parameters. This reliance adds engineering complexity and hyperparameter tuning, while also lacking a direct data-driven gradient signal, which might result in sub-optimal compression. In this paper, we introduce CoDeQ, a simple, fully differentiable method for joint pruning--quantization. Our approach builds on a key observation: the dead-zone of a scalar quantizer is equivalent to magnitude pruning, and can be used to induce sparsity directly within the quantization operator. Concretely, we parameterize the dead-zone width and learn it via backpropagation, alongside the quantization parameters. This design provides explicit control of sparsity, regularized by a single global hyperparameter, while decoupling sparsity selection from bit-width selection. The result is a method for Compression with Dead-zone Quantizer (CoDeQ) that supports both fixed-precision and mixed-precision quantization (controlled by an optional second hyperparameter). It simultaneously determines the sparsity pattern and quantization parameters in a single end-to-end optimization. Consequently, CoDeQ does not require any auxiliary procedures, making the method architecture-agnostic and straightforward to implement. On ImageNet with ResNet-18, CoDeQ reduces bit operations to ~5% while maintaining close to full precision accuracy in both fixed and mixed-precision regimes.

cs.LG↗

arXiv2025

Oscillations Make Neural Networks Robust to Quantization

We challenge the prevailing view that weight oscillations observed during Quantization Aware Training (QAT) are merely undesirable side-effects and argue instead that they are an essential part of QAT. We show in a univariate linear model that QAT results in an additional loss term that causes oscillations by pushing weights away from their nearest quantization level. Based on the mechanism from the analysis, we then derive a regularizer that induces oscillations in the weights of neural networks during training. Our empirical results on ResNet-18 and Tiny Vision Transformer, evaluated on CIFAR-10 and Tiny ImageNet datasets, demonstrate across a range of quantization levels that training with oscillations followed by post-training quantization (PTQ) is sufficient to recover the performance of QAT in most cases. With this work we provide further insight into the dynamics of QAT and contribute a novel insight into explaining the role of oscillations in QAT which until now have been considered to have a primarily negative effect on quantization.

cs.LG↗

arXiv2024

When Can Memorization Improve Fairness?

We study to which extent additive fairness metrics (statistical parity, equal opportunity and equalized odds) can be influenced in a multi-class classification problem by memorizing a subset of the population. We give explicit expressions for the bias resulting from memorization in terms of the label and group membership distribution of the memorized dataset and the classifier bias on the unmemorized dataset. We also characterize the memorized datasets that eliminate the bias for all three metrics considered. Finally we provide upper and lower bounds on the total probability mass in the memorized dataset that is necessary for the complete elimination of these biases.

cs.LG↗

arXiv2024

PePR: Performance Per Resource Unit as a Metric to Promote Small-Scale Deep Learning in Medical Image Analysis

The recent advances in deep learning (DL) have been accelerated by access to large-scale data and compute. These large-scale resources have been used to train progressively larger models which are resource intensive in terms of compute, data, energy, and carbon emissions. These costs are becoming a new type of entry barrier to researchers and practitioners with limited access to resources at such scale, particularly in the Global South. In this work, we take a comprehensive look at the landscape of existing DL models for medical image analysis tasks and demonstrate their usefulness in settings where resources are limited. To account for the resource consumption of DL models, we introduce a novel measure to estimate the performance per resource unit, which we call the PePR score. Using a diverse family of 131 unique DL architectures (spanning 1M to 130M trainable parameters) and three medical image datasets, we capture trends about the performance-resource trade-offs. In applications like medical image analysis, we argue that small-scale, specialized models are better than striving for large-scale models. Furthermore, we show that using existing pretrained models that are fine-tuned on new data can significantly reduce the computational resources and data required compared to training models from scratch. We hope this work will encourage the community to focus on improving AI equity by developing methods and models with smaller resource footprints.

cs.LG↗

arXiv2020

The Harmonic Indel Distance

This short note introduces the harmonic indel distance (HID), a new distance between strings where the cost of an insertion or deletion is inversely proportional to the string length. We present a closed-form formula and show that the HID is a proper distance metric. Then we perform an experimental comparison of HID to normalized and unnormalized versions of the indel distance on benchmark tasks for biomedical sequence data. We finally show planar embeddings of the benchmark datasets to provide some insights into the geometry of the metric spaces associated with the different distance metrics.

cs.DM↗

arXiv2020

Concentration Inequalities for Additive Functionals: a Martingale Approach

This work shows how exponential concentration inequalities for additive functionals of stochastic processes over a finite time interval can be derived from concentration inequalities for martingales. The approach is entirely probabilistic and naturally includes time-inhomogeneous and non-stationary processes as well as initial laws concentrated on a single point. The class of processes studied includes martingales, Markov processes and general square integrable processes. The general approach is complemented by a simple and direct method for martingales, diffusions and discrete-time Markov processes. The method is illustrated by deriving concentration inequalities for the Polyak-Ruppert algorithm, SDEs with time-dependent drift coefficients "contractive at infinity" with both Lipschitz and squared Lipschitz observables, some classical martingales and non-elliptic SDEs.

math.PR↗

arXiv2018

Time Averages of Markov Processes and Applications to Two-Timescale Problems

We show a decomposition into the sum of a martingale and a deterministic quantity for time averages of the solutions to non-autonomous SDEs and for discrete-time Markov processes. In the SDE case the martingale has an explicit representation in terms of the gradient of the associated semigroup or transition operator. We show how the results can be used to obtain quenched Gaussian concentration inequalities for time averages and to provide insights into the Averaging principle for two-timescale processes.

math.PR↗

arXiv2017

Towards a Quantitative Averaging Principle for Stochastic Differential Equations

This work explores the use of a forward-backward martingale method together with a decoupling argument and entropic estimates between the conditional and averaged measures to prove a strong averaging principle for stochastic differential equations with order of convergence 1/2. We obtain explicit expressions for all the constants involved. At the price of some extra assumptions on the time marginals and an exponential bound in time, we loosen the usual boundedness and Lipschitz assumptions. We conclude with an application of our result to Temperature-Accelerated Molecular Dynamics.

math.PR↗