arXiv subjects

Cabrel Teguemne Fokam

Publications and source records attributed to Cabrel Teguemne Fokam.

4 recordsLinked to original sources

arXiv2026

Breaking Chains with Trees: Model-Parallel Deep Learning with $\mathcal{O}(\log N)$ Time Complexity

Modern deep neural networks are trained using error backpropagation, which requires sequential forward and backward computations across network layers. As these networks become deeper, this introduces limitations, since layer-wise updates are strictly interdependent and cannot proceed in parallel. These constraints restrict training procedures to data-parallel schemes, thereby prohibiting model-parallel training. We propose TreeProp, an architecture-agnostic variational learning framework that organizes network layers into a tree-structured hierarchy. During training, TreeProp replaces sequential forward computations and backward gradient propagation with hierarchical computations. This allows intermediate representations and learning signals to be constructed in time complexity of $\mathcal{O}(\log N)$ for a network of $N$ layers. To the best of our knowledge, TreeProp is the first learning algorithm for deep neural networks with logarithmic parallel time complexity for both forward computation and backward gradient propagation during training. Furthermore, we show that multiple valid paths through the hierarchy exist, such that TreeProp implicitly learns subnetworks with different effective depths, but without additional training effort. We evaluate TreeProp on vision classification and autoregressive language modeling, matching the performance of conventional end-to-end training for a variety of tasks and outperforming previous contrastive training approaches. We further demonstrate the applicability of TreeProp to recurrent neural networks that otherwise rely on backpropagation through time.

cs.LG↗

arXiv2026

LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates

The increasing size of deep learning models has made distributed training across multiple devices essential. Synchronous, centralized methods incur large communication and synchronization overheads. Communication efficient algorithms can reduce these overheads, but often require extra buffers, remain sensitive to stragglers or parameter drift. We present LayUp, an asynchronous decentralized SGD method with layer-wise updates. LayUp asynchronously exchanges incremental layer-wise updates during backpropagation. It uses randomized gossip communication, enabling updates to be applied as soon as they are available without buffering. These design choices reduce parameter drift and improve robustness to stragglers. We establish a theoretical upper bound for the gradient bias introduced by layer-wise updates and prove convergence of LayUp. We empirically validate LayUp on vision and language modeling tasks, showing convergence up to ~32% faster in terms of wall-clock time compared to synchronous data parallel training and up to ~27% faster than comparable communication efficient algorithms while maintaining better task performance. This speed-up is partly due to higher model FLOPs utilization, as we demonstrate. By injecting delays into the communication between workers, we show that LayUp remains robust to stragglers while DDP and other methods degrade in performance. Overall, LayUp provides a novel practical, straggler-robust alternative for distributed training without sacrificing accuracy.

cs.LG↗

arXiv2024

AR-Sieve Bootstrap for the Random Forest and a simulation-based comparison with rangerts time series prediction

The Random Forest (RF) algorithm can be applied to a broad spectrum of problems, including time series prediction. However, neither the classical IID (Independent and Identically distributed) bootstrap nor block bootstrapping strategies (as implemented in rangerts) completely account for the nature of the Data Generating Process (DGP) while resampling the observations. We propose the combination of RF with a residual bootstrapping technique where we replace the IID bootstrap with the AR-Sieve Bootstrap (ARSB), which assumes the DGP to be an autoregressive process. To assess the new model's predictive performance, we conduct a simulation study using synthetic data generated from different types of DGPs. It turns out that ARSB provides more variation amongst the trees in the forest. Moreover, RF with ARSB shows greater accuracy compared to RF with other bootstrap strategies. However, these improvements are achieved at some efficiency costs.

stat.ML↗

arXiv2023

Block-local learning with probabilistic latent representations

The ubiquitous backpropagation algorithm requires sequential updates through the network introducing a locking problem. In addition, back-propagation relies on the transpose of forward weight matrices to compute updates, introducing a weight transport problem across the network. Locking and weight transport are problems because they prevent efficient parallelization and horizontal scaling of the training process. We propose a new method to address both these problems and scale up the training of large models. Our method works by dividing a deep neural network into blocks and introduces a feedback network that propagates the information from the targets backwards to provide auxiliary local losses. Forward and backward propagation can operate in parallel and with different sets of weights, addressing the problems of locking and weight transport. Our approach derives from a statistical interpretation of training that treats output activations of network blocks as parameters of probability distributions. The resulting learning framework uses these parameters to evaluate the agreement between forward and backward information. Error backpropagation is then performed locally within each block, leading to "block-local" learning. Several previously proposed alternatives to error backpropagation emerge as special cases of our model. We present results on a variety of tasks and architectures, demonstrating state-of-the-art performance using block-local learning. These results provide a new principled framework for training networks in a distributed setting.

cs.LG↗