Searcharxiv⌕ Search

arXiv subjects

Timothy Ngotiaoco

Publications and source records attributed to Timothy Ngotiaoco.

4 recordsLinked to original sources
arXiv2026

Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient training. Here we examine the bounds on error and how error scales when models operate outside of their expressivity regimes using a Lie-algebraic control perspective. Our theory formulates a correspondence between the depth of a sequence model and the tower of Lie algebra extensions. Echoing recent theoretical studies, we characterize the Lie-algebraic class of constant-depth sequence models and their corresponding expressivity bounds. Furthermore, we analytically derive an approximation error bound and show that error diminishes exponentially as the depth increases, consistent with the strong empirical performance of these models. We validate our theoretical predictions using experiments on symbolic word and continuous-valued state-tracking problems.

cs.LG↗
arXiv2025

Any-Order Flexible Length Masked Diffusion

Masked diffusion models (MDMs) have recently emerged as a promising alternative to autoregressive models over discrete domains. MDMs generate sequences in an any-order, parallel fashion, enabling fast inference and strong performance on non-causal tasks. However, a crucial limitation is that they do not support token insertions and are thus limited to fixed-length generations. To this end, we introduce Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion paradigm that simultaneously can model sequences of flexible length while provably retaining MDMs' flexibility of any-order inference. Grounded in an extension of the stochastic interpolant framework, FlexMDMs generate sequences by inserting mask tokens and unmasking them. Empirically, we show that FlexMDMs match MDMs in perplexity while modeling length statistics with much higher fidelity. On a synthetic maze planning task, they achieve $\approx 60 \%$ higher success rate than MDM baselines. Finally, we show pretrained MDMs can easily be retrofitted into FlexMDMs: on 16 H100s, it takes only three days to fine-tune LLaDA-8B into a FlexMDM, achieving superior performance on math (GSM8K, $58\% \to 67\%$) and code infilling performance ($52\% \to 65\%$).

cs.LG↗
arXiv2017

Compositionality of the Runge-Kutta Method

In Spivak's work, dynamical systems are described in terms of their inputs and outputs in a pictorial way using an operad of wiring diagrams. Each dynamical system is a box with certain inputs and outputs, and multiple dynamical systems are linked together using wiring diagrams, which describe how the outputs of one dynamical system to the inputs of another. By describing dynamical systems in this way, we can decompose a large dynamical system as a collection of smaller, simpler dynamical systems linked together. Of course, this decomposition is only useful if we can work with these smaller, simpler dynamical systems instead of the larger one. In his paper, Spivak shows that we can perform Euler's method on these smaller systems and still get the same results as working on the larger one. In this paper, we extend his results to prove that we can do something similar with the Runge-Kutta method. However, we need to modify the framework used in Spivak's paper to account for the fact that the Runge-Kutta method requires multiple steps, unlike Euler's method. To better describe these systems, we define wiring diagrams as objects of a double-category and dynamical systems in terms of double functors, giving a categorical description of this approximation method.

math.CT↗