arXiv subjects

Hongjian Jiang

Publications and source records attributed to Hongjian Jiang.

8 recordsLinked to original sources

arXiv2026

Length Generalization for Transformers via Compression

Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particular, the C-RASP hypothesis (a formalized version of the so-called RASP-l conjecture) posits that transformers length-generalize on a task if and only if a solution is expressible in the C-RASP language. While this hypothesis has strong empirical validation, theoretical problems arise from the fact that no computable length generalization bounds exist for C-RASP, alongside the discovery of seemingly contradictory experiments. To address these problems, we refine the C-RASP hypothesis utilizing the recently-proposed fragments C-RASP+ and C-RASP1. These fragments have computable length generalization bounds, though in the worst case requiring an extremely large (double exponential) sample size. It is an open question whether these sample size bounds are tight. In this paper, we resolve this open question by providing an exponentially tighter bound. In doing so, we show a polynomial length generalization bound for transformers if we adopt compressed strings, via a novel connection to power words. As an application, we show how this yields a fine-grained analysis of the C-RASP conjecture that resolves contradicting experimental evidence against it.

cs.LG↗

arXiv2026

Why Are Linear RNNs More Parallelizable?

The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressivity benefits of LRNNs over transformers, it is unclear what makes LRNNs -- but not traditional, nonlinear RNNs -- as easy to parallelize in practice as transformers. We answer this question by providing a tight connection between types of RNNs and standard complexity classes. We show that LRNNs can be viewed as log-depth (bounded fan-in) arithmetic circuits, which represents only a slight depth overhead relative to log-depth boolean circuits that transformers admit. Furthermore, we show that nonlinear RNNs can solve $\mathsf{L}$-complete problems (and even $\mathsf{P}$-complete ones, under polynomial precision), revealing a fundamental barrier to parallelizing them as efficiently as transformers. Our theory also identifies fine-grained expressivity differences between recent popular LRNN variants: permutation-diagonal LRNNs are $\mathsf{NC}^1$-complete whereas diagonal-plus-low-rank LRNNs are more expressive ($\mathsf{PNC}^1$-complete). We provide further insight by associating each type of RNN with a corresponding automata-theoretic model that it can simulate. Together, our results reveal fundamental tradeoffs between nonlinear RNNs and different variants of LRNNs, providing a foundation for designing LLM architectures that achieve an optimal balance between expressivity and parallelism.

cs.LG↗

arXiv2026

Synthesis and Verification of Transformer Programs (Technical Report)

C-RASP is a simple programming language that was recently shown to capture concepts expressible by transformers. In this paper, we develop new algorithmic techniques for automatically verifying C-RASPs. To this end, we establish a connection to the verification of synchronous dataflow programs in Lustre, which enables us to exploit state-of-the-art model checkers utilizing highly optimized SMT-solvers. Our second contribution addresses learning a C-RASP program in the first place. To this end, we provide a new algorithm for learning a C-RASP from examples using local search. We demonstrate efficacy of our implementation for benchmarks of C-RASPs in the literature, in particular in connection to the following applications: (1) transformer program optimization, and (2) constrained learning of transformer programs (based on a partial specification).

cs.LG↗

arXiv2025

Polymer-inspired mechanical metamaterials

Metamaterials benefit from unique architected patterns to achieve lightweight with exceptional mechanical properties inaccessible to conventional materials. Typical mechanical metamaterials are inspired by crystal-like lattice structures, whose closely packed frameworks often exhibit a rigid mechanical nature. Here, we present polymer-inspired metamaterials (PIMs) by programming deformation and strengthening mechanisms that mimic the mechanical roles of key constituent elements in polymer networks. By combining metamaterial programmability with polymer-inspired structures, we design crosslinking, proto-crystalline order, and entanglement in PIMs to enable macroscale strengthening mechanisms inspired by crosslink, molecular-density, and pre-stretch strengthening in polymers, expanding the metamaterial structure-property design space. This macroscale polymer-inspired programmability also suggests that PIMs could serve as a design platform incorporating the programmability strategies to achieve desired deformation and strengthening responses, holding a potential for applications in soft robotic joints and compliant connectors.

physics.app-ph↗

arXiv2025

Softmax Transformers are Turing-Complete

Hard attention Chain-of-Thought (CoT) transformers are known to be Turing-complete. However, it is an open problem whether softmax attention Chain-of-Thought (CoT) transformers are Turing-complete. In this paper, we prove a stronger result that length-generalizable softmax CoT transformers are Turing-complete. More precisely, our Turing-completeness proof goes via the CoT extension of the Counting RASP (C-RASP), which correspond to softmax CoT transformers that admit length generalization. We prove Turing-completeness for CoT C-RASP with causal masking over a unary alphabet (more generally, for letter-bounded languages). While we show this is not Turing-complete for arbitrary languages, we prove that its extension with relative positional encoding is Turing-complete for arbitrary languages. We empirically validate our theory by training transformers for languages requiring complex (non-linear) arithmetic reasoning.

cs.FL↗

arXiv2025

Extracting Robust Register Automata from Neural Networks over Data Sequences

Automata extraction is a method for synthesising interpretable surrogates for black-box neural models that can be analysed symbolically. Existing techniques assume a finite input alphabet, and thus are not directly applicable to data sequences drawn from continuous domains. We address this challenge with deterministic register automata (DRAs), which extend finite automata with registers that store and compare numeric values. Our main contribution is a framework for robust DRA extraction from black-box models: we develop a polynomial-time robustness checker for DRAs with a fixed number of registers, and combine it with passive and active automata learning algorithms. This combination yields surrogate DRAs with statistical robustness and equivalence guarantees. As a key application, we use the extracted automata to assess the robustness of neural networks: for a given sequence and distance metric, the DRA either certifies local robustness or produces a concrete counterexample. Experiments on recurrent neural networks and transformer architectures show that our framework reliably learns accurate automata and enables principled robustness evaluation. Overall, our results demonstrate that robust DRA extraction effectively bridges neural network interpretability and formal reasoning without requiring white-box access to the underlying network.

cs.AI↗

arXiv2025

HornStr: Invariant Synthesis for Regular Model Checking as Constrained Horn Clauses(Technical Report)

We present HornStr, the first solver for invariant synthesis for Regular Model Checking (RMC) with the specification provided in the SMT-LIB 2.6 theory of strings. It is well-known that invariant synthesis for RMC subsumes various important verification problems, including safety verification for parameterized systems. To achieve a simple and standardized file format, we treat the invariant synthesis problem as a problem of solving Constrained Horn Clauses (CHCs) over strings. Two strategies for synthesizing invariants in terms of regular constraints are supported: (1) L* automata learning, and (2) SAT-based automata learning. HornStr implements these strategies with the help of existing SMT solvers for strings, which are interfaced through SMT-LIB. HornStr provides an easy-to-use interface for string solver developers to apply their techniques to verification. At the same time, it allows verification researchers to painlessly tap into the wealth of modern string solving techniques. To assess the effectiveness of HornStr, we conducted a comprehensive evaluation using benchmarks derived from applications including parameterized verification and string rewriting tasks. Our experiments highlight HornStr's capacity to effectively handle these benchmarks, e.g., as the first solver to verify the challenging MU puzzle automatically. Finally, HornStr can be used to automatically generate a new class of interesting SMT-LIB 2.6 string constraint benchmarks, which might in the future be used in the SMT-COMP strings track. In particular, our experiments on the above invariant synthesis benchmarks produce more than 30000 new QF_S constraints. We also detail the performance of various integrated string solvers, providing insights into their effectiveness on our new benchmarks.

cs.LO↗

arXiv2018

Nonlinear effect of forced harmonic oscillator subject to sliding friction and simulation by a simple nonlinear circuit

We study the nonlinear behaviors of mass-spring systems damped by dry friction using simulation by a nonlinear LC circuit damped by anti-parallel diodes. We show that the differential equation for the electric oscillator is equivalent to the mechanical one's when a piecewise linear model is used to simplify the diodes I-V curve. We derive series solutions to the differential equation under weak nonlinear approximation which can describe the resonant response as well as amplitudes of superharmonic components. Experimental results are consistent with series solutions. We also present the phenomenon of hysteresis. Theoretical analysis along with numerical simulations are conducted to explore the stick-slip boundary. The correspondence between the mechanical and electric oscillators makes it easy to demonstrate the behaviors of this nonlinear oscillator on a digital oscilloscope. It can be used to extend the linear RLC experiment at the undergraduate level.

physics.class-ph↗