Searcharxiv⌕ Search

arXiv subjects

Martin Takáč

Publications and source records attributed to Martin Takáč.

At least 19 recordsLinked to original sources
arXiv2026

Matching the Lower Bounds: Stochastic Contracting Cubic Newton and Its Optimal Acceleration

We study second-order methods for convex stochastic optimization, where gradients and Hessians are available only through stochastic estimates with variances $σ_1^2$ and $σ_2^2$, respectively. First, we propose the Stochastic Contracting Cubic Newton method. At each iteration, it minimizes a cubic model with additional quadratic regularization and then contracts the step toward the current point. After $T$ iterations, the method achieves the expected convergence rate $\mathcal{O}(σ_1/\sqrt{T}+σ_2/T+1/T^2)$. Building on this construction, we develop an accelerated variant achieving $\mathcal{O}(σ_1/\sqrt{T}+σ_2/T^2+1/T^{7/2})$, matching the known lower bounds of Agafonov et al. (2024) in all three terms.

math.OC↗
arXiv2026

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

Vision foundation models (VFMs) provide transferable visual representations, but their inference cost limits deployment in many settings. Fine-tuning specializes a VFM without shrinking it; distillation shrinks it, but through expensive teacher-guided training. Structured pruning offers a more direct route by extracting a compact subnetwork from the VFM. However, as compression gets aggressive, existing methods fail because they treat subnetwork selection and recovery as separate axes. Across layers, capacity must follow task sensitivity rather than spread uniformly. Within a layer, a removal cannot be judged by importance alone; its impact also depends on how well the lost contribution can be recovered from what remains. Selection and recovery must therefore be decided jointly. We introduce Carve VFM, which resolves this interdependence. Given a VFM, a calibration set, and a parameter budget, it (i) searches for task-specific capacity allocations across layers; (ii) guides the search via an adaptive recovery-aware fitness criterion; and (iii) reconstructs removed behavior through iterative closed-form low-rank corrections merged into the surviving weights. The result is a dense encoder produced in minutes on a single GPU without backbone retraining. Across twelve benchmarks and four models from DINOv3 and MetaCLIP-2, Carve VFM achieves the highest reported mean accuracy in seven of eight backbone-budget configurations among the evaluated dense structured-pruning methods. Further fine-tuning surpasses the strongest evaluated edge model (88.1% vs. 85.8%), and carved models cut compute and memory while delivering 1.3-2.0x higher GPU throughput than the uncompressed one. These results show that the compact model you need is inside the VFM; it only has to be carved out.

cs.CV↗
arXiv2026

LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

Interactive benchmarks for LLMs usually report a single metric, even though succeeding in them requires models to show several different skills. While those benchmarks are useful for measuring the overall capability of models to perform the downstream task, it is harder to identify which skill was lacking and caused failure solely from the aggregate score. For example, in benchmarks where models are supposed to navigate a simulated world to perform a task, failures can come from perception, an ambiguous instruction, retrieval, missing commonsense about what actions do, unknown dynamics, or planning. \benchmark recasts the procedurally generated BabyAI gridworld as a fully observable, purely textual environment designed to evaluate planning separately from those factors. The nature of the environment and the structure of the prompt control for the other factors cited above. To evaluate the models, we define the PPD suite: \textbf{Predict} asks for the state that follows an action sequence, \textbf{Plan} asks for an action sequence that reaches a goal, and \textbf{Decompose} asks for a subgoal sequence that achieves a mission. Across seven frontier and open models, queried zero-shot, we find simulation far ahead of planning for every model, close to saturated for the strongest. The length of the required solution governs planning failure, and each of the four strongest models has a horizon beyond which single-attempt success collapses. The responses of claude-opus-4-6 and claude-sonnet-4-6, which show their working step by step, reveal that they plan by proposing L-shaped corridors, which is consistent with their successful plans staying near-optimal while success falls. At the subgoal level, decomposition precision likewise falls to zero on long missions, while comprehension of those missions persists.

cs.AI↗
arXiv2026

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every $P$ iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per $P$ steps, and the optimizer state is half of AdamW's. A single-EMA variant, SignMuon, already improves on Muon. We prove complexity bounds under heavy-tailed noise in which the period sets an interpolation between Muon's and Lion's smoothness and noise constants, and which say when LionMuon is faster than both. On 124M and 355M models trained on FineWeb, LionMuon with $P=2$ and $P=5$ reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training it reaches Muon's final loss with a third less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon

cs.LG↗
arXiv2026

Chebyshev-Exact Acceleration under Hessian Variation, II: Computer-Assisted Variational Lower Bound

We study two-step recurrences with positive step sizes and nonnegative momentum coefficients whose terminal residual is the Chebyshev minimax polynomial on a prescribed spectral interval $[μ,L]$. These recurrences have the same terminal error on a fixed quadratic, but their first-order responses to stepwise Hessian perturbations can differ. We prove that the sine-Jacobi recurrence of Part I is asymptotically optimal for this response norm within a factor of $1.000004$, uniformly over $0<μ 1.14268353624$, whereas the sine-Jacobi recurrence attains the constant $2\sqrt{c_{\sin}}$ with $c_{\sin}<1.14269218$. The proof reduces the gain to the diagonal energy of a Jacobi Green matrix. A centering lift represents this matrix by an exponential kernel; packing its diagonal into a density on the line preserves the energy and controls the change in the Birman-Schwinger spectrum. A calibrated identity gives a lower bound using the ground parameter and one Fredholm determinant. Two further determinants and a trace observation improve the bound through a pointwise inequality on an explicit compact state region. One rational certificate verifies this inequality by interval arithmetic. A second certificate constructs a density satisfying the selected observations and both normalizations with energy below the sine value. Finally, no $C^1$ function of finitely many observations of the specified kinds gives a locally sharp lower bound at the sine density.

math.OC↗
arXiv2026

Chebyshev-Exact Acceleration under Hessian Variation, I: Sine-Jacobi Method

We study finite-horizon one-gradient realizations with the Chebyshev minimax terminal residual on $[μ,L]$. Under time-dependent Hessian perturbations, the terminal first variation is governed by a time-ordered spectral kernel $K_s(λ,ν)$; its sharp $\ell_2$ gain is $A_N$. For the prefix-exact Chebyshev recurrence, $$A_N^{pref}=\frac4{\sqrt3}\frac{N^{3/2}}{L-μ}ε_N^\star(1+o(1)),$$ and this is sharp in the causal two-term class with Chebyshev exactness at every prefix. For terminal-only exactness, Jacobi coordinates give $P_N=2^{1-N}T_N$: the spectrum is fixed at the midpoint Chebyshev nodes, while the spectral weights parametrize the realizations. The sine weights give a final-exact Jacobi method with the same terminal residual and $$A_N(J_N^{\sin})=2\sqrt{c_{\sin}}\frac{N^{3/2}}{L-μ}ε_N^\star(1+o(1)), 2\sqrt{c_{\sin}}\approx2.1379<4/\sqrt3.$$ Thus the Chebyshev terminal polynomial does not determine the first-order Hessian-drift gain. Momentum is necessary for this order: every Chebyshev-exact stepsize schedule has $A_N\ge2\sqrt{2/3} N^2ε_N^\star/(L-μ)$. Beyond first order, the response to drift $E_s=H_s-H$ of size $δ=\max_s\|E_s\|$ is a finite polynomial expansion, and for both realizations the terminal error is at most $ε_N^\star T_N(1+2δ/(L-μ))$, the Chebyshev bound of the interval widened by $δ$. This bound is attained, holds for objectives with Lipschitz Hessian, and shows that drift reduces the accelerated rate exponent by the factor $1-\sqrt{δ/μ}$. The two realizations thus share the worst-case response and differ in the $\ell_2$-in-time response measured by $A_N$. The experiments show the finite-horizon effect: lower stochastic curvature overhead, larger admissible-block frontiers, accurate time-varying quadratic predictions, and lower restart cost on an endpoint-coupled smooth strongly convex GLM.

math.OC↗
arXiv2026

Information-Theoretic Upper Bounds for Deterministic Noise in Zeroth-Order Convex Optimization

We study zeroth-order convex optimization on Euclidean balls with function values corrupted by a fixed, uniformly bounded deterministic perturbation. For a query budget $T$, the maximum admissible level of noise (MALN) is the supremum of noise levels for which an $\eps$-accurate solution can be found with probability $1-β$. We construct an explicit hard family: the support function of a spherical belt combined with an affine branch along a hidden random direction. It yields finite-budget upper bounds on the MALN for Lipschitz, strongly convex, uniformly convex, smooth, Hölder, and smooth strongly convex classes. For Lipschitz objectives the bound has order $\eps^2\sqrt\ell/(\sqrt nMR)+\eps\ell/n$, where $\ell=\log(2(T+1)/(1-β))$ and $n-1\ge32\ell$; this is the scale of Li and Risteski with explicit logarithmic factors. A tangential-gradient estimate for the Moreau envelope yields the smooth scale $\eps^{3/2}/(\sqrt n\sqrt L\,R)$ and, for $L-μ\ge5μ$, the smooth strongly convex scale $\eps\sqrt{μ/((L-μ)n)}$. These bounds are tight: on explicit parameter ranges and subclasses, polynomial-query two-point methods tolerate noise of the order of the terms of order $n^{-1/2}$, so that the MALN is determined up to $O(\sqrt\ell)$ and absolute constants whenever these terms dominate. At $L=μ$, simplex interpolation with $n+1$ queries tolerates noise $R\sqrt{2μ\eps/n}$, which is optimal up to $O(\sqrt\ell)$ for $\eps\leμR^2/3$, while with at most $n$ exact queries no randomized algorithm guarantees success probability greater than $1/2$ uniformly over the class. We also quantify noise tolerance near this endpoint and illustrate the hiding mechanism and the breakdown of a two-point method numerically.

math.OC↗
arXiv2026

Dual Advantage Fields

Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state. We propose Dual Advantage Fields (DAF), a policy-extraction method that turns a bilinear dual value model into a local advantage signal. Under bilinear dual parameterization, the goal embedding is the gradient of the value field with respect to the state representation, which directly specifies which action should be preferred. DAF introduces an action-effect model that predicts the discounted direction of change in state representation space induced by actions and scores them by the similarity between this change and the goal direction. In the realizable case, this score equals the goal-conditioned Bellman advantage, yielding a standard local policy-improvement guarantee. On OGBench locomotion, manipulation, and puzzle tasks, DAF improves aggregate RLiable metrics and performs strongly in settings where locally correct actions differ from direct movement toward the final goal.

cs.LG↗
arXiv2026

Optimal Parameter-Free Second-order Acceleration with Inexact Hessians under Generalized Smoothness

We study second-order convex optimization under generalized Hessian smoothness, where the local Hessian Lipschitz constant may grow linearly with the gradient norm. For inexact Hessians, we develop adaptive algorithms, including an accelerated Monteiro-Svaiter-type method whose complexity matches the lower bound for second-order methods with $δ$-inexact Hessians in the classical Lipschitz-Hessian setting. Our methods are parameter-free in the sense that they require no problem-specific parameters as input, including the Hessian inexactness level $δ$ and smoothness constants. Instead, their backtracking adapts a single regularization parameter that jointly captures Hessian inexactness and local Hessian smoothness. The resulting rates split into two regimes. Far from the solution, the gradient-dependent smoothness term yields geometric convergence. Near the solution, the rate is governed by the ordinary Hessian smoothness and the inexactness level $δ$, recovering the classical Lipschitz-Hessian guarantees.

math.OC↗
arXiv2026

Optimizers for Diffusion Models: A Controlled Benchmark

Discrete diffusion models now match autoregressive language models on several benchmarks, while the question of how best to train them has received far less attention: the optimizer is inherited from one paper to the next and never compared. New optimizers, meanwhile, are validated almost exclusively on autoregressive pretraining, a different objective on a different loss surface. We present a controlled optimizer benchmark across four diffusion formulations, to our knowledge the first for discrete diffusion: seven optimizers (AdamW, Lion, Muon, SOAP, MARS, MARS-M, Schedule-Free) on masked diffusion (text8), uniform diffusion (QM9, and LM1B through the Gaussian duality) and Gaussian diffusion on images (CelebA-64), each on a task with published reference values. Every optimizer receives the same search protocol, and every winner is retrained at the full budget with three seeds. AdamW is a strong default but not always the right choice: it is beaten by a resolved margin on two of the four tasks, and the winner changes with the formulation, so the optimizer deserves the same care as the rest of the training recipe. Notably, methods validated on autoregressive language model pretraining transfer well: Muon, MARS-M and SOAP each beat the tuned AdamW on at least one diffusion formulation. The benchmark, all runs and every figure are reproducible end to end from the released code at https://github.com/armanbolatov/diffusion-baselines.

cs.LG↗
arXiv2026

Application of Optimal Inexact Second-Order Acceleration to Distributed Stochastic Optimization under Statistical Similarity

We consider distributed stochastic convex optimization with a fixed budget of $N$ independent samples split among $m$ workers. Sample average approximation reduces the problem to a regularized finite-sum problem whose local Hessians are statistically similar. This allows the Hessian of the local objective at the server to be used as an inexact Hessian of the global objective, while the workers communicate only gradients. We apply the optimal accelerated inexact Newton extragradient method of (Chen et al., 2026) and propose its distributed restarted variant for the strongly convex empirical problem. The method reaches the statistical accuracy of order $N^{-1/2}$ in $\widetilde O\left(\max\{N^{1/7},m^{1/4}\}\right)$ communication rounds. Hence, with $m=N^{4/7}$ workers, it requires $\widetilde O\left(N^{1/7}\right)$ rounds, improving the dependence on the total sample size from $\widetilde O\left(N^{1/6}\right)$ for the previous accelerated cubic Newton construction of (Agafonov et al., 2021). Each iteration uses two gradient aggregation rounds and does not require Hessian communication.

math.OC↗
arXiv2026

Certified Residual Quasi-Newton Methods for Distributed Variational Inequalities

Second-order methods for smooth monotone variational inequalities reach the optimal rate $O(T^{-3/2})$, but a distributed exact Jacobian costs $d$ times more communication than an operator value. We show that similarity does part of the work for free: if the server's Jacobian differs from the global one by at most $β$, using it gives $O(L_1D^3T^{-3/2}+βD^2T^{-1})$ at first-order communication cost. A quasi-Newton approximation of the residual Jacobian $\nabla F-\nabla F_1$, built from secants already communicated, improves the model but cannot remove the $T^{-1}$ term, because any uniform bound on the Jacobian error leaves it in the rate. We therefore certify the surrogate only along the candidate step: one Jacobian-vector product tests it, and a failed test is reused as an exact correction. This attains the exact rate $O(L_1D^3T^{-3/2})$ while transmitting only vectors. Experiments on LIBSVM and synthetic instances measure accuracy against communication.

math.OC↗
arXiv2026

Neither Adversarial Training Nor Purification: Emergent Adversarial Robustness from Oscillatory Predictive Learning

Adversarial robustness in computer vision is still largely achieved through adversarial training or test-time adversarial purification, both of which introduce significant computational overhead by generating adversarial examples during training or performing iterative denoising at test time. We study whether empirical robustness can instead emerge from architectural and representation-learning inductive biases. We introduce Oscillatory Predictive Learning (OPL), a two-stage framework that combines Artificial Kuramoto Oscillatory Neurons (AKOrN) with predictive self-supervised pretraining using X-PhiNet. Because our default checkpoint uses randomized initial oscillator states, we compare it with other randomized adversarial defense methods that provide precise, reproducible, and strong attack protocols. Experiments on CIFAR-10 and CIFAR-100, with additional corruption evaluation on CIFAR-10-C, demonstrate that our method achieves competitive results under the AutoAttack-rand evaluation protocol. On CIFAR-10 and CIFAR-100, OPL attains 76.63$\pm$0.76$\%$ and 50.44$\%$ robust accuracy, respectively, under $\ell_\infty$, $ε=8/255$, AutoAttack-rand with EoT $K=20$.

cs.LG↗
arXiv2026

Learning Latent Graph Geometry via Fixed-Point Schrödinger-Type Activation: A Theoretical Study

We study neural architectures in which each hidden layer is defined by the stationary state of a dissipative Schrödinger-type dynamics on a learned latent graph. On stable branches, the local stationary problem defines a differentiable implicit graph layer. To learn the graph itself, we optimize over the stratified moduli space of weighted graphs and equip each stratum with a non-degenerate Kähler-Hessian metric that keeps natural-gradient descent and face crossing well posed. We then show that a multilayer stationary network is equivalent to an exact global stationary problem on a supra-graph, and that it admits a penalized global relaxation whose stationary states converge to the exact one as the penalty parameter tends to infinity. Reverse-mode differentiation is recovered as the adjoint of the exact global system, and the penalized adjoint converges to it in the same limit. Finally, under finite-dimensional strong-monotonicity and admissible-lift assumptions, the corresponding represented hypothesis classes coincide among resolvent feed-forward networks, graph-stationary networks, supra-graph stationary systems, and sheaf-based architectures with unitary connection. The resulting structural identifications yield complexity bounds controlled by sparse graph or supra-graph geometry rather than dense ambient connectivity.

cs.LG↗
arXiv2026

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

In large-scale optimization, the cheapness and effectiveness of update steps are the most crucial factors for a successful optimizer. Sign-based optimizers like Lion or Signum produce cheap per-step updates, whereas Muon's spectral matrix-sign update gives a much stronger direction at a substantially higher per-step cost. In this work, we propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged iteration cost, similar to sign-based methods. It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them. The optimizer state memory therefore matches Lion and is exactly half of AdamW's. A simpler single-EMA variant, SignMuon, by itself already outperforms pure Muon. At P = 2, LionMuon Pareto-dominates Muon, Lion, Signum, and AdamW on every dataset and architecture we tested at 124M model size, reaching lower validation loss at lower compute, and the same advantage persists at 355M and 720M scale. On the theory side, we prove sharp complexity bounds under heavy-tailed noise which are governed by period-averaged smoothness and noise that interpolate between Muon's and Lion's constants. These bounds predict the compute-optimal period and the conditions under which LionMuon outruns Muon and Lion. Code: https://github.com/brain-lab-research/lion-muon

cs.LG↗
arXiv2026

WattCouncil: Context-Aware Household Energy Scenario Generation With Governed LLMs

The accelerating shift toward low-carbon power systems, together with the widespread adoption of behind-the-meter technologies such as rooftop solar and electric vehicles, is placing new operational and analytical demands on electricity grids. At the same time, smart-grid research increasingly relies on machine learning (ML), yet progress is constrained by limited access to high-resolution household energy data due to privacy concerns, regulatory barriers, and collection costs. This work presents WattCouncil, a data-generation framework in which household electricity demand is generated by a council of Large Language Model (LLM)-based agents operating in specialized roles to generate, audit, and validate structured energy scenarios under explicit cultural, temporal, and physical constraints. Rather than acting as static predictors, these agents serve as adaptive decision-makers within a governed pipeline. Motivated by studies highlighting the importance of contextual factors in energy use, our framework produces context-sensitive daily routines through a guided reasoning process that incorporates household composition, temporal factors, and environmental conditions. We evaluate the generated profiles against the detailed CER dataset, which contains over a year of load measurements for 4232 households together with survey-based socio-economic information. We further assess the consistency of the framework through ablation studies. Source code is available at https://github.com/Singularity-AI-Lab/wattcouncil

cs.AI↗
arXiv2026

CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor

Muon is an optimizer that computes updates using the polar factor of the momentum matrix and has shown strong empirical performance across a range of training settings. A key component of Muon is the Newton-Schulz iteration used to compute this polar factor. Although this avoids the cost of an exact singular value decomposition, it remains expensive in practice because it is applied at every optimization step. At the same time, the momentum matrix changes smoothly over training, suggesting strong temporal correlation in the corresponding polar factors. In this paper, we exploit this structure and propose CacheMuon, a temporal preconditioning method that reuses information from previous optimization steps to approximate the polar factor at the current step. This reduces redundant orthogonalization computation across iterations. We analyze CacheMuon as an inexact Muon update, with error controlled by fresh-solver error and cache staleness. Empirically, CacheMuon provides a controllable quality-efficiency frontier: conservative thresholds closely match fresh Muon on language-model and vision training while reducing orthogonalization FLOPs, whereas more aggressive thresholds yield larger arithmetic savings at the cost of modest validation-quality degradation.

cs.LG↗
arXiv2026

Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference

Running large language models locally is often impractical, pushing inference on sensitive text to third-party providers. Split inference partially mitigates this by keeping tokens on the client and sending only hidden representations, but these representations can still be recovered via nearest-neighbor search against the public embedding table. We propose an orthogonal obfuscation procedure in which the client multiplies embeddings by a secret orthogonal matrix before transmission. To enable correct inference under arbitrary rotations, we introduce ConjFormer, a transformer variant that is exactly $\mathrm{O}(d)$-equivariant via a lightweight normalization change (scalar RMSNorm) together with blockwise orthogonal conjugation of all linear weights. As a result, the server performs the full forward pass entirely in the rotated basis and never observes unrotated hidden states. Experiments on GPT-2 and Llama 3.2 1B models fine-tuned on PubMed show that orthogonal obfuscation eliminates direct cosine nearest-neighbor inversion and reduces token recovery from over 35% top-10 to at most 1.3%, while increasing perplexity by only 0.4% after fine-tuning. These results indicate that enforcing symmetry at the architectural level can provide a practical defense for privacy-preserving LLM inference without noise injection or heavy cryptographic machinery.

cs.LG↗