arXiv subjects

Lingjiong Zhu

Publications and source records attributed to Lingjiong Zhu.

At least 55 records · Page 3Linked to original sources

arXiv2025

Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models

Score-based generative models (SGMs) is a recent class of deep generative models with state-of-the-art performance in many applications. In this paper, we establish convergence guarantees for a general class of SGMs in 2-Wasserstein distance, assuming accurate score estimates and smooth log-concave data distribution. We specialize our result to several concrete SGMs with specific choices of forward processes modelled by stochastic differential equations, and obtain an upper bound on the iteration complexity for each model, which demonstrates the impacts of different choices of the forward processes. We also provide a lower bound when the data distribution is Gaussian. Numerically, we experiment SGMs with different forward processes, some of which are newly proposed in this paper, for unconditional image generation on CIFAR-10. We find that the experimental results are in good agreement with our theoretical predictions on the iteration complexity, and the models with our newly proposed forward processes can outperform existing models.

cs.LG↗

arXiv2025

Convergence Analysis for General Probability Flow ODEs of Diffusion Models in Wasserstein Distances

Score-based generative modeling with probability flow ordinary differential equations (ODEs) has achieved remarkable success in a variety of applications. While various fast ODE-based samplers have been proposed in the literature and employed in practice, the theoretical understandings about convergence properties of the probability flow ODE are still quite limited. In this paper, we provide the first non-asymptotic convergence analysis for a general class of probability flow ODE samplers in 2-Wasserstein distance, assuming accurate score estimates and smooth log-concave data distributions. We then consider various examples and establish results on the iteration complexity of the corresponding ODE-based samplers. Our proof technique relies on spelling out explicitly the contraction rate for the continuous-time ODE and analyzing the discretization and score-matching errors using synchronous coupling; the challenge in our analysis mainly arises from the inherent non-autonomy of the probability flow ODE and the specific exponential integrator that we study.

stat.ML↗

arXiv2025

Algorithmic Stability of Stochastic Gradient Descent with Momentum under Heavy-Tailed Noise

Understanding the generalization properties of optimization algorithms under heavy-tailed noise has gained growing attention. However, the existing theoretical results mainly focus on stochastic gradient descent (SGD) and the analysis of heavy-tailed optimizers beyond SGD is still missing. In this work, we establish generalization bounds for SGD with momentum (SGDm) under heavy-tailed gradient noise. We first consider the continuous-time limit of SGDm, i.e., a Levy-driven stochastic differential equation (SDE), and establish quantitative Wasserstein algorithmic stability bounds for a class of potentially non-convex loss functions. Our bounds reveal a remarkable observation: For quadratic loss functions, we show that SGDm admits a worse generalization bound in the presence of heavy-tailed noise, indicating that the interaction of momentum and heavy tails can be harmful for generalization. We then extend our analysis to discrete-time and develop a uniform-in-time discretization error bound, which, to our knowledge, is the first result of its kind for SDEs with degenerate noise. This result shows that, with appropriately chosen step-sizes, the discrete dynamics retain the generalization properties of the limiting SDE. We illustrate our theory on both synthetic quadratic problems and neural networks.

stat.ML↗

arXiv2024

Short-maturity asymptotics for VIX and European options in local-stochastic volatility models

We derive the short-maturity asymptotics for European and VIX option prices in local-stochastic volatility models where the volatility follows a continuous-path Markov process. Both out-of-the-money (OTM) and at-the-money (ATM) asymptotics are considered. Using large deviations theory methods, the asymptotics for the OTM options are expressed as a two-dimensional variational problem, which is reduced to an extremal problem for a function of two real variables. This extremal problem is solved explicitly in an expansion in log-moneyness. We derive series expansions for the implied volatility for European and VIX options which should be useful for model calibration. We give explicit results for two classes of local-stochastic volatility models relevant in practice, with Heston-type and SABR-type stochastic volatility. The leading-order asymptotics for at-the-money options are computed in closed-form. The asymptotic results reproduce known results in the literature for the Heston and SABR models and for the uncorrelated local-stochastic volatility model. The asymptotic results are tested against numerical simulations for a local-stochastic volatility model with bounded local volatility.

q-fin.PR↗

arXiv2024

Penalized Overdamped and Underdamped Langevin Monte Carlo Algorithms for Constrained Sampling

We consider the constrained sampling problem where the goal is to sample from a target distribution $π(x)\propto e^{-f(x)}$ when $x$ is constrained to lie on a convex body $\mathcal{C}$. Motivated by penalty methods from continuous optimization, we propose penalized Langevin Dynamics (PLD) and penalized underdamped Langevin Monte Carlo (PULMC) methods that convert the constrained sampling problem into an unconstrained sampling problem by introducing a penalty function for constraint violations. When $f$ is smooth and gradients are available, we get $\tilde{\mathcal{O}}(d/\varepsilon^{10})$ iteration complexity for PLD to sample the target up to an $\varepsilon$-error where the error is measured in the TV distance and $\tilde{\mathcal{O}}(\cdot)$ hides logarithmic factors. For PULMC, we improve the result to $\tilde{\mathcal{O}}(\sqrt{d}/\varepsilon^{7})$ when the Hessian of $f$ is Lipschitz and the boundary of $\mathcal{C}$ is sufficiently smooth. To our knowledge, these are the first convergence results for underdamped Langevin Monte Carlo methods in the constrained sampling that handle non-convex $f$ and provide guarantees with the best dimension dependency among existing methods with deterministic gradient. If unbiased stochastic estimates of the gradient of $f$ are available, we propose PSGLD and PSGULMC methods that can handle stochastic gradients and are scaleable to large datasets without requiring Metropolis-Hasting correction steps. For PSGLD and PSGULMC, when $f$ is strongly convex and smooth, we obtain $\tilde{\mathcal{O}}(d/\varepsilon^{18})$ and $\tilde{\mathcal{O}}(d\sqrt{d}/\varepsilon^{39})$ iteration complexity in W2 distance. When $f$ is smooth and can be non-convex, we provide finite-time performance bounds and iteration complexity results. Finally, we illustrate the performance on Bayesian LASSO regression and Bayesian constrained deep learning problems.

stat.ML↗

arXiv2024

Asymptotics for Short Maturity Asian Options in Jump-Diffusion models with Local Volatility

We present a study of the short maturity asymptotics for Asian options in a jump-diffusion model with a local volatility component, where the jumps are modeled as a compound Poisson process. The analysis for out-of-the-money Asian options is extended to models with Lévy jumps, including the exponential Lévy model as a special case. Both fixed and floating strike Asian options are considered. Explicit results are obtained for the first-order asymptotics of the Asian options prices for a few popular models in the literature: the Merton jump-diffusion model, the double-exponential jump model, and the Variance Gamma model. We propose an analytical approximation for Asian option prices which satisfies the constraints from the short-maturity asymptotics, and test it against Monte Carlo simulations. The asymptotic results are in good agreement with numerical simulations for sufficiently small maturity.

q-fin.PR↗

arXiv2024

Short-maturity asymptotics for option prices with interest rates effects

We derive the short-maturity asymptotics for option prices in the local volatility model in a new short-maturity limit $T\to 0$ at fixed $ρ= (r-q) T$, where $r$ is the interest rate and $q$ is the dividend yield. In cases of practical relevance $ρ$ is small, however our result holds for any fixed $ρ$. The result is a generalization of the Berestycki-Busca-Florent formula for the short-maturity asymptotics of the implied volatility which includes interest rates and dividend yield effects of $O(((r-q) T)^n)$ to all orders in $n$. We obtain analytical results for the ATM volatility and skew in this asymptotic limit. Explicit results are derived for the CEV model. The asymptotic result is tested numerically against exact evaluation in the square-root model model $σ(S)=σ/\sqrt{S}$, which demonstrates that the new asymptotic result is in very good agreement with exact evaluation in a wide range of model parameters relevant for practical applications.

q-fin.PR↗

arXiv2024

Intriguing Differences Between Zero-Shot and Systematic Evaluations of Vision-Language Transformer Models

Transformer-based models have dominated natural language processing and other areas in the last few years due to their superior (zero-shot) performance on benchmark datasets. However, these models are poorly understood due to their complexity and size. While probing-based methods are widely used to understand specific properties, the structures of the representation space are not systematically characterized; consequently, it is unclear how such models generalize and overgeneralize to new inputs beyond datasets. In this paper, based on a new gradient descent optimization method, we are able to explore the embedding space of a commonly used vision-language model. Using the Imagenette dataset, we show that while the model achieves over 99\% zero-shot classification performance, it fails systematic evaluations completely. Using a linear approximation, we provide a framework to explain the striking differences. We have also obtained similar results using a different model to support that our results are applicable to other transformer models with continuous inputs. We also propose a robust way to detect the modified images.

cs.CV↗

arXiv2023

Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent

Algorithmic stability is an important notion that has proven powerful for deriving generalization bounds for practical algorithms. The last decade has witnessed an increasing number of stability bounds for different algorithms applied on different classes of loss functions. While these bounds have illuminated various properties of optimization algorithms, the analysis of each case typically required a different proof technique with significantly different mathematical tools. In this study, we make a novel connection between learning theory and applied probability and introduce a unified guideline for proving Wasserstein stability bounds for stochastic optimization algorithms. We illustrate our approach on stochastic gradient descent (SGD) and we obtain time-uniform stability bounds (i.e., the bound does not increase with the number of iterations) for strongly convex losses and non-convex losses with additive noise, where we recover similar results to the prior art or extend them to more general cases by using a single proof technique. Our approach is flexible and can be generalizable to other popular optimizers, as it mainly requires developing Lyapunov functions, which are often readily available in the literature. It also illustrates that ergodicity is an important component for obtaining time-uniform bounds -- which might not be achieved for convex or non-convex losses unless additional noise is injected to the iterates. Finally, we slightly stretch our analysis technique and prove time-uniform bounds for SGD under convex and non-convex losses (without additional additive noise), which, to our knowledge, is novel.

stat.ML↗

arXiv2023

Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize

Cyclic and randomized stepsizes are widely used in the deep learning practice and can often outperform standard stepsize choices such as constant stepsize in SGD. Despite their empirical success, not much is currently known about when and why they can theoretically improve the generalization performance. We consider a general class of Markovian stepsizes for learning, which contain i.i.d. random stepsize, cyclic stepsize as well as the constant stepsize as special cases, and motivated by the literature which shows that heaviness of the tails (measured by the so-called "tail-index") in the SGD iterates is correlated with generalization, we study tail-index and provide a number of theoretical results that demonstrate how the tail-index varies on the stepsize scheduling. Our results bring a new understanding of the benefits of cyclic and randomized stepsizes compared to constant stepsize in terms of the tail behavior. We illustrate our theory on linear regression experiments and show through deep learning experiments that Markovian stepsizes can achieve even a heavier tail and be a viable alternative to cyclic and i.i.d. randomized stepsize rules.

stat.ML↗

arXiv2023

Fluctuations and moderate deviations for the mean fields of Hawkes processes

The Hawkes process is a counting process that has self- and mutually-exciting features with many applications in various fields. In recent years, there have been many interests in the mean-field results of the Hawkes process and its extensions. It is known that the mean-field limit of a multivariate nonlinear Hawkes process is a time-inhomogeneous Poisson process. In this paper, we study the fluctuations for the mean fields and the large deviations associated with the fluctuations, i.e., the moderate deviations.

math.PR↗

arXiv2023

Asymptotics for the Laplace transform of the time integral of the geometric Brownian motion

We present an asymptotic result for the Laplace transform of the time integral of the geometric Brownian motion $F(θ,T) = \mathbb{E}[e^{-θX_T}]$ with $X_T = \int_0^T e^{σW_s + ( a - \frac12 σ^2)s} ds$, which is exact in the limit $σ^2 T \to 0$ at fixed $σ^2 θT^2$ and $aT$. This asymptotic result is applied to pricing zero coupon bonds in the Dothan model of stochastic interest rates. The asymptotic result provides an approximation for bond prices which is in good agreement with numerical evaluations in a wide range of model parameters. As a side result we obtain the asymptotics for Asian option prices in the Black-Scholes model, taking into account interest rates and dividend yield contributions in the $σ^{2}T\to 0$ limit.

q-fin.PR↗

arXiv2023

Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares

Recent studies have shown that heavy tails can emerge in stochastic optimization and that the heaviness of the tails have links to the generalization error. While these studies have shed light on interesting aspects of the generalization behavior in modern settings, they relied on strong topological and statistical regularity assumptions, which are hard to verify in practice. Furthermore, it has been empirically illustrated that the relation between heavy tails and generalization might not always be monotonic in practice, contrary to the conclusions of existing theory. In this study, we establish novel links between the tail behavior and generalization properties of stochastic gradient descent (SGD), through the lens of algorithmic stability. We consider a quadratic optimization problem and use a heavy-tailed stochastic differential equation (and its Euler discretization) as a proxy for modeling the heavy-tailed behavior emerging in SGD. We then prove uniform stability bounds, which reveal the following outcomes: (i) Without making any exotic assumptions, we show that SGD will not be stable if the stability is measured with the squared-loss $x\mapsto x^2$, whereas it in turn becomes stable if the stability is instead measured with a surrogate loss $x\mapsto |x|^p$ with some $p<2$. (ii) Depending on the variance of the data, there exists a \emph{`threshold of heavy-tailedness'} such that the generalization error decreases as the tails become heavier, as long as the tails are lighter than this threshold. This suggests that the relation between heavy tails and generalization is not globally monotonic. (iii) We prove matching lower-bounds on uniform stability, implying that our bounds are tight in terms of the heaviness of the tails. We support our theory with synthetic and real neural network experiments.

stat.ML↗

arXiv2023

A State-Dependent Dual Risk Model

In a dual risk model, the premiums are considered as the costs and the claims are regarded as the profits. The surplus can be interpreted as the wealth of a venture capital, whose profits depend on research and development. In most of the existing literature of dual risk models, the profits follow the compound Poisson model and the cost is constant. In this paper, we develop a state-dependent dual risk model, in which the arrival rate of the profits and the costs depend on the current state of the wealth process. Ruin probabilities are obtained in closed-forms. Further properties and results will also be discussed.

q-fin.RM↗

arXiv2023

Optimal Investment in a Dual Risk Model

Dual risk models are popular for modeling a venture capital or high tech company, for which the running cost is deterministic and the profits arrive stochastically over time. Most of the existing literature on dual risk models concentrated on the optimal dividend strategies. In this paper, we propose to study the optimal investment strategy on research and development for the dual risk models to minimize the ruin probability of the underlying company. We will also study the optimization problem when in addition the investment in a risky asset is allowed.

q-fin.RM↗

arXiv2023

Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions

Heavy-tail phenomena in stochastic gradient descent (SGD) have been reported in several empirical studies. Experimental evidence in previous works suggests a strong interplay between the heaviness of the tails and generalization behavior of SGD. To address this empirical phenomena theoretically, several works have made strong topological and statistical assumptions to link the generalization error to heavy tails. Very recently, new generalization bounds have been proven, indicating a non-monotonic relationship between the generalization error and heavy tails, which is more pertinent to the reported empirical observations. While these bounds do not require additional topological assumptions given that SGD can be modeled using a heavy-tailed stochastic differential equation (SDE), they can only apply to simple quadratic problems. In this paper, we build on this line of research and develop generalization bounds for a more general class of objective functions, which includes non-convex functions as well. Our approach is based on developing Wasserstein stability bounds for heavy-tailed SDEs and their discretizations, which we then convert to generalization bounds. Our results do not require any nontrivial assumptions; yet, they shed more light to the empirical observations, thanks to the generality of the loss functions.

stat.ML↗

arXiv2023

A delayed dual risk model

In this paper, we study a dual risk model with delays in the spirit of Dassios-Zhao. When a new innovation occurs, there is a delay before the innovation turns into a profit. We obtain large initial surplus asymptotics for the ruin probability and ruin time distributions. For some special cases, we get closed-form formulas. Numerical illustrations will also be provided.

q-fin.RM↗

arXiv2023

Sensitivities of Asian options in the Black-Scholes model

We propose analytical approximations for the sensitivities (Greeks) of the Asian options in the Black-Scholes model, following from a small maturity/volatility approximation for the option prices which has the exact short maturity limit, obtained using large deviations theory. Numerical tests demonstrate good agreement of the proposed approximation with alternative numerical simulation results for cases of practical interest. We also study the qualitative properties of Asian Greeks, including new results for Rho, the sensitivity with respect to changes in the risk-free rate, and Psi, the sensitivity with respect to the dividend yield. In particular we show that the Rho of a fixed-strike Asian option and the Psi of a floating-strike Asian option can change sign.

q-fin.PR↗