arXiv subjects

Guoqiang Zhang

Publications and source records attributed to Guoqiang Zhang.

At least 37 records · Page 2Linked to original sources

arXiv2025

High Quality Diffusion Distillation on a Single GPU with Relative and Absolute Position Matching

We introduce relative and absolute position matching (RAPM), a diffusion distillation method resulting in high quality generation that can be trained efficiently on a single GPU. Recent diffusion distillation research has achieved excellent results for high-resolution text-to-image generation with methods such as phased consistency models (PCM) and improved distribution matching distillation (DMD2). However, these methods generally require many GPUs (e.g.~8-64) and significant batchsizes (e.g.~128-2048) during training, resulting in memory and compute requirements that are beyond the resources of some researchers. RAPM provides effective single-GPU diffusion distillation training with a batchsize of 1. The new method attempts to mimic the sampling trajectories of the teacher model by matching the relative and absolute positions. The design of relative positions is inspired by PCM. Two discriminators are introduced accordingly in RAPM, one for matching relative positions and the other for absolute positions. Experimental results on StableDiffusion (SD) V1.5 and SDXL indicate that RAPM with 4 timesteps produces comparable FID scores as the best method with 1 timestep under very limited computational resources.

cs.CV↗

arXiv2025

Rigorous analysis of large-space and long-time asymptotics for the short-pulse soliton gases

We rigorously analyze the asymptotics of soliton gases to the short-pulse (SP) equation. The soliton gas is formulated in terms of a RH problem, which is derived from the RH problems of the $N$-soliton solutions with $N \to \infty$. Building on prior work in the study of the KdV soliton gas and orthogonal polynomials with Jacobi-type weights, we extend the reflection coefficient to two generalized forms on the interval $\left[η_1, η_2\right]$: $r_0(λ) = \left(λ- η_1\right)^{β_1}\left(η_2 - λ\right)^{β_2}|λ- η_0|^{β_0}γ(λ)$, $r_c(λ) = \left(λ- η_1\right)^{β_1}\left(η_2 - λ\right)^{β_2}χ_c(λ)γ(λ)$, where $0 < η_1 < η_0 < η_2$ and $β_j > -1$ ($j = 0, 1, 2$), $γ(λ)$ is continuous and positive on $\left[η_1, η_2\right]$, with an analytic extension to a neighborhood of this interval, $χ_c(λ) = 1$ for $λ\in \left[η_1, η_0\right)$ and $χ_c(λ) = c^2$ for $λ\in \left(η_0, η_2\right]$, where $c>0$ with $c \neq 1$. The asymptotic analysis is performed using the steepest descent method. A key aspect of the analysis is the construction of the $g$-function. To address the singularity at the origin, we introduce an innovative piecewise definition of $g$-function. To establish the order of the error term, we construct local parametrices near $η_j$ for $j = 1, 2$, and singularity $η_0$. At the endpoints, we employ the Airy parametrix and the first type of modified Bessel parametrix. At the singularity $η_0$, we use the second type of modified Bessel parametrix for $r_0$ and confluent hypergeometric parametrix for $r_c(λ)$.

nlin.SI↗

arXiv2025

Long-time asymptotics for the $N_{\infty}$-soliton solution to the KdV equation with two types of generalized reflection coefficients

We systematically investigate the long-time asymptotics for the $N_{\infty}$-soliton solution to the KdV equation in the different regions with the aid of the Riemann-Hilbert (RH) problems with two types of generalized reflection coefficients on the interval $\left[η_1, η_2\right]\in \mathbb{R}^+$: $r_0(λ,η_0; β_0, β_1,β_2)=\left(λ-η_1\right)^{β_1}\left(η_2-λ\right)^{β_2}\left|λ-η_0\right|^{β_0}γ\left(λ\right)$, $r_c(λ,η_0; β_1,β_2)=\left(λ-η_1\right)^{β_1}\left(η_2-λ\right)^{β_2}χ_c\left(λ, η_0\right)γ\left(λ\right)$, where the singularity $η_0\in (η_1, η_2)$ and $β_j>-1$ ($j=0, 1, 2$), $γ: \left[η_1, η_2\right] \to\mathbb{R}^+$ is continuous and positive on $\left[η_1, η_2\right]$, with an analytic extension to a neighborhood of this interval, and the step-like function $χ_c$ is defined as $χ_c\left(λ,η_0\right)=1$ for $λ\in\left[η_1, η_0\right)$ and $χ_c\left(λ,η_0\right)=c^2$ for $λ\in\left(η_0, η_2\right]$ with $c>0, \, c\ne1$. A critical step in the analysis of RH problems via the Deift-Zhou steepest descent technique is how to construct local parametrices around the endpoints $η_j$'s and the singularity $η_0$. Specifically, the modified Bessel functions of indexes $β_j$'s are utilized for the endpoints $η_j$'s, and the modified Bessel functions of index $\left(β_0\pm 1\right)\left/\right.2$ and confluent hypergeometric functions are employed around the singularity $η_0$ if the reflection coefficients are $r_0$ and $r_c$, respectively. This comprehensive study extends the understanding of generalized reflection coefficients and provides valuable insights into the asymptotics of soliton gases.

nlin.SI↗

arXiv2025

On Large-Space and Long-Time Asymptotic Behaviors of Kink-Soliton Gases in the Sine-Gordon Equation

We conduct a comprehensive analysis of the large-space and long-time asymptotics of kink-soliton gases in the sine-Gordon (sG) equation, addressing an important open problem highlighted in the recent work [Phys. Rev. E 109 (2024) 061001]. We focus on kink-soliton gases modeled within a Riemann-Hilbert framework and characterized by two types of generalized reflection coefficients, each defined on the interval $[η_1, η_2]$: $r_0(λ) = (λ- η_1)^{β_1} (η_2 - λ)^{β_2} |λ- η_0|^{β_0} γ(λ)$ and $r_c(λ) = (λ- η_1)^{β_1} (η_2 - λ)^{β_2} χ_c(λ) γ(λ)$, where $0 < η_1 < η_0 < η_2$ and $β_j > -1$, $γ(λ)$ is a continuous, strictly positive function defined on $[η_1, η_2]$. The function $χ_c(λ)$ demonstrates a step-like behavior: it is given by $χ_c(λ) = 1$ for $λ\in [η_1, η_0)$ and $χ_c(λ) = c^2$ for $λ\in (η_0, η_2]$, with $c$ as a positive constant distinct from one. To rigorously derive the asymptotic results, we leverage the Deift-Zhou steepest descent method. A central component of this approach is constructing an appropriate $g$-function for the conjugation process. Unlike in the KdV equation, the sG presents unique challenges for $g$-function formulation, particularly concerning the singularity at the origin. The Riemann-Hilbert problem also requires carefully constructed local parametrices near endpoints $η_j$ and the singularity $η_0$. At the endpoints $η_j$, we employ a modified Bessel parametrix of the first kind. For the singularity $η_0$, the parametrix selection depends on the reflection coefficient: the second kind of modified Bessel parametrix is used for $r_0(λ)$, while a confluent hypergeometric parametrix is applied for $r_c(λ)$.

nlin.SI↗

arXiv2024

The focusing complex mKdV equation with nonzero background: Large $N$-order asymptotics of multi-rational solitons and related Painlevé-III hierarchy

In this paper, we investigate the large-order asymptotics of multi-rational solitons of the focusing complex modified Korteweg-de Vries (c-mKdV) equation with nonzero background via the Riemann-Hilbert problems. First, based on the Lax pair, inverse scattering transform, and a series of deformations, we construct a multi-rational soliton of the c-mKdV equation via a solvable Riemann-Hilbert problem (RHP). Then, through a scale transformation, we construct a RHP corresponding to the limit function which is a new solution of the c-mKdV equation in the rescaled variables $X,\,T$, and prove the existence and uniqueness of the RHP's solution. Moreover, we also find that the limit function satisfies the ordinary differential equations (ODEs) with respect to space $X$ and time $T$, respectively. The ODEs with respect to space $X$ are identified with certain members of the Painlevé-III hierarchy. We study the large $X$ and transitional asymptotic behaviors of near-field limit solutions, and we provide some part results for the case of large $T$. These results will be useful to understand and apply the large-order rational solitons in the nonlinear wave equations.

nlin.SI↗

arXiv2024

AttentionX: Exploiting Consensus Discrepancy In Attention from A Distributed Optimization Perspective

In this paper, we extend the standard Attention in transformer by exploiting the consensus discrepancy from a distributed optimization perspective, referred to as AttentionX. It is noted that the primal-dual method of multipliers (PDMM) \cite{Zhang16PDMM} is designed to iteratively solve a broad class of distributed optimization problems over a pear-to-pear (P2P) network, where neighbouring nodes gradually reach consensus as specified by predefined linear edge-constraints in the optimization process. In particular, at each iteration of PDMM, each node in a network first performs information-gathering from neighbours and then performs local information-fusion. From a high-level point of view, the $KQ$-softmax-based weighted summation of $V$-representations in Attention corresponds information-gathering from neighbours while the feature-processing via the feed-forward network (FFN) in transformer corresponds to local information fusion. PDMM exploits the Lagrangian multipliers to capture the historical consensus discrepancy in the form of residual errors of the linear edge-constraints, which plays a crucial role for the algorithm to converge. Inspired by PDMM, we propose AttentionX to incorporate the consensus discrepancy in the output update-expression of the standard Attention. The consensus discrepancy in AttentionX refers to the difference between the weighted summation of $V$-representations and scaled $V$-representions themselves. Experiments on ViT and nanoGPT show promising performance.

cs.LG↗

arXiv2024

On Exact Bit-level Reversible Transformers Without Changing Architectures

Various reversible deep neural networks (DNN) models have been proposed to reduce memory consumption in the training process. However, almost all existing reversible DNNs either require special non-standard architectures or are constructed by modifying existing DNN architectures considerably to enable reversibility. In this work we present the BDIA-transformer, which is an exact bit-level reversible transformer that uses an unchanged standard architecture for inference. The basic idea is to first treat each transformer block as the Euler integration approximation for solving an ordinary differential equation (ODE) and then incorporate the technique of bidirectional integration approximation (BDIA) into the neural architecture, together with activation quantization to make it exactly bit-level reversible. In the training process, we let a hyper-parameter $γ$ in BDIA-transformer randomly take one of the two values $\{0.5, -0.5\}$ per training sample per transformer block for averaging every two consecutive integration approximations. As a result, BDIA-transformer can be viewed as training an ensemble of ODE solvers parameterized by a set of binary random variables, which regularizes the model and results in improved validation accuracy. Lightweight side information per transformer block is required to be stored in the forward process to account for binary quantization loss to enable exact bit-level reversibility. In the inference procedure, the expectation $\mathbb{E}(γ)=0$ is taken to make the resulting architectures of BDIA-transformer identical to transformers up to activation quantization. Our experiments in both image classification and language translation show that BDIA-transformers outperform their conventional counterparts significantly in terms of validation performance while also requiring considerably less training memory.

cs.LG↗

arXiv2024

On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performance

A number of recent adaptive optimizers improve the generalisation performance of Adam by essentially reducing the variance of adaptive stepsizes to get closer to SGD with momentum. Following the above motivation, we suppress the range of the adaptive stepsizes of Adam by exploiting the layerwise gradient statistics. In particular, at each iteration, we propose to perform three consecutive operations on the second momentum v_t before using it to update a DNN model: (1): down-scaling, (2): epsilon-embedding, and (3): down-translating. The resulting algorithm is referred to as SET-Adam, where SET is a brief notation of the three operations. The down-scaling operation on v_t is performed layerwise by making use of the angles between the layerwise subvectors of v_t and the corresponding all-one subvectors. Extensive experimental results show that SET-Adam outperforms eight adaptive optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the eight adaptive methods when training WGAN-GP models for image generation tasks. Furthermore, SET-Adam produces higher validation accuracies than Adam and AdaBelief for training ResNet18 over ImageNet.

cs.LG↗

arXiv2024

Distributed Nonlinear Conic Optimisation with partially separable Structure

In this paper we consider the problem of distributed nonlinear optimisation of a separable convex cost function over a graph subject to cone constraints. We show how to generalise, using convex analysis, monotone operator theory and fixed-point theory, the primal-dual method of multipliers (PDMM), originally designed for equality constraint optimisation and recently extended to include linear inequality constraints, to accommodate for cone constraints. The resulting algorithm can be used to implement a variety of optimisation problems, including the important class of semidefinite programs with partially separable structure, in a fully distributed fashion. We derive update equations by applying the Peaceman-Rachford splitting algorithm to the monotonic inclusion related to the lifted dual problem. The cone constraints are implemented by a reflection method in the lifted dual domain where auxiliary variables are reflected with respect to the intersection of the polar cone and a subspace relating the dual and lifted dual domain. Convergence results for both synchronous and stochastic update schemes are provided and an application of the proposed algorithm is demonstrated to implement an approximate algorithm for maximum cut problems based on semidefinite programming in a fully distributed fashion.

cs.DC↗

arXiv2024

Propensity-score matching analysis in COVID-19-related studies: a method and quality systematic review

Objectives: To provide an overall quality assessment of the methods used for COVID-19-related studies using propensity score matching (PSM). Study Design and Setting: A systematic search was conducted in June 2021 on PubMed to identify COVID-19-related studies that use the PSM analysis between 2020 and 2021. Key information about study design and PSM analysis were extracted, such as covariates, matching algorithm, and reporting of estimated treatment effect type. Results: One-hundred-and-fifty (87.72%) cohort studies and thirteen (7.60%) case-control studies were found among 171 identified articles. Forty-five studies (26.32%) provided a reasonable justification for covariates selection. One-hundred-and-three (60.23%) and Sixty-nine (40.35%) studies did not provide the model that was used for calculating the propensity score or did not report the matching algorithm, respectively. Seventy-three (42.69%) studies reported the method(s) for checking covariates balance. Forty studies (23.39%) had a statistician co-author. All the case-control studies (n=13) did not have a statistician co-author (p=0.006) and all studies that clarified the treatment effect estimation (n=6) had a statistician co-author (p<0.001). Conclusions: The reporting quality of the PSM analysis is suboptimal in some COVID-19 epidemiological studies. Some pitfalls may undermine study findings that involve PSM analysis, such as a mismatch between PSM analysis and study design.

q-bio.OT↗

arXiv2024

Ultra-short lifetime isomer studies from photonuclear reactions using laser-driven ultra-intense γ-ray

Isomers, ubiquitous populations of relatively long-lived nuclear excited states, play a crucial role in nuclear physics. However, isomers with half-life times of several seconds or less barely had experimental cross section data due to the lack of a suitable measuring method. We report a method of online γ spectroscopy for ultra-short-lived isomers from photonuclear reactions using laser-driven ultra-intense γ-rays. The fastest time resolution can reach sub-ps level with γ-ray intensities >10^{19}/s ({\geqslant} 8 MeV). The ^{115}In(γ, n)^{114m2}In reaction (T_{1/2} = 43.1 ms) was first measured in the high-energy region which shed light on the nuclear structure studies of In element. Simulations showed it would be an efficient way to study ^{229m}Th (T_{1/2} = 7 μs), which is believed to be the next generation of nuclear clock. This work offered a unique way of gaining insight into ultra-short lifetimes and promised an effective way to fill the gap in relevant experimental data.

nucl-ex↗

arXiv2024

Distributed Optimisation with Linear Equality and Inequality Constraints using PDMM

In this paper, we consider the problem of distributed optimisation of a separable convex cost function over a graph, where every edge and node in the graph could carry both linear equality and/or inequality constraints. We show how to modify the primal-dual method of multipliers (PDMM), originally designed for linear equality constraints, such that it can handle inequality constraints as well. The proposed algorithm does not need any slack variables, which is similar to the recent work [1] which extends the alternating direction method of multipliers (ADMM) for addressing decomposable optimisation with linear equality and inequality constraints. Using convex analysis, monotone operator theory and fixed-point theory, we show how to derive the update equations of the modified PDMM algorithm by applying Peaceman-Rachford splitting to the monotonic inclusion related to the lifted dual problem. To incorporate the inequality constraints, we impose a non-negativity constraint on the associated dual variables. This additional constraint results in the introduction of a reflection operator to model the data exchange in the network, instead of a permutation operator as derived for equality constraint PDMM. Convergence for both synchronous and stochastic update schemes of PDMM are provided. The latter includes asynchronous update schemes and update schemes with transmission losses. Experiments show that PDMM converges notably faster than extended ADMM of [1].

cs.DC↗

arXiv2024

Enhanced α particle generation via proton-boron fusion reactions in laser-modulated plasma

Aneutronic and nonradioactive properties make the proton-boron fusion a prospective candidate for fusion energy production through reactions following p+$^{11}$B$\rightarrow$3$α$ (p-$^{11}$B). However, it is difficult to achieve a thermal fusion ignition, since the low reaction cross-sections for center-of-mass energy below $\sim$100 keV. To realize fusion energy gain, it is essential to consider utilization of the maximum cross-section at the resonant peak of p-$^{11}$B fusion, and explore the nuclear reactions in plasma environment. In this work, p-$^{11}$B reactions triggered by interactions between energetic proton beams and laser-ablated boron plasma have been investigated. More than 200 times enhancement of $α$ particle emission efficiency (number ratio of escaping $α$ particles and boron nuclei) in plasma has been observed, compared with the cold boron. The proton beam transport path modulated by strong electro-magnetic fields in plasma could dominate the enhanced $α$ particle generation, due to a longer collisional length. In addition, an $α$ particle yield up to 1$\times$10$^{10}$ /sr has been measured via the pitcher-catcher scheme in plasma. This work could benefit understanding of the plasma effects on nuclear reaction dynamics, and also enable opportunities to explore physics in laser fusion associated with advanced fusion fuels.

physics.plasm-ph↗

arXiv2023

Exact Diffusion Inversion via Bi-directional Integration Approximation

Recently, various methods have been proposed to address the inconsistency issue of DDIM inversion to enable image editing, such as EDICT [36] and Null-text inversion [22]. However, the above methods introduce considerable computational overhead. In this paper, we propose a new technique, named \emph{bi-directional integration approximation} (BDIA), to perform exact diffusion inversion with neglible computational overhead. Suppose we would like to estimate the next diffusion state $\boldsymbol{z}_{i-1}$ at timestep $t_i$ with the historical information $(i,\boldsymbol{z}_i)$ and $(i+1,\boldsymbol{z}_{i+1})$. We first obtain the estimated Gaussian noise $\hat{\boldsymbolε}(\boldsymbol{z}_i,i)$, and then apply the DDIM update procedure twice for approximating the ODE integration over the next time-slot $[t_i, t_{i-1}]$ in the forward manner and the previous time-slot $[t_i, t_{t+1}]$ in the backward manner. The DDIM step for the previous time-slot is used to refine the integration approximation made earlier when computing $\boldsymbol{z}_i$. A nice property of BDIA-DDIM is that the update expression for $\boldsymbol{z}_{i-1}$ is a linear combination of $(\boldsymbol{z}_{i+1}, \boldsymbol{z}_i, \hat{\boldsymbolε}(\boldsymbol{z}_i,i))$. This allows for exact backward computation of $\boldsymbol{z}_{i+1}$ given $(\boldsymbol{z}_i, \boldsymbol{z}_{i-1})$, thus leading to exact diffusion inversion. It is demonstrated with experiments that (round-trip) BDIA-DDIM is particularly effective for image editing. Our experiments further show that BDIA-DDIM produces markedly better image sampling qualities than DDIM for text-to-image generation. BDIA can also be applied to improve the performance of other ODE solvers in addition to DDIM. In our work, it is found that applying BDIA to the EDM sampling procedure produces consistently better performance over four pre-trained models.

cs.CV↗

arXiv2023

A Distributed Algorithm for Personal Sound Zones Systems

A Personal Sound Zones (PSZ) system aims to generate two or more independent listening zones that allow multiple users to listen to different music/audio content in a shared space without the need for wearing headphones. Most existing studies assume that the acoustic paths between loudspeakers and microphones are measured beforehand in a stationary environment. Recently, adaptive PSZ systems have been explored to adapt the system in a time-varying acoustic environment. However, because a PSZ system usually requires multiple loudspeakers, the multichannel adaptive algorithms impose a high computational load on the processor. To overcome that problem, this paper proposes an efficient distributed algorithm for PSZ systems, which not only spreads the computational burden over multiple nodes but also reduces the overall computational complexity, at the expense of a slight decrease in performance. Simulation results with true room impulse responses measured in a Hemi-Anechoic chamber are performed to verify the proposed distributed PSZ system.

eess.AS↗

arXiv2023

On Accelerating Diffusion-Based Sampling Process via Improved Integration Approximation

A popular approach to sample a diffusion-based generative model is to solve an ordinary differential equation (ODE). In existing samplers, the coefficients of the ODE solvers are pre-determined by the ODE formulation, the reverse discrete timesteps, and the employed ODE methods. In this paper, we consider accelerating several popular ODE-based sampling processes (including EDM, DDIM, and DPM-Solver) by optimizing certain coefficients via improved integration approximation (IIA). We propose to minimize, for each time step, a mean squared error (MSE) function with respect to the selected coefficients. The MSE is constructed by applying the original ODE solver for a set of fine-grained timesteps, which in principle provides a more accurate integration approximation in predicting the next diffusion state. The proposed IIA technique does not require any change of a pre-trained model, and only introduces a very small computational overhead for solving a number of quadratic optimization problems. Extensive experiments show that considerably better FID scores can be achieved by using IIA-EDM, IIA-DDIM, and IIA-DPM-Solver than the original counterparts when the neural function evaluation (NFE) is small (i.e., less than 25).

cs.LG↗

arXiv2023

Lookahead Diffusion Probabilistic Models for Refining Mean Estimation

We propose lookahead diffusion probabilistic models (LA-DPMs) to exploit the correlation in the outputs of the deep neural networks (DNNs) over subsequent timesteps in diffusion probabilistic models (DPMs) to refine the mean estimation of the conditional Gaussian distributions in the backward process. A typical DPM first obtains an estimate of the original data sample $\boldsymbol{x}$ by feeding the most recent state $\boldsymbol{z}_i$ and index $i$ into the DNN model and then computes the mean vector of the conditional Gaussian distribution for $\boldsymbol{z}_{i-1}$. We propose to calculate a more accurate estimate for $\boldsymbol{x}$ by performing extrapolation on the two estimates of $\boldsymbol{x}$ that are obtained by feeding $(\boldsymbol{z}_{i+1},i+1)$ and $(\boldsymbol{z}_{i},i)$ into the DNN model. The extrapolation can be easily integrated into the backward process of existing DPMs by introducing an additional connection over two consecutive timesteps, and fine-tuning is not required. Extensive experiments showed that plugging in the additional connection into DDPM, DDIM, DEIS, S-PNDM, and high-order DPM-Solvers leads to a significant performance gain in terms of FID score.

cs.AI↗

arXiv2023

A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range

We make contributions towards improving adaptive-optimizer performance. Our improvements are based on suppression of the range of adaptive stepsizes in the AdaBelief optimizer. Firstly, we show that the particular placement of the parameter epsilon within the update expressions of AdaBelief reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum. Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve the above goal, we perform mutual layerwise vector projections between the gradient g_t and its first momentum m_t before using them to estimate the second momentum. The new optimization method is referred to as Aida. Thirdly, extensive experimental results show that Aida outperforms nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the nine methods when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher validation accuracies than AdaBelief for training ResNet18 over ImageNet. Code is available at this URL

cs.LG↗