arXiv subjects

Zheng Zhao

Publications and source records attributed to Zheng Zhao.

At least 73 records · Page 4Linked to original sources

arXiv2023

Image Restoration with Mean-Reverting Stochastic Differential Equations

This paper presents a stochastic differential equation (SDE) approach for general-purpose image restoration. The key construction consists in a mean-reverting SDE that transforms a high-quality image into a degraded counterpart as a mean state with fixed Gaussian noise. Then, by simulating the corresponding reverse-time SDE, we are able to restore the origin of the low-quality image without relying on any task-specific prior knowledge. Crucially, the proposed mean-reverting SDE has a closed-form solution, allowing us to compute the ground truth time-dependent score and learn it with a neural network. Moreover, we propose a maximum likelihood objective to learn an optimal reverse trajectory that stabilizes the training and improves the restoration results. The experiments show that our proposed method achieves highly competitive performance in quantitative comparisons on image deraining, deblurring, and denoising, setting a new state-of-the-art on two deraining datasets. Finally, the general applicability of our approach is further demonstrated via qualitative results on image super-resolution, inpainting, and dehazing. Code is available at https://github.com/Algolzw/image-restoration-sde.

cs.LG↗

arXiv2023

Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models

This work aims to improve the applicability of diffusion models in realistic image restoration. Specifically, we enhance the diffusion model in several aspects such as network architecture, noise level, denoising steps, training image size, and optimizer/scheduler. We show that tuning these hyperparameters allows us to achieve better performance on both distortion and perceptual scores. We also propose a U-Net based latent diffusion model which performs diffusion in a low-resolution latent space while preserving high-resolution information from the original input for the decoding process. Compared to the previous latent-diffusion model which trains a VAE-GAN to compress the image, our proposed U-Net compression strategy is significantly more stable and can recover highly accurate images without relying on adversarial optimization. Importantly, these modifications allow us to apply diffusion models to various image restoration tasks, including real-world shadow removal, HR non-homogeneous dehazing, stereo super-resolution, and bokeh effect transformation. By simply replacing the datasets and slightly changing the noise network, our model, named Refusion, is able to deal with large-size images (e.g., 6000 x 4000 x 3 in HR dehazing) and produces good results on all the above restoration problems. Our Refusion achieves the best perceptual performance in the NTIRE 2023 Image Shadow Removal Challenge and wins 2nd place overall.

cs.CV↗

arXiv2023

Determination of parameters of a potential model for tetraquark study by studying all S-wave mesons

The masses of low-lying S-wave mesons are evaluated in a constituent quark model (CQM) where the Cornell-like potential and one-gluon exchange spin-spin interaction are employed. To make the model applicable to both the light and heavy quark sectors, we introduce mass-dependent coupling coefficients. There are four free parameters in the model, which are determined by comparing the theoretical results with experimental data. The established model with one set of parameters may be applied to study higher excited meson states as well as multiquark systems in both the light and heavy quark sectors.

hep-ph↗

arXiv2023

Study of light tetraquark spectroscopy

We calculate the masses of the $qq\bar q\bar q$ tetraquark ground state and first radial excited state in a constituent quark model where the Cornell-like potential and one-gluon exchange spin-spin coupling are employed. The three coupling parameters for the Cornell-like potential and one-gluon exchange spin-spin coupling are proposed mass-dependent in accordance with Lattice QCD data, and all model parameters are predetermined by studying light, charmed and bottom mesons. The theoretical predictions for light tetraquarks are compared with the observed exotic meson states in the light-unflavored meson sector, and tentative assignments are suggested. The work suggests that the $f_0(1500)$ and $f_0(1710)$ might be ground light tetraquark states with $J=0$.

hep-ph↗

arXiv2023

Stochastic filtering with moment representation

Stochastic filtering refers to estimating the probability distribution of the latent stochastic process conditioned on the observed measurements in time. In this paper, we introduce a new class of convergent filters that represent the filtering distributions by their moments. The key enablement is a quadrature method that uses orthonormal polynomials spanned by the moments. We prove that this moment-based filter is asymptotically exact in the order of moments, and show that the filter is also computationally efficient and is in line with the state of the art.

stat.ME↗

arXiv2023

Advance in Reversible Covalent Kinase Inhibitors

Reversible covalent kinase inhibitors (RCKIs) are a class of novel kinase inhibitors attracting increasing attention because they simultaneously show the selectivity of covalent kinase inhibitors, yet avoid permanent protein-modification-induced adverse effects. Over the last decade, RCKIs have been reported to target different kinases, including atypical kinases. Currently, three RCKIs are undergoing clinical trials to treat specific diseases, for example, Pemphigus, an autoimmune disorder. In this perspective, first, RCKIs are systematically summarized, including characteristics of electrophilic groups, chemical scaffolds, nucleophilic residues, and binding modes. Second, we provide insights into privileged electrophiles, the distribution of nucleophiles and hence effective design strategies for RCKIs. Finally, we provide a brief perspective on future design strategies for RCKIs, including those that target proteins other than kinases.

q-bio.MN↗

arXiv2023

Probabilistic Estimation of Instantaneous Frequencies of Chirp Signals

We present a continuous-time probabilistic approach for estimating the chirp signal and its instantaneous frequency function when the true forms of these functions are not accessible. Our model represents these functions by non-linearly cascaded Gaussian processes represented as non-linear stochastic differential equations. The posterior distribution of the functions is then estimated with stochastic filters and smoothers. We compute a (posterior) Cramér--Rao lower bound for the Gaussian process model, and derive a theoretical upper bound for the estimation error in the mean squared sense. The experiments show that the proposed method outperforms a number of state-of-the-art methods on a synthetic data. We also show that the method works out-of-the-box for two real-world datasets.

stat.ML↗

arXiv2022

Understanding Domain Learning in Language Models Through Subpopulation Analysis

We investigate how different domains are encoded in modern neural network architectures. We analyze the relationship between natural language domains, model size, and the amount of training data used. The primary analysis tool we develop is based on subpopulation analysis with Singular Vector Canonical Correlation Analysis (SVCCA), which we apply to Transformer-based language models (LMs). We compare the latent representations of such a language model at its different layers from a pair of models: a model trained on multiple domains (an experimental model) and a model trained on a single domain (a control model). Through our method, we find that increasing the model capacity impacts how domain information is stored in upper and lower layers differently. In addition, we show that larger experimental models simultaneously embed domain-specific information as if they were conjoined control models. These findings are confirmed qualitatively, demonstrating the validity of our method.

cs.CL↗

arXiv2022

A Unified Model for Reverse Dictionary and Definition Modelling

We build a dual-way neural dictionary to retrieve words given definitions, and produce definitions for queried words. The model learns the two tasks simultaneously and handles unknown words via embeddings. It casts a word or a definition to the same representation space through a shared layer, then generates the other form in a multi-task fashion. Our method achieves promising automatic scores on previous benchmarks without extra resources. Human annotators prefer the model's outputs in both reference-less and reference-based evaluation, indicating its practicality. Analysis suggests that multiple objectives benefit learning.

cs.CL↗

arXiv2022

A possible interpretation of $Λ$ baryon spectrum with pentaquark components

The $Λ$ baryon spectrum is studied within the SU(3) flavor symmetry in a constituent quark model. We found that it is rather difficult to accommodate some negative-parity $Λ$ resonances as single $q^2s$ ($q = u,\,d$ quarks) states in the conventional three-quark picture. The ground $q^3s\bar q$ pentaquark mass spectrum is evaluated and a possible interpretation is proposed in the work: the observed $Λ(1405)1/2^{-}$, $Λ(1670)1/2^{-}$ and $Λ(1800)1/2^{-}$ are three-state mixtures of two $p$-wave $q^2s$ states and one ground $q^3s\bar q$ pentaquark state, so are the $Λ(1520)3/2^{-}$, $Λ(1690)3/2^{-}$ and $Λ(2050)3/2^{-}$ resonances.

hep-ph↗

arXiv2022

Multidimensional Projection Filters via Automatic Differentiation and Sparse-Grid Integration

The projection filter is a technique for approximating the solutions of optimal filtering problems. In projection filters, the Kushner--Stratonovich stochastic partial differential equation that governs the propagation of the optimal filtering density is projected to a manifold of parametric densities, resulting in a finite-dimensional stochastic differential equation. Despite the fact that projection filters are capable of representing complicated probability densities, their current implementations are limited to Gaussian family or unidimensional filtering applications. This work considers a combination of numerical integration and automatic differentiation to construct projection filter algorithms for more generic problems. Specifically, we provide a detailed exposition of this combination for the manifold of the exponential family, and show how to apply the projection filter to multidimensional cases. We demonstrate numerically that based on comparison to a finite-difference solution to the Kushner--Stratonovich equation and a bootstrap particle filter with systematic resampling, the proposed algorithm retains an accurate approximation of the filtering density while requiring a comparatively low number of quadrature points. Due to the sparse-grid integration and automatic differentiation used to calculate the expected values of the natural statistics and the Fisher metric, the proposed filtering algorithms are highly scalable. They therefore are suitable to many applications in which the number of dimensions exceeds the practical limit of particle filters, but where the Gaussian-approximations are deemed unsatisfactory.

math.OC↗

arXiv2022

To Adapt or to Fine-tune: A Case Study on Abstractive Summarization

Recent advances in the field of abstractive summarization leverage pre-trained language models rather than train a model from scratch. However, such models are sluggish to train and accompanied by a massive overhead. Researchers have proposed a few lightweight alternatives such as smaller adapters to mitigate the drawbacks. Nonetheless, it remains uncertain whether using adapters benefits the task of summarization, in terms of improved efficiency without an unpleasant sacrifice in performance. In this work, we carry out multifaceted investigations on fine-tuning and adapters for summarization tasks with varying complexity: language, domain, and task transfer. In our experiments, fine-tuning a pre-trained language model generally attains a better performance than using adapters; the performance gap positively correlates with the amount of training data used. Notably, adapters exceed fine-tuning under extremely low-resource conditions. We further provide insights on multilinguality, model convergence, and robustness, hoping to shed light on the pragmatic choice of fine-tuning or adapters in abstractive summarization.

cs.CL↗

arXiv2022

A compact butterfly-style silicon photonic-electronic neural chip for hardware-efficient deep learning

The optical neural network (ONN) is a promising hardware platform for next-generation neurocomputing due to its high parallelism, low latency, and low energy consumption. Previous ONN architectures are mainly designed for general matrix multiplication (GEMM), leading to unnecessarily large area cost and high control complexity. Here, we move beyond classical GEMM-based ONNs and propose an optical subspace neural network (OSNN) architecture, which trades the universality of weight representation for lower optical component usage, area cost, and energy consumption. We devise a butterfly-style photonic-electronic neural chip to implement our OSNN with up to 7x fewer trainable optical components compared to GEMM-based ONNs. Additionally, a hardware-aware training framework is provided to minimize the required device programming precision, lessen the chip area, and boost the noise robustness. We experimentally demonstrate the utility of our neural chip in practical image recognition tasks, showing that a measured accuracy of 94.16% can be achieved in hand-written digit recognition tasks with 3-bit weight programming precision.

cs.ET↗

arXiv2022

Revisiting Shallow Discourse Parsing in the PDTB-3: Handling Intra-sentential Implicits

In the PDTB-3, several thousand implicit discourse relations were newly annotated \textit{within} individual sentences, adding to the over 15,000 implicit relations annotated \textit{across} adjacent sentences in the PDTB-2. Given that the position of the arguments to these \textit{intra-sentential implicits} is no longer as well-defined as with \textit{inter-sentential implicits}, a discourse parser must identify both their location and their sense. That is the focus of the current work. The paper provides a comprehensive analysis of our results, showcasing model performance under different scenarios, pointing out limitations and noting future directions.

cs.CL↗

arXiv2021

State-space deep Gaussian processes with applications

This thesis is mainly concerned with state-space approaches for solving deep (temporal) Gaussian process (DGP) regression problems. More specifically, we represent DGPs as hierarchically composed systems of stochastic differential equations (SDEs), and we consequently solve the DGP regression problem by using state-space filtering and smoothing methods. The resulting state-space DGP (SS-DGP) models generate a rich class of priors compatible with modelling a number of irregular signals/functions. Moreover, due to their Markovian structure, SS-DGPs regression problems can be solved efficiently by using Bayesian filtering and smoothing methods. The second contribution of this thesis is that we solve continuous-discrete Gaussian filtering and smoothing problems by using the Taylor moment expansion (TME) method. This induces a class of filters and smoothers that can be asymptotically exact in predicting the mean and covariance of stochastic differential equations (SDEs) solutions. Moreover, the TME method and TME filters and smoothers are compatible with simulating SS-DGPs and solving their regression problems. Lastly, this thesis features a number of applications of state-space (deep) GPs. These applications mainly include, (i) estimation of unknown drift functions of SDEs from partially observed trajectories and (ii) estimation of spectro-temporal features of signals.

stat.ME↗

arXiv2021

Non-linear Gaussian smoothing with Taylor moment expansion

This letter is concerned with solving continuous-discrete Gaussian smoothing problems by using the Taylor moment expansion (TME) scheme. In the proposed smoothing method, we apply the TME method to approximate the transition density of the stochastic differential equation in the dynamic model. Furthermore, we derive a theoretical error bound (in the mean square sense) of the TME smoothing estimates showing that the smoother is stable under weak assumptions. Numerical experiments are presented in order to illustrate practical use of the method.

math.NA↗

arXiv2021

Deep State-Space Gaussian Processes

This paper is concerned with a state-space approach to deep Gaussian process (DGP) regression. We construct the DGP by hierarchically putting transformed Gaussian process (GP) priors on the length scales and magnitudes of the next level of Gaussian processes in the hierarchy. The idea of the state-space approach is to represent the DGP as a non-linear hierarchical system of linear stochastic differential equations (SDEs), where each SDE corresponds to a conditional GP. The DGP regression problem then becomes a state estimation problem, and we can estimate the state efficiently with sequential methods by using the Markov property of the state-space DGP. The computational complexity scales linearly with respect to the number of measurements. Based on this, we formulate state-space MAP as well as Bayesian filtering and smoothing solutions to the DGP regression problem. We demonstrate the performance of the proposed models and methods on synthetic non-stationary signals and apply the state-space DGP to detection of the gravitational waves from LIGO measurements.

stat.ML↗

arXiv2021

Efficient On-Chip Learning for Optical Neural Networks Through Power-Aware Sparse Zeroth-Order Optimization

Optical neural networks (ONNs) have demonstrated record-breaking potential in high-performance neuromorphic computing due to their ultra-high execution speed and low energy consumption. However, current learning protocols fail to provide scalable and efficient solutions to photonic circuit optimization in practical applications. In this work, we propose a novel on-chip learning framework to release the full potential of ONNs for power-efficient in situ training. Instead of deploying implementation-costly back-propagation, we directly optimize the device configurations with computation budgets and power constraints. We are the first to model the ONN on-chip learning as a resource-constrained stochastic noisy zeroth-order optimization problem, and propose a novel mixed-training strategy with two-level sparsity and power-aware dynamic pruning to offer a scalable on-chip training solution in practical ONN deployment. Compared with previous methods, we are the first to optimize over 2,500 optical components on chip. We can achieve much better optimization stability, 3.7x-7.6x higher efficiency, and save >90% power under practical device variations and thermal crosstalk.

cs.ET↗