arXiv subjects

Jiacheng Sun

Publications and source records attributed to Jiacheng Sun.

At least 73 records · Page 4Linked to original sources

arXiv2023

Elucidating The Design Space of Classifier-Guided Diffusion Generation

Guidance in conditional diffusion generation is of great importance for sample quality and controllability. However, existing guidance schemes are to be desired. On one hand, mainstream methods such as classifier guidance and classifier-free guidance both require extra training with labeled data, which is time-consuming and unable to adapt to new conditions. On the other hand, training-free methods such as universal guidance, though more flexible, have yet to demonstrate comparable performance. In this work, through a comprehensive investigation into the design space, we show that it is possible to achieve significant performance improvements over existing guidance schemes by leveraging off-the-shelf classifiers in a training-free fashion, enjoying the best of both worlds. Employing calibration as a general guideline, we propose several pre-conditioning techniques to better exploit pretrained off-the-shelf classifiers for guiding diffusion generation. Extensive experiments on ImageNet validate our proposed method, showing that state-of-the-art diffusion models (DDPM, EDM, DiT) can be further improved (up to 20%) using off-the-shelf classifiers with barely any extra computational cost. With the proliferation of publicly available pretrained classifiers, our proposed approach has great potential and can be readily scaled up to text-to-image generation tasks. The code is available at https://github.com/AlexMaOLS/EluCD/tree/main.

cs.LG↗

arXiv2023

Training Energy-Based Models with Diffusion Contrastive Divergences

Energy-Based Models (EBMs) have been widely used for generative modeling. Contrastive Divergence (CD), a prevailing training objective for EBMs, requires sampling from the EBM with Markov Chain Monte Carlo methods (MCMCs), which leads to an irreconcilable trade-off between the computational burden and the validity of the CD. Running MCMCs till convergence is computationally intensive. On the other hand, short-run MCMC brings in an extra non-negligible parameter gradient term that is difficult to handle. In this paper, we provide a general interpretation of CD, viewing it as a special instance of our proposed Diffusion Contrastive Divergence (DCD) family. By replacing the Langevin dynamic used in CD with other EBM-parameter-free diffusion processes, we propose a more efficient divergence. We show that the proposed DCDs are both more computationally efficient than the CD and are not limited to a non-negligible gradient term. We conduct intensive experiments, including both synthesis data modeling and high-dimensional image denoising and generation, to show the advantages of the proposed DCDs. On the synthetic data learning and image denoising experiments, our proposed DCD outperforms CD by a large margin. In image generation experiments, the proposed DCD is capable of training an energy-based model for generating the Celab-A $32\times 32$ dataset, which is comparable to existing EBMs.

cs.LG↗

arXiv2023

Topological and stacked flat bands in bilayer graphene with a superlattice potential

We show that bilayer graphene in the presence of a 2D superlattice potential provides a highly tunable setup that can realize a variety of flat band phenomena. We focus on two regimes: (i) topological flat bands with non-zero Chern numbers, C, including bands with higher Chern numbers |C| > 1; and (ii) an unprecedented phase consisting of a stack of nearly perfect flat bands with C = 0. For realistic values of the potential and superlattice periodicity, this stack can span nearly 100 meV, encompassing nearly all of the low-energy spectrum. We further show that in the topological regime, the topological flat band has a favorable band geometry for realizing a fractional Chern insulator (FCI) and use exact diagonalization to show that the FCI is in fact the ground state at 1/3 filling. Our results provide a realistic guide for future experiments to realize a new platform for flat band phenomena.

cond-mat.mes-hall↗

arXiv2023

Signature of Correlated Insulator in Electric Field Controlled Superlattice

The Bloch electron energy spectrum of a crystalline solid is determined by the underlying lattice structure at the atomic level. In a 2-dimensional (2d) crystal it is possible to impose a superlattice with nanometer-scale periodicity, allowing to tune the fundamental Bloch electron spectrum, and enabling novel physical properties which are not accessible in the original crystal. In recent years, a top-down approach for creating 2d superlattices on monolayer graphene by means of nanopatterned electric gates has been studied, which allows the formation of isolated energy bands and Hofstadter Butterfly physics in quantizing magnetic fields. Within this approach, however, evidence of electron correlations which drive many problems at the forefront of physics research remains to be uncovered. In this work we demonstrate signatures of a correlated insulator phase in Bernal-stacked bilayer graphene (BLG) modulated by a gate-defined superlattice potential, manifested as a set of resistance peaks centered at carrier densities of integer multiples of a single electron per unit cell of the superlattice potential. We associate the correlated insulator phase to the formation of flat energy bands due to the superlattice potential combined with inversion symmetry breaking. Inducing correlated electron phases with nanopatterning defined electric gates paves the way to custom-designed superlattices with arbitrary geometries and symmetries for studying band structure engineering and strongly correlated electrons in 2d materials.

cond-mat.str-el↗

arXiv2023

On the Generalization of Diffusion Model

The diffusion probabilistic generative models are widely used to generate high-quality data. Though they can synthetic data that does not exist in the training set, the rationale behind such generalization is still unexplored. In this paper, we formally define the generalization of the generative model, which is measured by the mutual information between the generated data and the training set. The definition originates from the intuition that the model which generates data with less correlation to the training set exhibits better generalization ability. Meanwhile, we show that for the empirical optimal diffusion model, the data generated by a deterministic sampler are all highly related to the training set, thus poor generalization. This result contradicts the observation of the trained diffusion model's (approximating empirical optima) extrapolation ability (generating unseen data). To understand this contradiction, we empirically verify the difference between the sufficiently trained diffusion model and the empirical optima. We found, though obtained through sufficient training, there still exists a slight difference between them, which is critical to making the diffusion model generalizable. Moreover, we propose another training objective whose empirical optimal solution has no potential generalization problem. We empirically show that the proposed training objective returns a similar model to the original one, which further verifies the generalization ability of the trained diffusion model.

cs.LG↗

arXiv2023

Fast-speed and low-power-consumption optical phased array based on thin-film lithium niobate platform

Fast scanning-speed and low-power-consumption are becoming progressively more and more important in realizing high-performance chiplet optical phased arrays (OPAs). Here, we establish an integrated OPA based on thin-film lithium niobate-on-insulator (LNOI) platform to access these outstanding performances. Significantly, a lithium niobate (LN) OPA chip is implemented by 32/48 channels LN waveguides enabled by electro-optic modulations, which showcases the low power consumption (1.11nJ/π}) and fast operation speed (14.4 ns) promising the advantage of the LNOI platform for integrated OPAs. As results, we experimentally achieved a beam steering with a 62.2°*8.8° field of view (FOV) and a beam divergence of 2.4°*1.2°. Moreover, by employing sparse aperiodic arrays in waveguides design we obtained a significant reduction of lateral divergence to 0.33° for the radiation beam. This work demonstrate that remarkable advantage of LNOI platform for power-saving and scalable OPA chips for various applications.

physics.optics↗

arXiv2022

Why Are Conditional Generative Models Better Than Unconditional Ones?

Extensive empirical evidence demonstrates that conditional generative models are easier to train and perform better than unconditional ones by exploiting the labels of data. So do score-based diffusion models. In this paper, we analyze the phenomenon formally and identify that the key of conditional learning is to partition the data properly. Inspired by the analyses, we propose self-conditioned diffusion models (SCDM), which is trained conditioned on indices clustered by the k-means algorithm on the features extracted by a model pre-trained in a self-supervised manner. SCDM significantly improves the unconditional model across various datasets and achieves a record-breaking FID of 3.94 on ImageNet 64x64 without labels. Besides, SCDM achieves a slightly better FID than the corresponding conditional model on CIFAR10.

cs.LG↗

arXiv2022

Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models

Diffusion probabilistic models (DPMs) are a class of powerful deep generative models (DGMs). Despite their success, the iterative generation process over the full timesteps is much less efficient than other DGMs such as GANs. Thus, the generation performance on a subset of timesteps is crucial, which is greatly influenced by the covariance design in DPMs. In this work, we consider diagonal and full covariances to improve the expressive power of DPMs. We derive the optimal result for such covariances, and then correct it when the mean of DPMs is imperfect. Both the optimal and the corrected ones can be decomposed into terms of conditional expectations over functions of noise. Building upon it, we propose to estimate the optimal covariance and its correction given imperfect mean by learning these conditional expectations. Our method can be applied to DPMs with both discrete and continuous timesteps. We consider the diagonal covariance in our implementation for computational efficiency. For an efficient practical implementation, we adopt a parameter sharing scheme and a two-stage training process. Empirically, our method outperforms a wide variety of covariance design on likelihood results, and improves the sample quality especially on a small number of timesteps.

cs.LG↗

arXiv2021

Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks

Gradient-based methods for the distributed training of residual networks (ResNets) typically require a forward pass of the input data, followed by back-propagating the error gradient to update model parameters, which becomes time-consuming as the network goes deeper. To break the algorithmic locking and exploit synchronous module parallelism in both the forward and backward modes, auxiliary-variable methods have attracted much interest lately but suffer from significant communication overhead and lack of data augmentation. In this work, a novel joint learning framework for training realistic ResNets across multiple compute devices is established by trading off the storage and recomputation of external auxiliary variables. More specifically, the input data of each independent processor is generated from its low-capacity auxiliary network (AuxNet), which permits the use of data augmentation and realizes forward unlocking. The backward passes are then executed in parallel, each with a local loss function that originates from the penalty or augmented Lagrangian (AL) methods. Finally, the proposed AuxNet is employed to reproduce the updated auxiliary variables through an end-to-end training process. We demonstrate the effectiveness of our methods on ResNets and WideResNets across CIFAR-10, CIFAR-100, and ImageNet datasets, achieving speedup over the traditional layer-serial training method while maintaining comparable testing accuracy.

cs.LG↗

arXiv2021

Towards Understanding the Generative Capability of Adversarially Robust Classifiers

Recently, some works found an interesting phenomenon that adversarially robust classifiers can generate good images comparable to generative models. We investigate this phenomenon from an energy perspective and provide a novel explanation. We reformulate adversarial example generation, adversarial training, and image generation in terms of an energy function. We find that adversarial training contributes to obtaining an energy function that is flat and has low energy around the real data, which is the key for generative capability. Based on our new understanding, we further propose a better adversarial training method, Joint Energy Adversarial Training (JEAT), which can generate high-quality images and achieve new state-of-the-art robustness under a wide range of attacks. The Inception Score of the images (CIFAR-10) generated by JEAT is 8.80, much better than original robust classifiers (7.50).

cs.LG↗

arXiv2021

Improved OOD Generalization via Adversarial Training and Pre-training

Recently, learning a model that generalizes well on out-of-distribution (OOD) data has attracted great attention in the machine learning community. In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data. Inspired by previous findings that adversarial training helps improve input-robustness, we theoretically show that adversarially trained models have converged excess risk on OOD data, and empirically verify it on both image classification and natural language understanding tasks. Besides, in the paradigm of first pre-training and then fine-tuning, we theoretically show that a pre-trained model that is more robust to input perturbation provides a better initialization for generalization on downstream OOD data. Empirically, after fine-tuning, this better-initialized model from adversarial pre-training also has better OOD generalization.

cs.LG↗

arXiv2021

A Practical Layer-Parallel Training Algorithm for Residual Networks

Gradient-based algorithms for training ResNets typically require a forward pass of the input data, followed by back-propagating the objective gradient to update parameters, which are time-consuming for deep ResNets. To break the dependencies between modules in both the forward and backward modes, auxiliary-variable methods such as the penalty and augmented Lagrangian (AL) approaches have attracted much interest lately due to their ability to exploit layer-wise parallelism. However, we observe that large communication overhead and lacking data augmentation are two key challenges of these methods, which may lead to low speedup ratio and accuracy drop across multiple compute devices. Inspired by the optimal control formulation of ResNets, we propose a novel serial-parallel hybrid training strategy to enable the use of data augmentation, together with downsampling filters to reduce the communication cost. The proposed strategy first trains the network parameters by solving a succession of independent sub-problems in parallel and then corrects the network parameters through a full serial forward-backward propagation of data. Such a strategy can be applied to most of the existing layer-parallel training methods using auxiliary variables. As an example, we validate the proposed strategy using penalty and AL methods on ResNet and WideResNet across MNIST, CIFAR-10 and CIFAR-100 datasets, achieving significant speedup over the traditional layer-serial training methods while maintaining comparable accuracy.

cs.LG↗

arXiv2020

Batch Group Normalization

Deep Convolutional Neural Networks (DCNNs) are hard and time-consuming to train. Normalization is one of the effective solutions. Among previous normalization methods, Batch Normalization (BN) performs well at medium and large batch sizes and is with good generalizability to multiple vision tasks, while its performance degrades significantly at small batch sizes. In this paper, we find that BN saturates at extreme large batch sizes, i.e., 128 images per worker, i.e., GPU, as well and propose that the degradation/saturation of BN at small/extreme large batch sizes is caused by noisy/confused statistic calculation. Hence without adding new trainable parameters, using multiple-layer or multi-iteration information, or introducing extra computation, Batch Group Normalization (BGN) is proposed to solve the noisy/confused statistic calculation of BN at small/extreme large batch sizes with introducing the channel, height and width dimension to compensate. The group technique in Group Normalization (GN) is used and a hyper-parameter G is used to control the number of feature instances used for statistic calculation, hence to offer neither noisy nor confused statistic for different batch sizes. We empirically demonstrate that BGN consistently outperforms BN, Instance Normalization (IN), Layer Normalization (LN), GN, and Positional Normalization (PN), across a wide spectrum of vision tasks, including image classification, Neural Architecture Search (NAS), adversarial learning, Few Shot Learning (FSL) and Unsupervised Domain Adaptation (UDA), indicating its good performance, robust stability to batch size and wide generalizability. For example, for training ResNet-50 on ImageNet with a batch size of 2, BN achieves Top1 accuracy of 66.512% while BGN achieves 76.096% with notable improvement.

cs.LG↗

arXiv2020

DARTS+: Improved Differentiable Architecture Search with Early Stopping

Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of "{\em skip-connect}s" are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exists overfitting in the optimization of DARTS. Therefore, we propose a simple and effective algorithm, named "DARTS+", to avoid the collapse and improve the original DARTS, by "early stopping" the search procedure when meeting a certain criterion. We also conduct comprehensive experiments on benchmark datasets and different search spaces and show the effectiveness of our DARTS+ algorithm, and DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of "early stopping" is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an {\em explicit} criterion for "early stopping".

cs.CV↗

arXiv2020

High-Q plasmonic crystal laser for ultra-sensitive biomolecule detection

Plasmonic lasers provide a paradigm-changing approach for the generation of coherent light at the nanoscale. In addition to the usual properties of coherent radiation, the emission of plasmonic lasers can feature high sensitivity to the surrounding environment, which makes this technology attractive for developing high-performance and highly-integrated sensing devices. Here, we investigate a plasmonic laser architecture based on a high-Q plasmonic crystal consisting of a periodic arrangement of nanoholes on a thin gold film cladded with an organic-dye-doped SiO$_2$ gain layer as the gain material. We report an extensive full-wave numerical analysis of the device's lasing performance and its application as a biochemical sensor, showing that the proposed design features excellent figures of merit for surface sensing that in principle can be over an order of magnitude larger than those of previously reported high-performance plasmonic biosensor architectures.

physics.optics↗

arXiv2020

New Interpretations of Normalization Methods in Deep Learning

In recent years, a variety of normalization methods have been proposed to help train neural networks, such as batch normalization (BN), layer normalization (LN), weight normalization (WN), group normalization (GN), etc. However, mathematical tools to analyze all these normalization methods are lacking. In this paper, we first propose a lemma to define some necessary tools. Then, we use these tools to make a deep analysis on popular normalization methods and obtain the following conclusions: 1) Most of the normalization methods can be interpreted in a unified framework, namely normalizing pre-activations or weights onto a sphere; 2) Since most of the existing normalization methods are scaling invariant, we can conduct optimization on a sphere with scaling symmetry removed, which can help stabilize the training of network; 3) We prove that training with these normalization methods can make the norm of weights increase, which could cause adversarial vulnerability as it amplifies the attack. Finally, a series of experiments are conducted to verify these claims.

cs.LG↗

arXiv2019

Dynamics of a micro-VCSEL operated in the threshold region under low-level optical feedback

Semiconductor lasers are notoriously sensitive to optical feedback, and their dynamics and coherence can be significantly modified through optical reinjection. We concentrate on the dynamical properties of a very small (i.e., microscale) Vertical Cavity Surface Emitting Laser (VCSEL) operated in the low coherence region between the emission of (partially) coherent pulses and ending below the accepted macroscopic lasing threshold, with the double objective of: 1. studying the feedback influence in a regime of very low energy consumption; 2. using the micro-VCSEL as a surrogate for nanolasers, where measurements can only be based on photon statistics. The experimental investigation is based on time traces and radiofrequency spectra (common for macroscale devices) and correlation functions (required at the nanoscale). Comparison of these results confirms the ability of correlation functions to satisfactorily characterize the action of feedback on the laser dynamics. Numerical predictions obtained from a previously developed, fully stochastic modeling technique provide very close agreement with the experimental observations, thus supporting the possible extension of our observations to the nanoscale.

physics.optics↗

arXiv2017

Uniqueness of the Mean Field Equation and Rigidity of Hawking Mass

In this paper, we prove that the even solution of the mean field equation $Δu=λ(1-e^u) $ on $S^2$ must be axially symmetric when $4<λ\leq 8$. In particular, zero is the only even solution for $λ=6$. This implies the rigidity of Hawking mass for stable constant mean curvature(CMC) sphere with even symmetry.

math.AP↗