arXiv subjects

Nan Ding

Publications and source records attributed to Nan Ding.

At least 37 records · Page 2Linked to original sources

arXiv2024

CausalLM is not optimal for in-context learning

Recent empirical evidence indicates that transformer based in-context learning performs better when using a prefix language model (prefixLM), in which in-context samples can all attend to each other, compared to causal language models (causalLM), which use auto-regressive attention that prohibits in-context samples to attend to future samples. While this result is intuitive, it is not understood from a theoretical perspective. In this paper we take a theoretical approach and analyze the convergence behavior of prefixLM and causalLM under a certain parameter construction. Our analysis shows that both LM types converge to their stationary points at a linear rate, but that while prefixLM converges to the optimal solution of linear regression, causalLM convergence dynamics follows that of an online gradient descent algorithm, which is not guaranteed to be optimal even as the number of samples grows infinitely. We supplement our theoretical claims with empirical experiments over synthetic and real tasks and using various types of transformers. Our experiments verify that causalLM consistently underperforms prefixLM in all settings.

cs.LG↗

arXiv2024

Transient quasi-periodic oscillations in the gamma-ray light curves of bright blazars

Transient quasi-periodic oscillations (QPOs) are extremely interesting observational phenomena. However, the precise physical mechanisms leading to their generation are still hotly debated. We performed a systematic search for transient QPO signals using Weighted Wavelet Z-transforms on the gamma-ray light curves of 134 bright blazars with peak flux exceeding $1\times10^{-6}$~ph~cm$^{-2}$~s$^{-1}$ as monitored by Fermi-LAT. Artificial light curves were generated from the power spectral density and probability distribution functions of the original light curves to assess the significance level of transient QPO. We discuss several physical mechanisms that produce transient QPOs, with the helical jet model providing the best explanation. This study identified four new transient QPO events. Interestingly, repetitive transient QPOs are observed in PKS 0537-441, and nested transient QPOs are detected in PKS 1424-41. Additionally, we find that transient QPOs tend to occur in the flare state of the blazar. Finally, we estimate the incidence of transient QPO events to be only about 3\%.

astro-ph.HE↗

arXiv2023

The comparison of optical variability of broad-line Seyfert 1 and narrow-line Seyfert 1 galaxies from the view of Pan-STARRS

By means of the data sets of the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS), we investigate the relationship between the variability amplitude and luminosity at 5100 Å, black hole mass, Eddington ratio, $ R_{\rm Fe \, II}$ ( the ratio of the flux of Fe II line within 4435-4685 Å~to the broad proportion of $\rm Hβ$ line) as well as $ R_{5007}$ (the ratio of the flux [O III] line to the total $\rm Hβ$ line) of the broad line Seyfert 1 (BLS1) and narrow line Seyfert 1 (NLS1) galaxies sample in g,r,i,z and y bands, respectively. We also analyze the similarities and differences of the variability characteristics between the BLS1 galaxies and NLS1 galaxies. The results are listed as follows. (1). The cumulative probability distribution of the variability amplitude shows that NLS1 galaxies are lower than that in BLS1 galaxies. (2). We analyze the dependence of the variability amplitude with the luminosity at 5100 Å, black hole mass, Eddington ratio, $ R_{\rm Fe \,II}$ and $ R_{5007}$, respectively. We find significantly negative correlations between the variability amplitude and Eddington ratio, insignificant correlations with the luminosity at 5100 Å. The results also show significantly positive correlations with the black hole mass and $ R_{5007}$, significantly negative correlations with $ R_{\rm Fe\, II}$ which are consistent with Rakshit and Stalin(2017) in low redshift bins (z<0.4) and Ai et al.(2010). (3). The relationship between the variability amplitude and the radio loudness is investigated for 155 BLS1 galaxies and 188 NLS1 galaxies. No significant correlations are found in our results.

astro-ph.HE↗

arXiv2023

General Physical Properties of Fermi blazars

We study the general physical properties of Fermi blazars using the Fermi fourth source catalog data (4FGL-DR2). The quasi-simultaneous multiwavelength data of Fermi blazar are fitted by using the one-zone leptonic model to obtain some physical parameters, such as jet power, magnetic field and Doppler factor. We study the distributions of the derived physical parameter as a function of black hole mass and accretion disk luminosity. The main results are as follows. (1) For a standard thin accretion disk, the jet kinetic power of most FSRQs can be explained by the BP mechanism. However, the jet kinetic power of most BL Lacs can not be explained by both the BZ mechanism or the BP mechanism. The BL Lacs may have ADAFs surrounding their massive black holes. (2) After excluding the redshift, there is a moderately strong correlation between the jet kinetic power and jet radiation power and the accretion disk luminosity for Fermi blazars. These results confirm a close connection between jet and accretion. The jet kinetic power is slightly larger than the accretion disk luminosity for Fermi blazars. (3) There is a significant correlation between jet kinetic power and gamma-ray luminosity and radio luminosity for Fermi blazars, which suggests that gamma-ray luminosity and radio luminosity can be used to indicate the jet kinetic power.

astro-ph.HE↗

arXiv2023

Evaluating the Potential of Disaggregated Memory Systems for HPC applications

Disaggregated memory is a promising approach that addresses the limitations of traditional memory architectures by enabling memory to be decoupled from compute nodes and shared across a data center. Cloud platforms have deployed such systems to improve overall system memory utilization, but performance can vary across workloads. High-performance computing (HPC) is crucial in scientific and engineering applications, where HPC machines also face the issue of underutilized memory. As a result, improving system memory utilization while understanding workload performance is essential for HPC operators. Therefore, learning the potential of a disaggregated memory system before deployment is a critical step. This paper proposes a methodology for exploring the design space of a disaggregated memory system. It incorporates key metrics that affect performance on disaggregated memory systems: memory capacity, local and remote memory access ratio, injection bandwidth, and bisection bandwidth, providing an intuitive approach to guide machine configurations based on technology trends and workload characteristics. We apply our methodology to analyze thirteen diverse workloads, including AI training, data analysis, genomics, protein, fusion, atomic nuclei, and traditional HPC bookends. Our methodology demonstrates the ability to comprehend the potential and pitfalls of a disaggregated memory system and provides motivation for machine configurations. Our results show that eleven of our thirteen applications can leverage injection bandwidth disaggregated memory without affecting performance, while one pays a rack bisection bandwidth penalty and two pay the system-wide bisection bandwidth penalty. In addition, we also show that intra-rack memory disaggregation would meet the application's memory requirement and provide enough remote memory bandwidth.

cs.DC↗

arXiv2023

PaLI: A Jointly-Scaled Multilingual Language-Image Model

Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design.

cs.CV↗

arXiv2023

General Physical Properties of Gamma-Ray-emitting Radio Galaxies

We study the radio galaxies with known redshift detected by the Fermi satellite after 10 years of data (4FGL-DR2). We use a one-zone leptonic model to fit the quasi-simultaneous multiwavelength data of these radio galaxies and study the distributions of the derived physical parameter as a function of black hole mass and accretion disk luminosity. The main results are as follows. (1) We find that the jet kinetic power of most radio galaxies can be explained by the hybrid jet model based on ADAFs surrounding Kerr black holes. (2) After excluding the redshift, there is a significant correlation between the radiation jet power and the accretion disk luminosity, while the jet kinetic power is weakly correlated with the accretion disk luminosity. (3) We also find a significant correlation between inverse Compton luminosity and synchrotron luminosity. The slope of the correlation for radio galaxies is consistent with the synchrotron self-Compton (SSC) process. The result may suggest that the high-energy component of radio galaxies is dominated by the SSC process.

astro-ph.HE↗

arXiv2023

Multiwavelength Analysis of a Nearby Heavily Obscured AGN in NGC 449

We presented the multiwavelength analysis of a heavily obscured active galactic nucleus (AGN) in NGC 449. We first constructed a broadband X-ray spectrum using the latest NuSTAR and XMM-Newton data. Its column density ($\simeq 10^{24} \rm{cm}^{-2}$) and photon index ($Γ\simeq 2.4$) were reliably obtained by analyzing the broadband X-ray spectrum. However, the scattering fraction and the intrinsic X-ray luminosity could not be well constrained. Combined with the information obtained from the mid-infrared (mid-IR) spectrum and spectral energy distribution (SED) fitting, we derived its intrinsic X-ray luminosity ($\simeq 8.54\times 10^{42} \ \rm{erg\ s}^{-1}$) and scattering fraction ($f_{\rm{scat}}\simeq 0.26\%$). In addition, we also derived the following results: (1). The mass accretion rate of central AGN is about $2.54 \times 10^{-2} \rm{M}_\odot\ \rm{yr}^{-1}$, and the Eddington ratio is $8.39\times 10^{-2}$; (2). The torus of this AGN has a high gas-to-dust ratio ($N_{\rm H}/A_{\rm V}=8.40\times 10^{22}\ \rm{cm}^{-2}\ \rm{mag}^{-1}$); (3). The host galaxy and the central AGN are both in the early stage of co-evolution.

astro-ph.GA↗

arXiv2022

Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization

Recent research in robust optimization has shown an overfitting-like phenomenon in which models trained against adversarial attacks exhibit higher robustness on the training set compared to the test set. Although previous work provided theoretical explanations for this phenomenon using a robust PAC-Bayesian bound over the adversarial test error, related algorithmic derivations are at best only loosely connected to this bound, which implies that there is still a gap between their empirical success and our understanding of adversarial robustness theory. To close this gap, in this paper we consider a different form of the robust PAC-Bayesian bound and directly minimize it with respect to the model posterior. The derivation of the optimal solution connects PAC-Bayesian learning to the geometry of the robust loss surface through a Trace of Hessian (TrH) regularizer that measures the surface flatness. In practice, we restrict the TrH regularizer to the top layer only, which results in an analytical solution to the bound whose computational cost does not depend on the network depth. Finally, we evaluate our TrH regularization approach over CIFAR-10/100 and ImageNet using Vision Transformers (ViT) and compare against baseline adversarial robustness algorithms. Experimental results show that TrH regularization leads to improved ViT robustness that either matches or surpasses previous state-of-the-art approaches while at the same time requires less memory and computational cost.

cs.LG↗

arXiv2022

PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks

With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been gaining increased attention. Although several methods have recently been proposed to tackle the selection problem (e.g. LEEP, H-score), these methods resort to applying heuristics that are not well motivated by learning theory. In this paper we present PACTran, a theoretically grounded family of metrics for pretrained model selection and transferability measurement. We first show how to derive PACTran metrics from the optimal PAC-Bayesian bound under the transfer learning setting. We then empirically evaluate three metric instantiations of PACTran on a number of vision tasks (VTAB) as well as a language-and-vision (OKVQA) task. An analysis of the results shows PACTran is a more consistent and effective transferability measure compared to existing selection methods.

cs.LG↗

arXiv2022

All You May Need for VQA are Image Captions

Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation. In this paper, we propose a method that automatically derives VQA examples at volume, by leveraging the abundance of existing image-caption annotations combined with neural models for textual question generation. We show that the resulting data is of high-quality. VQA models trained on our data improve state-of-the-art zero-shot accuracy by double digits and achieve a level of robustness that lacks in the same model trained on human-annotated VQA data.

cs.CV↗

arXiv2021

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

Despite recent advances in its theoretical understanding, there still remains a significant gap in the ability of existing PAC-Bayesian theories on meta-learning to explain performance improvements in the few-shot learning setting, where the number of training examples in the target tasks is severely limited. This gap originates from an assumption in the existing theories which supposes that the number of training examples in the observed tasks and the number of training examples in the target tasks follow the same distribution, an assumption that rarely holds in practice. By relaxing this assumption, we develop two PAC-Bayesian bounds tailored for the few-shot learning setting and show that two existing meta-learning algorithms (MAML and Reptile) can be derived from our bounds, thereby bridging the gap between practice and PAC-Bayesian theories. Furthermore, we derive a new computationally-efficient PACMAML algorithm, and show it outperforms existing meta-learning algorithms on several few-shot benchmark datasets.

cs.LG↗

arXiv2021

Do Transformer Modifications Transfer Across Implementations and Applications?

The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.

cs.LG↗

arXiv2021

Observations of cold gas and star formation in dwarf S0 galaxies

Very little work has been done on star formation in dwarf lenticular galaxies (S0s). We present 2D-spectroscopic and millimetre observations made by Centro Astronomico Hispano Aleman (CAHA) 3.5 m optical and the IRAM-30 m millimetre telescopes, respectively, for a sample of four dwarf S0 galaxies with multiple star formation regions in the field environment. We find that although most of the sources deviate from the star forming main sequence relation, they all follow the Kennicutt-Schmidt law. After comparing the stellar and Halpha kinematics, we find that the velocity fields of both stars and ionized gas do not show regular motion and the velocity dispersions of stars and ionized gas are low in the regions with high star formation, suggesting these star-forming S0 galaxies still have significant rotation. This view can be supported by the result that most of these dwarf S0 galaxies are classified as fast rotators. The ratio of average atomic gas mass to stellar mass (~ 47%) is much greater than that of molecular gas mass to stellar mass (~ 1%). In addition, the gas-phase metallicities in the star-forming regions are lower than that of the non-star-forming regions. These results indicate that the extended star formation may originate from the combination of abundant atomic hydrogen, long dynamic time scale and low-density environment.

astro-ph.GA↗

arXiv2021

The origin of the soft X-ray excess in the narrow-line Seyfert 1 galaxy SBS 1353+564

We present for the first time the timing and spectral analyses for a narrow-line Seyfert 1 galaxy, SBS 1353+564, using \it{XMM-Newton} and \it{Swift} multi-band observations from 2007 to 2019. Our main results are as follows: 1) The temporal variability of SBS 1353+564 is random, while the hardness ratio is relatively constant over a time span of 13 years; 2) We find a prominent soft X-ray excess feature below 2 keV, which cannot be well described by a simple blackbody component; 3) After comparing the two most prevailing models for interpreting the origin of the soft X-ray excess, we find that the relativistically smeared reflection model is unable to fit the data above 5 keV well and the X-ray spectra do not show any reflection features, such as the Fe Kαemission line. However, the warm corona model can obtain a good fitting result. For the warm corona model, we try to use three different sets of spin values to fit the data and derive different best-fitting parameter sets; 4) We compare the UV/optical spectral data with the extrapolated values of the warm corona model to determine which spin value is more appropriate for this source, and we find that the warm corona model with non-spin can sufficiently account for the soft X-ray excess in SBS 1353+564.

astro-ph.HE↗

arXiv2021

Detection of a possible high-confidence radio quasi-periodic oscillation in the BL Lac PKS J2134-0153

We have searched quasi-periodic oscillations (QPOs) for BL Lac PKS J2134-0153 in the 15 GHz radio light curve announced by the Owens Valley Radio Observatory 40-m telescope during the period from 2008-01-05 to 2019-05-18, utilizing the Lomb-Scargle periodogram (LSP) and the weighted wavelet Z-transform (WWZ) techniques. This is the first time that to search for periodic radio signal in BL Lac PKS J2134-0153 by these two methods. These two methods consistently reveal a QPO of 4.69 $\pm$ 0.14 years (>5 $σ$ confidence level). We discuss possible causes for this QPO, and we expected that the binary black holes scenario, where the QPO is caused by the precession of the binary black holes, is the most likely explanation. BL Lac PKS J2134-0153 thus could be a good binary black hole candidate. In the binary black holes scenario, the distance between the primary black hole and the secondary black hole is 1.83$\times$10$^{16}$ cm.

astro-ph.HE↗

arXiv2021

The Powers of Relativistic Jets depend on the Spin of Accreting Supermassive Black Hole

Theoretical models show that the power of relativistic jets of active galactic nuclei depends on the spin and mass of the central supermassive black holes, as well as the accretion. Here we report an analysis of archival observations of a sample of blazars. We find a significant correlation between jet kinetic power and the spin of supermassive black holes. At the same time, we use multiple linear regression to analyze the relationship between jet kinetic power and accretion, spin and black hole mass. We find that the spin of supermassive black holes and accretion are the most important contribution to the jet kinetic power. The contribution rates of both the spin of supermassive black holes and accretion are more than 95\%. These results suggest that the spin energy of supermassive black holes powers the relativistic jets. The jet production efficiency of almost all Fermi blazars can be explained by moderately thin magnetically arrested accretion disks around rapidly spinning black holes.

astro-ph.HE↗

arXiv2021

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pre-training. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pre-training data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [Sharma et al. 2018] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for vision-and-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks.

cs.CV↗