arXiv subjects

Guangyao Chen

Publications and source records attributed to Guangyao Chen.

At least 37 records · Page 2Linked to original sources

arXiv2023

LLaSM: Large Language and Speech Model

Multi-modal large language models have garnered significant interest recently. Though, most of the works focus on vision-language multi-modal models providing strong capabilities in following vision-and-language instructions. However, we claim that speech is also an important modality through which humans interact with the world. Hence, it is crucial for a general-purpose assistant to be able to follow multi-modal speech-and-language instructions. In this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an end-to-end trained large multi-modal speech-language model with cross-modal conversational abilities, capable of following speech-and-language instructions. Our early experiments show that LLaSM demonstrates a more convenient and natural way for humans to interact with artificial intelligence. Specifically, we also release a large Speech Instruction Following dataset LLaSM-Audio-Instructions. Code and demo are available at https://github.com/LinkSoul-AI/LLaSM and https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions dataset is available at https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.

cs.CL↗

arXiv2023

Learning Sparse Neural Networks with Identity Layers

The sparsity of Deep Neural Networks is well investigated to maximize the performance and reduce the size of overparameterized networks as possible. Existing methods focus on pruning parameters in the training process by using thresholds and metrics. Meanwhile, feature similarity between different layers has not been discussed sufficiently before, which could be rigorously proved to be highly correlated to the network sparsity in this paper. Inspired by interlayer feature similarity in overparameterized models, we investigate the intrinsic link between network sparsity and interlayer feature similarity. Specifically, we prove that reducing interlayer feature similarity based on Centered Kernel Alignment (CKA) improves the sparsity of the network by using information bottleneck theory. Applying such theory, we propose a plug-and-play CKA-based Sparsity Regularization for sparse network training, dubbed CKA-SR, which utilizes CKA to reduce feature similarity between layers and increase network sparsity. In other words, layers of our sparse network tend to have their own identity compared to each other. Experimentally, we plug the proposed CKA-SR into the training process of sparse network training methods and find that CKA-SR consistently improves the performance of several State-Of-The-Art sparse training methods, especially at extremely high sparsity. Code is included in the supplementary materials.

cs.LG↗

arXiv2023

Picking Up Quantization Steps for Compressed Image Classification

The sensitivity of deep neural networks to compressed images hinders their usage in many real applications, which means classification networks may fail just after taking a screenshot and saving it as a compressed file. In this paper, we argue that neglected disposable coding parameters stored in compressed files could be picked up to reduce the sensitivity of deep neural networks to compressed images. Specifically, we resort to using one of the representative parameters, quantization steps, to facilitate image classification. Firstly, based on quantization steps, we propose a novel quantization aware confidence (QAC), which is utilized as sample weights to reduce the influence of quantization on network training. Secondly, we utilize quantization steps to alleviate the variance of feature distributions, where a quantization aware batch normalization (QABN) is proposed to replace batch normalization of classification networks. Extensive experiments show that the proposed method significantly improves the performance of classification networks on CIFAR-10, CIFAR-100, and ImageNet. The code is released on https://github.com/LiMaPKU/QSAM.git

cs.CV↗

arXiv2023

Training Full Spike Neural Networks via Auxiliary Accumulation Pathway

Due to the binary spike signals making converting the traditional high-power multiply-accumulation (MAC) into a low-power accumulation (AC) available, the brain-inspired Spiking Neural Networks (SNNs) are gaining more and more attention. However, the binary spike propagation of the Full-Spike Neural Networks (FSNN) with limited time steps is prone to significant information loss. To improve performance, several state-of-the-art SNN models trained from scratch inevitably bring many non-spike operations. The non-spike operations cause additional computational consumption and may not be deployed on some neuromorphic hardware where only spike operation is allowed. To train a large-scale FSNN with high performance, this paper proposes a novel Dual-Stream Training (DST) method which adds a detachable Auxiliary Accumulation Pathway (AAP) to the full spiking residual networks. The accumulation in AAP could compensate for the information loss during the forward and backward of full spike propagation, and facilitate the training of the FSNN. In the test phase, the AAP could be removed and only the FSNN remained. This not only keeps the lower energy consumption but also makes our model easy to deploy. Moreover, for some cases where the non-spike operations are available, the APP could also be retained in test inference and improve feature discrimination by introducing a little non-spike consumption. Extensive experiments on ImageNet, DVS Gesture, and CIFAR10-DVS datasets demonstrate the effectiveness of DST.

cs.NE↗

arXiv2022

OpenOOD: Benchmarking Generalized Out-of-Distribution Detection

Out-of-distribution (OOD) detection is vital to safety-critical machine learning applications and has thus been extensively studied, with a plethora of methods developed in the literature. However, the field currently lacks a unified, strictly formulated, and comprehensive benchmark, which often results in unfair comparisons and inconclusive results. From the problem setting perspective, OOD detection is closely related to neighboring fields including anomaly detection (AD), open set recognition (OSR), and model uncertainty, since methods developed for one domain are often applicable to each other. To help the community to improve the evaluation and advance, we build a unified, well-structured codebase called OpenOOD, which implements over 30 methods developed in relevant fields and provides a comprehensive benchmark under the recently proposed generalized OOD detection framework. With a comprehensive comparison of these methods, we are gratified that the field has progressed significantly over the past few years, where both preprocessing methods and the orthogonal post-hoc methods show strong potential.

cs.CV↗

arXiv2022

Light-front wavefunctions of mesons by design

We develop a mechanism to build the light-front wavefunctions (LFWFs) of meson bound states on a small-sized basis function representation. Unlike in a standard Hamiltonian formalism, the Hamiltonian in this method is implicit, and the information of the system is carried directly by the functional form and adjustable parameters of the LFWFs. In this work, we model the LFWFs for four charmonium states, $η_c$, $J/ψ$, $ψ'$, and $ψ(3770)$ as superpositions of orthonormal basis functions. We choose the basis functions as eigenfunctions of an effective Hamiltonian, which has a longitudinal confining potential in addition to the transverse confining potential from light-front holographic QCD. We determine the basis function parameters and superposition coefficients by employing both guidance from the nonrelativistic description of the meson states and the experimental measurements of the meson decay widths. With the obtained wavefunctions, we study the features of those meson states, including charge radii and parton distribution functions. We use the $J/ψ$ LFWF to calculate the meson production in diffractive deep inelastic scattering and ultra-peripheral heavy-ion collisions, and the $η_c$ LFWF to calculate its diphoton transition form factor. Both results show good agreement with experiments. The obtained LFWFs have simple-functional forms and can be readily used to predict additional experimental observables.

hep-ph↗

arXiv2021

Annotation-Efficient Untrimmed Video Action Recognition

Deep learning has achieved great success in recognizing video actions, but the collection and annotation of training data are still quite laborious, which mainly lies in two aspects: (1) the amount of required annotated data is large; (2) temporally annotating the location of each action is time-consuming. Works such as few-shot learning or untrimmed video recognition have been proposed to handle either one aspect or the other. However, very few existing works can handle both issues simultaneously. In this paper, we target a new problem, Annotation-Efficient Video Recognition, to reduce the requirement of annotations for both large amount of samples and the action location. Such problem is challenging due to two aspects: (1) the untrimmed videos only have weak supervision; (2) video segments not relevant to current actions of interests (background, BG) could contain actions of interests (foreground, FG) in novel classes, which is a widely existing phenomenon but has rarely been studied in few-shot untrimmed video recognition. To achieve this goal, by analyzing the property of BG, we categorize BG into informative BG (IBG) and non-informative BG (NBG), and we propose (1) an open-set detection based method to find the NBG and FG, (2) a contrastive learning method to learn IBG and distinguish NBG in a self-supervised way, and (3) a self-weighting mechanism for the better distinguishing of IBG and FG. Extensive experiments on ActivityNet v1.2 and ActivityNet v1.3 verify the rationale and effectiveness of the proposed methods.

cs.CV↗

arXiv2021

Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain

Recently, the generalization behavior of Convolutional Neural Networks (CNN) is gradually transparent through explanation techniques with the frequency components decomposition. However, the importance of the phase spectrum of the image for a robust vision system is still ignored. In this paper, we notice that the CNN tends to converge at the local optimum which is closely related to the high-frequency components of the training images, while the amplitude spectrum is easily disturbed such as noises or common corruptions. In contrast, more empirical studies found that humans rely on more phase components to achieve robust recognition. This observation leads to more explanations of the CNN's generalization behaviors in both robustness to common perturbations and out-of-distribution detection, and motivates a new perspective on data augmentation designed by re-combing the phase spectrum of the current image and the amplitude spectrum of the distracter image. That is, the generated samples force the CNN to pay more attention to the structured information from phase components and keep robust to the variation of the amplitude. Experiments on several image datasets indicate that the proposed method achieves state-of-the-art performances on multiple generalizations and calibration tasks, including adaptability for common corruptions and surface variations, out-of-distribution detection, and adversarial attack.

cs.CV↗

arXiv2021

Adversarial Reciprocal Points Learning for Open Set Recognition

Open set recognition (OSR), aiming to simultaneously classify the seen classes and identify the unseen classes as 'unknown', is essential for reliable machine learning.The key challenge of OSR is how to reduce the empirical classification risk on the labeled known data and the open space risk on the potential unknown data simultaneously. To handle the challenge, we formulate the open space risk problem from the perspective of multi-class integration, and model the unexploited extra-class space with a novel concept Reciprocal Point. Follow this, a novel learning framework, termed Adversarial Reciprocal Point Learning (ARPL), is proposed to minimize the overlap of known distribution and unknown distributions without loss of known classification accuracy. Specifically, each reciprocal point is learned by the extra-class space with the corresponding known category, and the confrontation among multiple known categories are employed to reduce the empirical classification risk. Then, an adversarial margin constraint is proposed to reduce the open space risk by limiting the latent open space constructed by reciprocal points. To further estimate the unknown distribution from open space, an instantiated adversarial enhancement method is designed to generate diverse and confusing training samples, based on the adversarial mechanism between the reciprocal points and known classes. This can effectively enhance the model distinguishability to the unknown classes. Extensive experimental results on various benchmark datasets indicate that the proposed method is significantly superior to other existing approaches and achieves state-of-the-art performance.

cs.CV↗

arXiv2021

Environment Modeling During Model Checking of Cyber-Physical Systems

Ensuring the safety and efficacy of Cyber-Physical Systems (CPSs) is challenging due to the large variability of their operating environment. Model checking has been proposed for validation of CPSs, but the models of the environment are either too specific to capture the variability of the environment, or too abstract to provide counter-examples interpretable by experts in the application domain. Domain-specific solutions to this problem require expertise in both formal methods and the application domain, which prevents effective application of model checking in CPSs validation. A domain-independent framework based on timed-automata is proposed for abstraction and refinement of environment models during model checking. The framework maintains an abstraction tree of environment models, which provides interpretable counter-examples while ensuring coverage of environment behaviors. With the framework, experts in the application domain can effectively use model checking without expertise in formal methods.

cs.FL↗

arXiv2020

Learning Open Set Network with Discriminative Reciprocal Points

Open set recognition is an emerging research area that aims to simultaneously classify samples from predefined classes and identify the rest as 'unknown'. In this process, one of the key challenges is to reduce the risk of generalizing the inherent characteristics of numerous unknown samples learned from a small amount of known data. In this paper, we propose a new concept, Reciprocal Point, which is the potential representation of the extra-class space corresponding to each known category. The sample can be classified to known or unknown by the otherness with reciprocal points. To tackle the open set problem, we offer a novel open space risk regularization term. Based on the bounded space constructed by reciprocal points, the risk of unknown is reduced through multi-category interaction. The novel learning framework called Reciprocal Point Learning (RPL), which can indirectly introduce the unknown information into the learner with only known classes, so as to learn more compact and discriminative representations. Moreover, we further construct a new large-scale challenging aircraft dataset for open set recognition: Aircraft 300 (Air-300). Extensive experiments on multiple benchmark datasets indicate that our framework is significantly superior to other existing approaches and achieves state-of-the-art performance on standard open set benchmarks.

cs.CV↗

arXiv2020

Ultrarelativistic quark-nucleus scattering in a light-front Hamiltonian approach

We investigate the scattering of a quark on a heavy nucleus at high energies using the time-dependent basis light-front quantization (tBLFQ) formalism, which is the first application of the tBLFQ formalism in QCD. We present the real-time evolution of the quark wave function in a strong classical color field of the relativistic nucleus, described as the Color Glass Condensate. The quark and the nucleus color field are simulated in the QCD SU(3) color space. We calculate the total and the differential cross sections, and the quark distribution in coordinate and color spaces using the tBLFQ approach. We recover the eikonal cross sections in the eikonal limit. We find that the differential cross section from the tBLFQ simulation is in agreement with a perturbative calculation at large $p_\perp$, and it deviates from the perturbative calculation at small $p_\perp$ due to higher-order contributions. In particular, we relax the eikonal limit by letting the quark carry realistic finite longitudinal momenta. We study the sub-eikonal effect on the quark through the transverse coordinate distribution of the quark with different longitudinal momentum, and we find the sub-eikonal effect to be sizable. Our results can significantly reduce the theoretical uncertainties in small $p_\perp$ region which has important implications to the phenomenology of the hadron-nucleus and deep inelastic scattering at high energies.

nucl-th↗

arXiv2018

Heavy quarkonia production at LHC and EIC using Basis Light-Front Quantization wavefunctions

We study exclusive charmonium and bottomonium production in ultra-peripheral heavy-ion collisions and electron-ion collisions within the dipole picture. We employ heavy quarkonium light-front wavefunctions obtained within the basis light-front quantization framework, which has some features of the light-front holographic QCD. We focus on comparison with measurements of exclusive charmonium and bottomonium production in ultraperipheral $pp$, $p$Pb and PbPb collisions at LHC and find reasonable agreement with the cross sections. We also discuss the coherent production cross-section ratios of excited states to the ground state for charmonium and bottomonium, which exhibit insensitivity to the dipole model parameters. We show that electron-ion collisions provide an opportunity for a quantitative study of heavy quarkonium structure.

nucl-th↗

arXiv2018

Hadron Spectra, Decays and Scattering Properties within Basis Light Front Quantization

We survey recent progress in calculating properties of the electron and hadrons within the Basis Light Front Quantization (BLFQ) approach. We include applications to electromagnetic and strong scattering processes in relativistic heavy ion collisions. We present an initial investigation into the glueball states by applying BLFQ with multigluon sectors, introducing future research possibilities on multi-quark and multi-gluon systems.

nucl-th↗

arXiv2018

Coulomb Excitation of Deuteron in Peripheral Collisions with a Heavy Ion

We develop an ab initio, non-perturbative, time-dependent Basis Function (tBF) method to solve the nuclear structure and scattering problems in a unified manner. We apply this method to a test problem: the Coulomb excitation of a trapped deuteron by an impinging heavy ion. The states of the deuteron system are obtained by the ab initio nuclear structure calculation implementing a realistic inter-nucleon interaction with a weak external trap to localize the center of mass and to discretize the continuum. The evolution of the internal state of the deuteron system is directly solved using the equation of motion for the scattering. We analyze the excitation mechanism of the deuteron system by investigating its internal transition probabilities and observables as functions of the exposure time and the incident speed. In this investigation, the dynamics of the Coulomb excitation are revealed by the time evolution of the system's internal charge distribution.

nucl-th↗

arXiv2017

Initial Angular Momentum and Flow in High Energy Nuclear Collisions

We study the transfer of angular momentum in high energy nuclear collisions from the colliding nuclei to the region around midrapidity, using the classical approximation of the Color Glass Condensate (CGC) picture. We find that the angular momentum shortly after the collision (up to times ~ 1/Q_s, where Q_s is the saturation scale) is carried by the "beta-type" flow of the initial classical gluon field, introduced by some of us earlier. beta^i ~ mu_1 nabla^i mu_2 - mu_2 nabla^i mu_1 (i=1,2) describes the rapidity-odd transverse energy flow and emerges from Gauss' Law for gluon fields. Here mu_1 and mu_2 are the averaged color charge fluctuation densities in the two nuclei, respectively. Interestingly, strong coupling calculations using AdS/CFT techniques also find an energy flow term featuring this particular combination of nuclear densities. In classical CGC the order of magnitude of the initial angular momentum per rapidity in the reaction plane, at a time 1/Q_s, is |dL_2/d eta| ~ R_A/Q_s^3 epsilon_0/2 at midrapidity, where R_A is the nuclear radius, and epsilon_0 is the average initial energy density. This result emerges as a cancellation between a vortex of energy flow in the reaction plane aligned with the total angular momentum, and energy shear flow opposed to it. We discuss in detail the process of matching classical Yang-Mills results to fluid dynamics. We will argue that dissipative corrections should not be discarded to ensure that macroscopic conservation laws, e.g. for angular momentum, hold. Viscous fluid dynamics tends to dissipate the shear flow contribution that carries angular momentum in boost-invariant fluid systems. This leads to small residual angular momentum around midrapidity at late times for collisions at high energies.

nucl-th↗

arXiv2017

Diffractive charmonium spectrum in high energy collisions in the basis light-front quantization approach

Using the charmonium light-front wavefunctions obtained by diagonalizing an effective Hamiltonian with the one-gluon exchange interaction and a confining potential inspired by light-front holography in the basis light-front quantization formalism, we compute production of charmonium states in diffractive deep inelastic scattering and ultra-peripheral heavy ion collisions within the dipole picture. Our method allows us to predict yields of all vector charmonium states below the open flavor thresholds in high-energy deep inelastic scattering, proton-nucleus and ultra-peripheral heavy ion collisions, without introducing any new parameters in the light-front wavefunctions. The obtained charmonium cross section is in reasonable agreement with experimental data at HERA, RHIC and LHC. We observe that the cross-section ratio $σ_{Ψ(2s)}/σ_{J/Ψ}$ reveals significant independence of model parameters.

nucl-th↗

arXiv2017

Deuteron Coulomb Excitation in Peripheral Collisions with a Heavy Ion

We develop and test an ab initio time-dependent Basis Function (tBF) method to solve non-perturbative and time-dependent problems in quantum mechanics. For our test problem, we apply this method to the Coulomb excitation of the deuteron by an impinging heavy ion. In the tBF method applied to deuterium, we employ wave functions for its bound and excited states to calculate its transition probabilities and the r.m.s. radius during the scattering process. For comparison, corresponding results based on first-order perturbation theory are also provided. For the Coulomb excitation process in a weak and time-varying Coulomb field, where higher-order effects are negligible, we obtain good agreement of the results based on these two methods. The tBF method is then applied to the Coulomb excitation process with stronger external field. The higher-order effects, such as those appearing in the reorientation of the polarization of the deuteron system, are analyzed.

nucl-th↗