arXiv subjects

Dai Shi

Publications and source records attributed to Dai Shi.

53 records · Page 3Linked to original sources

arXiv2023

From Continuous Dynamics to Graph Neural Networks: Neural Diffusion and Beyond

Graph neural networks (GNNs) have demonstrated significant promise in modelling relational data and have been widely applied in various fields of interest. The key mechanism behind GNNs is the so-called message passing where information is being iteratively aggregated to central nodes from their neighbourhood. Such a scheme has been found to be intrinsically linked to a physical process known as heat diffusion, where the propagation of GNNs naturally corresponds to the evolution of heat density. Analogizing the process of message passing to the heat dynamics allows to fundamentally understand the power and pitfalls of GNNs and consequently informs better model design. Recently, there emerges a plethora of works that proposes GNNs inspired from the continuous dynamics formulation, in an attempt to mitigate the known limitations of GNNs, such as oversmoothing and oversquashing. In this survey, we provide the first systematic and comprehensive review of studies that leverage the continuous perspective of GNNs. To this end, we introduce foundational ingredients for adapting continuous dynamics to GNNs, along with a general framework for the design of graph neural dynamics. We then review and categorize existing works based on their driven mechanisms and underlying dynamics. We also summarize how the limitations of classic GNNs can be addressed under the continuous framework. We conclude by identifying multiple open research directions.

cs.LG↗

arXiv2023

Linear analysis of flow mode transition triggered by finite-sized particles in Rayleigh-Bénard convection

A mathematical model to study the flow evolution in RB convection laden with finite-sized particles after a flow perturbation is developed with an Euler-Lagrange viewpoint. A linear analysis is conducted by combining the averaged flow-scale momentum equation with the averaged particle-scale heat exchange equation. A flow mode of regular oscillation, which is triggered by the heat exchange process between every single particle and its surrounding fluid element, is predicted by this model. Three time scales, namely $τ_f $, $τ_p$, and $Δt$, are shown to be involved in the heat exchange process, giving significance to the temperature distribution inside the single particle. By assuming $τ_f / τ_p \gg 1$, the detailed correlation between the oscillation period and $κ_p / κ_f $ is established, suggesting a flow mode transition with the change of heat diffusivity ratio of particle to fluid.

physics.flu-dyn↗

arXiv2023

Flow evolution in particle-laden Rayleigh-Bénard convection

A theoretical analysis is carried out to study flow evolution inside the laminar Rayleigh-Bénard convection system laden with small particles. By describing particle dynamics and particle heat as sources of drag and heat respectively, the physics of particle impact on the flow evolution is studied. It is found that due to the relative velocity of the particulate phase to the fluid phase, particles work as a superimposed moment on the whole flow, attenuating the flow intensity. When the relative temperature of the particulate phase to the fluid phase occurs, particles act like a superimposed moment of buoyancy force on the whole flow, causing alterations in both flow intensity and flow direction.

physics.flu-dyn↗

arXiv2023

Revisiting Generalized p-Laplacian Regularized Framelet GCNs: Convergence, Energy Dynamic and Training with Non-Linear Diffusion

This paper presents a comprehensive theoretical analysis of the graph p-Laplacian regularized framelet network (pL-UFG) to establish a solid understanding of its properties. We conduct a convergence analysis on pL-UFG, addressing the gap in the understanding of its asymptotic behaviors. Further by investigating the generalized Dirichlet energy of pL-UFG, we demonstrate that the Dirichlet energy remains non-zero throughout convergence, ensuring the avoidance of over-smoothing issues. Additionally, we elucidate the energy dynamic perspective, highlighting the synergistic relationship between the implicit layer in pL-UFG and graph framelets. This synergy enhances the model's adaptability to both homophilic and heterophilic data. Notably, we reveal that pL-UFG can be interpreted as a generalized non-linear diffusion process, thereby bridging the gap between pL-UFG and differential equations on the graph. Importantly, these multifaceted analyses lead to unified conclusions that offer novel insights for understanding and implementing pL-UFG, as well as other graph neural network (GNN) models. Finally, based on our dynamic analysis, we propose two novel pL-UFG models with manually controlled energy dynamics. We demonstrate empirically and theoretically that our proposed models not only inherit the advantages of pL-UFG but also significantly reduce computational costs for training on large-scale graph datasets.

cs.LG↗

arXiv2023

Unifying over-smoothing and over-squashing in graph neural networks: A physics informed approach and beyond

Graph Neural Networks (GNNs) have emerged as one of the leading approaches for machine learning on graph-structured data. Despite their great success, critical computational challenges such as over-smoothing, over-squashing, and limited expressive power continue to impact the performance of GNNs. In this study, inspired from the time-reversal principle commonly utilized in classical and quantum physics, we reverse the time direction of the graph heat equation. The resulted reversing process yields a class of high pass filtering functions that enhance the sharpness of graph node features. Leveraging this concept, we introduce the Multi-Scaled Heat Kernel based GNN (MHKG) by amalgamating diverse filtering functions' effects on node features. To explore more flexible filtering conditions, we further generalize MHKG into a model termed G-MHKG and thoroughly show the roles of each element in controlling over-smoothing, over-squashing and expressive power. Notably, we illustrate that all aforementioned issues can be characterized and analyzed via the properties of the filtering functions, and uncover a trade-off between over-smoothing and over-squashing: enhancing node feature sharpness will make model suffer more from over-squashing, and vice versa. Furthermore, we manipulate the time again to show how G-MHKG can handle both two issues under mild conditions. Our conclusive experiments highlight the effectiveness of proposed models. It surpasses several GNN baseline models in performance across graph datasets characterized by both homophily and heterophily.

cs.LG↗

arXiv2023

Bregman Graph Neural Network

Numerous recent research on graph neural networks (GNNs) has focused on formulating GNN architectures as an optimization problem with the smoothness assumption. However, in node classification tasks, the smoothing effect induced by GNNs tends to assimilate representations and over-homogenize labels of connected nodes, leading to adverse effects such as over-smoothing and misclassification. In this paper, we propose a novel bilevel optimization framework for GNNs inspired by the notion of Bregman distance. We demonstrate that the GNN layer proposed accordingly can effectively mitigate the over-smoothing issue by introducing a mechanism reminiscent of the "skip connection". We validate our theoretical results through comprehensive empirical studies in which Bregman-enhanced GNNs outperform their original counterparts in both homophilic and heterophilic graphs. Furthermore, our experiments also show that Bregman GNNs can produce more robust learning accuracy even when the number of layers is high, suggesting the effectiveness of the proposed method in alleviating the over-smoothing issue.

cs.LG↗

arXiv2023

How Curvature Enhance the Adaptation Power of Framelet GCNs

Graph neural network (GNN) has been demonstrated powerful in modeling graph-structured data. However, despite many successful cases of applying GNNs to various graph classification and prediction tasks, whether the graph geometrical information has been fully exploited to enhance the learning performance of GNNs is not yet well understood. This paper introduces a new approach to enhance GNN by discrete graph Ricci curvature. Specifically, the graph Ricci curvature defined on the edges of a graph measures how difficult the information transits on one edge from one node to another based on their neighborhoods. Motivated by the geometric analogy of Ricci curvature in the graph setting, we prove that by inserting the curvature information with different carefully designed transformation function $ζ$, several known computational issues in GNN such as over-smoothing can be alleviated in our proposed model. Furthermore, we verified that edges with very positive Ricci curvature (i.e., $κ_{i,j} \approx 1$) are preferred to be dropped to enhance model's adaption to heterophily graph and one curvature based graph edge drop algorithm is proposed. Comprehensive experiments show that our curvature-based GNN model outperforms the state-of-the-art baselines in both homophily and heterophily graph datasets, indicating the effectiveness of involving graph geometric information in GNNs.

cs.LG↗

arXiv2023

Generalized Laplacian Regularized Framelet Graph Neural Networks

This paper introduces a novel Framelet Graph approach based on p-Laplacian GNN. The proposed two models, named p-Laplacian undecimated framelet graph convolution (pL-UFG) and generalized p-Laplacian undecimated framelet graph convolution (pL-fUFG) inherit the nature of p-Laplacian with the expressive power of multi-resolution decomposition of graph signals. The empirical study highlights the excellent performance of the pL-UFG and pL-fUFG in different graph learning tasks including node classification and signal denoising.

cs.LG↗

arXiv2023

Frameless Graph Knowledge Distillation

Knowledge distillation (KD) has shown great potential for transferring knowledge from a complex teacher model to a simple student model in which the heavy learning task can be accomplished efficiently and without losing too much prediction accuracy. Recently, many attempts have been made by applying the KD mechanism to the graph representation learning models such as graph neural networks (GNNs) to accelerate the model's inference speed via student models. However, many existing KD-based GNNs utilize MLP as a universal approximator in the student model to imitate the teacher model's process without considering the graph knowledge from the teacher model. In this work, we provide a KD-based framework on multi-scaled GNNs, known as graph framelet, and prove that by adequately utilizing the graph knowledge in a multi-scaled manner provided by graph framelet decomposition, the student model is capable of adapting both homophilic and heterophilic graphs and has the potential of alleviating the over-squashing issue with a simple yet effectively graph surgery. Furthermore, we show how the graph knowledge supplied by the teacher is learned and digested by the student model via both algebra and geometry. Comprehensive experiments show that our proposed model can generate learning accuracy identical to or even surpass the teacher model while maintaining the high speed of inference.

cs.LG↗

arXiv2022

Generalized energy and gradient flow via graph framelets

In this work, we provide a theoretical understanding of the framelet-based graph neural networks through the perspective of energy gradient flow. By viewing the framelet-based models as discretized gradient flows of some energy, we show it can induce both low-frequency and high-frequency-dominated dynamics, via the separate weight matrices for different frequency components. This substantiates its good empirical performance on both homophilic and heterophilic graphs. We then propose a generalized energy via framelet decomposition and show its gradient flow leads to a novel graph neural network, which includes many existing models as special cases. We then explain how the proposed model generally leads to more flexible dynamics, thus potentially enhancing the representation power of graph neural networks.

cs.LG↗

arXiv2021

A Discussion On the Validity of Manifold Learning

Dimensionality reduction (DR) and manifold learning (ManL) have been applied extensively in many machine learning tasks, including signal processing, speech recognition, and neuroinformatics. However, the understanding of whether DR and ManL models can generate valid learning results remains unclear. In this work, we investigate the validity of learning results of some widely used DR and ManL methods through the chart mapping function of a manifold. We identify a fundamental problem of these methods: the mapping functions induced by these methods violate the basic settings of manifolds, and hence they are not learning manifold in the mathematical sense. To address this problem, we provide a provably correct algorithm called fixed points Laplacian mapping (FPLM), that has the geometric guarantee to find a valid manifold representation (up to a homeomorphism). Combining one additional condition(orientation preserving), we discuss a sufficient condition for an algorithm to be bijective for any d-simplex decomposition result on a d-manifold. However, constructing such a mapping function and its computational method satisfying these conditions is still an open problem in mathematics.

cs.LG↗

arXiv2020

A Fast Radio Burst discovered in FAST drift scan survey

We report the discovery of a highly dispersed fast radio burst, FRB~181123, from an analysis of $\sim$1500~hr of drift-scan survey data taken using the Five-hundred-meter Aperture Spherical radio Telescope (FAST). The pulse has three distinct emission components, which vary with frequency across our 1.0--1.5~GHz observing band. We measure the peak flux density to be $>0.065$~Jy and the corresponding fluence $>0.2$~Jy~ms. Based on the observed dispersion measure of 1812~cm$^{-3}$~pc, we infer a redshift of $\sim 1.9$. From this, we estimate the peak luminosity and isotropic energy to be $\lesssim 2\times10^{43}$~erg~s$^{-1}$ and $\lesssim 2\times10^{40}$~erg, respectively. With only one FRB from the survey detected so far, our constraints on the event rate are limited. We derive a 95\% confidence lower limit for the event rate of 900 FRBs per day for FRBs with fluences $>0.025$~Jy~ms. We performed follow-up observations of the source with FAST for four hours and have not found a repeated burst. We discuss the implications of this discovery for our understanding of the physical mechanisms of FRBs.

astro-ph.HE↗

arXiv2019

Coupling Matrix Manifolds and Their Applications in Optimal Transport

Optimal transport (OT) is a powerful tool for measuring the distance between two defined probability distributions. In this paper, we develop a new manifold named the coupling matrix manifold (CMM), where each point on CMM can be regarded as the transportation plan of the OT problem. We firstly explore the Riemannian geometry of CMM with the metric expressed by the Fisher information. These geometrical features of CMM have paved the way for developing numerical Riemannian optimization algorithms such as Riemannian gradient descent and Riemannian trust-region algorithms, forming a uniform optimization method for all types of OT problems. The proposed method is then applied to solve several OT problems studied by previous literature. The results of the numerical experiments illustrate that the optimization algorithms that are based on the method proposed in this paper are comparable to the classic ones, for example, the Sinkhorn algorithm, while outperforming other state-of-the-art algorithms without considering the geometry information, especially in the case of non-entropy optimal transport.

cs.LG↗

arXiv2019

A new method to determine multi-angular reflectance factor from lightweight multispectral cameras with sky sensor in a target-less workflow applicable to UAV

A new physically based method to estimate hemispheric-directional reflectance factor (HDRF) from lightweight multispectral cameras that have a downwelling irradiance sensor is presented. It combines radiometry with photogrammetric computer vision to derive geometrically and radiometrically accurate data purely from the images, without requiring reflectance targets or any other additional information apart from the imagery. The sky sensor orientation is initially computed using photogrammetric computer vision and revised with a non-linear regression comprising radiometric and photogrammetry-derived information. It works for both clear sky and overcast conditions. A ground-based test acquisition of a Spectralon target observed from different viewing directions and with different sun positions using a typical multispectral sensor configuration for clear sky and overcast showed that both the overall value and the directionality of the reflectance factor as reported in the literature were well retrieved. An RMSE of 3% for clear sky and up to 5% for overcast sky was observed.

physics.data-an↗

arXiv2013

Asymptotic Joint Distribution of Extreme Sample Eigenvalues and Eigenvectors in the Spiked Population Model

In this paper, we consider a data matrix $X_N\in\mathbb{R}^{N\times p}$ where all the rows are i.i.d. samples in $\mathbb{R}^p$ of mean zero and covariance matrix $Σ\in\mathbb{R}^{p\times p}$. Here the population matrix $Σ$ is of finite rank perturbation of the identity matrix. This is the "spiked population model" first proposed by Johnstone. As $N, p\to\infty$ but $N/p \to γ\in(1, \infty)$, for the sample covariance matrix $S_N := X_NX_N^T/N$, we establish the joint distribution of the largest and the smallest few packs of eigenvalues. Inside each pack, they will behave the same as the eigenvalues drawn from a Gaussian matrix of the corresponding size. Among different packs, we also calculate the covariance between the Gaussian matrices entries. As a corollary, if all the rows of the data matrix are Gaussian, then these packs will be asymptotically independent. Also, the asymptotic behavior of sample eigenvectors are obtained. Their local fluctuation is also Gaussian with covariance explicitly calculated.

math.PR↗

arXiv2012

Smallest Gaps Between Eigenvalues of Random Matrices With Complex Ginibre, Wishart and Universal Unitary Ensembles

In this paper we study the limiting distribution of the $k$ smallest gaps between eigenvalues of three kinds of random matrices -- the Ginibre ensemble, the Wishart ensemble and the universal unitary ensemble. All of them follow a Poissonian ansatz. More precisely, for the Ginibre ensemble we have a global result in which the $k$-th smallest gap has typical length $n^{-3/4}$ with density $x^{4k-1}e^{-x^4}$ after normalization. For the Wishart and the universal unitary ensemble, it has typical length $n^{-4/3}$ and has density $x^{3k-1}e^{-x^3}$ after normalization.

math.PR↗

arXiv2012

Asymptotic Joint Distribution of Extreme Eigenvalues of the Sample Covariance Matrix in the Spiked Population Model

In this paper, we consider a data matrix $X\in\mathbb{C}^{N\times M}$ where all the columns are i.i.d. samples being $N$ dimensional complex Gaussian of mean zero and covariance $Σ\in\mathbb{C}^{N\times N}$. Here the population matrix $Σ$ is of finite rank perturbation of the identity matrix. This is the "spiked population model" first proposed by Johnstone in \cite{21}. As $M, N\to\infty$ but $M/N = γ\in(1, \infty)$, we first establish in this paper the asymptotic distribution of the smallest eigenvalue of the sample covariance matrix $S:= XX^*/M$. It also exhibits a phase transition phenomenon proposed in \cite{1} --- the local fluctuation will be the generalized Tracy-Widom or the generalized Gaussian to be defined in the paper. Moreover we prove that the largest and the smallest eigenvalue are asymptotically independent as $M, N\to\infty$.

math.PR↗