arXiv subjects

Ji Zhu

Publications and source records attributed to Ji Zhu.

At least 37 records · Page 2Linked to original sources

arXiv2025

Flexible Modeling of Information Diffusion on Networks with Statistical Guarantees

Modeling information spread through a network is one of the key problems of network analysis, with applications in a wide array of areas such as marketing and public health. Most approaches assume that the spread is governed by some probabilistic diffusion model, often parameterized by the strength of connections between network members (edge weights), highlighting the need for methods that can accurately estimate them. Multiple prior works suggest such estimators for particular diffusion models; however, most of them lack a rigorous statistical analysis that would establish the asymptotic properties of the estimator and allow for uncertainty quantification. In this paper, we develop a likelihood-based approach to estimate edge weights from the observed information diffusion paths under the proposed General Linear Threshold (GLT) model, a broad class of discrete-time information diffusion models that includes both the well-known linear threshold (LT) and independent cascade (IC) models. We first derive necessary and sufficient conditions that make the edge weights identifiable under this model. Then, we derive a finite sample error bound for the estimator and demonstrate that it is asymptotically normal under mild conditions. We conclude by studying the GLT model in the context of the Influence Maximization (IM) problem, that is, the task of selecting a subset of $k$ nodes to start the diffusion, so that the average information spread is maximized. Extensive experiments on synthetic and real-world networks demonstrate that the flexibility of the proposed class of GLT models, coupled with the proposed estimation and inference framework for its parameters, can significantly improve estimation of spread from a given subset of nodes, prediction of node activation, and the quality of the IM problem solutions.

cs.SI↗

arXiv2025

Minimax Regret Learning for Data with Heterogeneous Subgroups

Modern complex datasets often consist of various sub-populations with known group information. In the presence of sub-population heterogeneity, it is crucial to develop robust and generalizable learning methods that (1) can enjoy robust performance on each of the training populations, and (2) is generalizable to an unseen testing population. While various min-max formulations have been proposed to achieve (1) in the robust learning literature, their generalization to an unseen testing is less explored. Moreover, a general min-max formulation can be sensitive to the noise heterogeneity, and, in the extreme case, be degenerate such that a single high-noise population dominates. The min-max-regret (MMR) can mitigate these challenges. In this work, we consider a distribution-free robust hierarchical model for the generalization from multiple training populations to an unseen testing population. Under the robust hierarchical model, the empirical MMR can enjoy the regret guarantees on each of the training populations as well as the unseen testing population. We further specialize the general MMR framework to linear regression and generalized linear model, where we characterize the geometry of MMR and its distinction from other robust methods. We demonstrate the effectiveness of MMR through extensive simulation studies and an application to image recognition.

stat.ME↗

arXiv2025

Modeling Non-Uniform Hypergraphs Using Determinantal Point Processes

Most statistical models for networks focus on pairwise interactions between nodes. However, many real-world networks involve higher-order interactions among multiple nodes, such as co-authors collaborating on a paper. Hypergraphs provide a natural representation for these networks, with each hyperedge representing a set of nodes. The majority of existing hypergraph models assume uniform hyperedges (i.e., edges of the same size) or rely on diversity among nodes. In this work, we propose a new hypergraph model based on non-symmetric determinantal point processes. The proposed model naturally accommodates non-uniform hyperedges, has tractable probability mass functions, and accounts for both node similarity and diversity in hyperedges. For model estimation, we maximize the likelihood function under constraints using a computationally efficient projected adaptive gradient descent algorithm. We establish the consistency and asymptotic normality of the estimator. Simulation studies confirm the efficacy of the proposed model, and its utility is further demonstrated through edge predictions on several real-world datasets.

stat.ME↗

arXiv2025

Statistical Inference on Latent Space Models for Network Data

Latent space models are powerful statistical tools for modeling and understanding network data. While the importance of accounting for uncertainty in network analysis has been well recognized, the current literature predominantly focuses on point estimation and prediction, leaving the statistical inference of latent space models an open question. This work aims to fill this gap by providing a general framework to analyze the theoretical properties of the maximum likelihood estimators. In particular, we establish the uniform consistency and asymptotic distribution results for the latent space models under different edge types and link functions. Furthermore, the proposed framework enables us to generalize our results to the dependent-edge and sparse scenarios. Our theories are supported by simulation studies and have the potential to be applied in downstream inferences, such as link prediction and network testing problems.

math.ST↗

arXiv2025

GSON: A Group-based Social Navigation Framework with Large Multimodal Model

With the increasing presence of service robots and autonomous vehicles in human environments, navigation systems need to evolve beyond simple destination reach to incorporate social awareness. This paper introduces GSON, a novel group-based social navigation framework that leverages Large Multimodal Models (LMMs) to enhance robots' social perception capabilities. Our approach uses visual prompting to enable zero-shot extraction of social relationships among pedestrians and integrates these results with robust pedestrian detection and tracking pipelines to overcome the inherent inference speed limitations of LMMs. The planning system incorporates a mid-level planner that sits between global path planning and local motion planning, effectively preserving both global context and reactive responsiveness while avoiding disruption of the predicted social group. We validate GSON through extensive real-world mobile robot navigation experiments involving complex social scenarios such as queuing, conversations, and photo sessions. Comparative results show that our system significantly outperforms existing navigation approaches in minimizing social perturbations while maintaining comparable performance on traditional navigation metrics.

cs.RO↗

arXiv2025

Recurrent Event Analysis with Ordinary Differential Equations

This paper introduces a general framework for analyzing recurrent event data by modeling the conditional mean function of the recurrent event process as the solution to an Ordinary Differential Equation (ODE). This approach not only accommodates a wide range of semi-parametric recurrent event models, including both non-homogeneous Poisson processes (NHPPs) and non-Poisson processes, but also is scalable and easy-to-implement. Based on this framework, we propose a Sieve Maximum Pseudo-Likelihood Estimation (SMPLE) method, employing the NHPP as a working model. We establish the consistency and asymptotic normality of the proposed estimator, demonstrating that it achieves semi-parametric efficiency when the NHPP working model is valid. Furthermore, we develop an efficient resampling procedure to estimate the asymptotic covariance matrix. To assess the statistical efficiency and computational scalability of the proposed method, we conduct extensive numerical studies, including simulations under various settings and an application to a real-world dataset analyzing risk factors associated with Intensive Care Unit (ICU) readmission frequency.

stat.ME↗

arXiv2025

Likelihood ratio tests in random graph models with increasing dimensions

We explore the Wilks phenomena in two random graph models: the $β$-model and the Bradley-Terry model. For two increasing dimensional null hypotheses, including a specified null $H_0: β_i=β_i^0$ for $i=1,\ldots, r$ and a homogenous null $H_0: β_1=\cdots=β_r$, we reveal high dimensional Wilks' phenomena that the normalized log-likelihood ratio statistic, $[2\{\ell(\widehat{\mathbfβ}) - \ell(\widehat{\mathbfβ}^0)\} - r]/(2r)^{1/2}$, converges in distribution to the standard normal distribution as $r$ goes to infinity. Here, $\ell( \mathbfβ)$ is the log-likelihood function on the model parameter $\mathbfβ=(β_1, \ldots, β_n)^\top$, $\widehat{\mathbfβ}$ is its maximum likelihood estimator (MLE) under the full parameter space, and $\widehat{\mathbfβ}^0$ is the restricted MLE under the null parameter space. For the homogenous null with a fixed $r$, we establish Wilks-type theorems that $2\{\ell(\widehat{\mathbfβ}) - \ell(\widehat{\mathbfβ}^0)\}$ converges in distribution to a chi-square distribution with $r-1$ degrees of freedom, as the total number of parameters, $n$, goes to infinity. When testing the fixed dimensional specified null, we find that its asymptotic null distribution is a chi-square distribution in the $β$-model. However, unexpectedly, this is not true in the Bradley-Terry model. By developing several novel technical methods for asymptotic expansion, we explore Wilks type results in a principled manner; these principled methods should be applicable to a class of random graph models beyond the $β$-model and the Bradley-Terry model. Simulation studies and real network data applications further demonstrate the theoretical results.

math.ST↗

arXiv2025

Modeling Hypergraphs with Diversity and Heterogeneous Popularity

While relations among individuals make an important part of data with scientific and business interests, existing statistical modeling of relational data has mainly been focusing on dyadic relations, i.e., those between two individuals. This article addresses the less studied, though commonly encountered, polyadic relations that can involve more than two individuals. In particular, we propose a new latent space model for hypergraphs using determinantal point processes, which is driven by the diversity within hyperedges and each node's popularity. This model mechanism is in contrast to existing hypergraph models, which are predominantly driven by similarity rather than diversity. Additionally, the proposed model accommodates broad types of hypergraphs, with no restriction on the cardinality and multiplicity of hyperedges, which previous models often have. Consistency and asymptotic normality of the maximum likelihood estimates of the model parameters have been established. The proof is challenging, owing to the special configuration of the parameter space. Further, we apply the projected accelerated gradient descent algorithm to obtain the parameter estimates, and we show its effectiveness in simulation studies. We also demonstrate an application of the proposed model on the What's Cooking data and present the embedding of food ingredients learned from cooking recipes using the model.

stat.ME↗

arXiv2024

Distributed Community Detection in Large Networks

Community detection for large networks poses challenges due to the high computational cost as well as heterogeneous community structures. In this paper, we consider widely existing real-world networks with ``grouped communities'' (or ``the group structure''), where nodes within grouped communities are densely connected and nodes across grouped communities are relatively loosely connected. We propose a two-step community detection approach for such networks. Firstly, we leverage modularity optimization methods to partition the network into groups, where between-group connectivity is low. Secondly, we employ the stochastic block model (SBM) or degree-corrected SBM (DCSBM) to further partition the groups into communities, allowing for varying levels of between-community connectivity. By incorporating this two-step structure, we introduce a novel divide-and-conquer algorithm that asymptotically recovers both the group structure and the community structure. Numerical studies confirm that our approach significantly reduces computational costs while achieving competitive performance. This framework provides a comprehensive solution for detecting community structures in networks with grouped communities, offering a valuable tool for various applications.

stat.CO↗

arXiv2024

EmbSum: Leveraging the Summarization Capabilities of Large Language Models for Content-Based Recommendations

Content-based recommendation systems play a crucial role in delivering personalized content to users in the digital world. In this work, we introduce EmbSum, a novel framework that enables offline pre-computations of users and candidate items while capturing the interactions within the user engagement history. By utilizing the pretrained encoder-decoder model and poly-attention layers, EmbSum derives User Poly-Embedding (UPE) and Content Poly-Embedding (CPE) to calculate relevance scores between users and candidate items. EmbSum actively learns the long user engagement histories by generating user-interest summary with supervision from large language model (LLM). The effectiveness of EmbSum is validated on two datasets from different domains, surpassing state-of-the-art (SoTA) methods with higher accuracy and fewer parameters. Additionally, the model's ability to generate summaries of user interests serves as a valuable by-product, enhancing its usefulness for personalized content recommendations.

cs.IR↗

arXiv2024

Latent process models for functional network data

Network data are often sampled with auxiliary information or collected through the observation of a complex system over time, leading to multiple network snapshots indexed by a continuous variable. Many methods in statistical network analysis are traditionally designed for a single network, and can be applied to an aggregated network in this setting, but that approach can miss important functional structure. Here we develop an approach to estimating the expected network explicitly as a function of a continuous index, be it time or another indexing variable. We parameterize the network expectation through low dimensional latent processes, whose components we represent with a fixed, finite-dimensional functional basis. We derive a gradient descent estimation algorithm, establish theoretical guarantees for recovery of the low dimensional structure, compare our method to competitors, and apply it to a data set of international political interactions over time, showing our proposed method to adapt well to data, outperform competitors, and provide interpretable and meaningful results.

stat.ME↗

arXiv2024

Variational Estimators of the Degree-corrected Latent Block Model for Bipartite Networks

Bipartite graphs are ubiquitous across various scientific and engineering fields. Simultaneously grouping the two types of nodes in a bipartite graph via biclustering represents a fundamental challenge in network analysis for such graphs. The latent block model (LBM) is a commonly used model-based tool for biclustering. However, the effectiveness of the LBM is often limited by the influence of row and column sums in the data matrix. To address this limitation, we introduce the degree-corrected latent block model (DC-LBM), which accounts for the varying degrees in row and column clusters, significantly enhancing performance on real-world data sets and simulated data. We develop an efficient variational expectation-maximization algorithm by creating closed-form solutions for parameter estimates in the M steps. Furthermore, we prove the label consistency and the rate of convergence of the variational estimator under the DC-LBM, allowing the expected graph density to approach zero as long as the average expected degrees of rows and columns approach infinity when the size of the graph increases.

stat.ML↗

arXiv2024

High-dimensional Factor Analysis for Network-linked Data

Factor analysis is a widely used statistical tool in many scientific disciplines, such as psychology, economics, and sociology. As observations linked by networks become increasingly common, incorporating network structures into factor analysis remains an open problem. In this paper, we focus on high-dimensional factor analysis involving network-connected observations, and propose a generalized factor model with latent factors that account for both the network structure and the dependence structure among high-dimensional variables. These latent factors can be shared by the high-dimensional variables and the network, or exclusively applied to either of them. We develop a computationally efficient estimation procedure and establish asymptotic inferential theories. Notably, we show that by borrowing information from the network, the proposed estimator of the factor loading matrix achieves optimal asymptotic variance under much milder identifiability constraints than existing literature. Furthermore, we develop a hypothesis testing procedure to tackle the challenge of discerning the shared and individual latent factors' structure. The finite sample performance of the proposed method is demonstrated through simulation studies and a real-world dataset involving a statistician co-authorship network.

stat.ME↗

arXiv2023

Population-level Balance in Signed Networks

Statistical network models are useful for understanding the underlying formation mechanism and characteristics of complex networks. However, statistical models for \textit{signed networks} have been largely unexplored. In signed networks, there exist both positive (e.g., like, trust) and negative (e.g., dislike, distrust) edges, which are commonly seen in real-world scenarios. The positive and negative edges in signed networks lead to unique structural patterns, which pose challenges for statistical modeling. In this paper, we introduce a statistically principled latent space approach for modeling signed networks and accommodating the well-known \textit{balance theory}, i.e., ``the enemy of my enemy is my friend'' and ``the friend of my friend is my friend''. The proposed approach treats both edges and their signs as random variables, and characterizes the balance theory with a novel and natural notion of population-level balance. This approach guides us towards building a class of balanced inner-product models, and towards developing scalable algorithms via projected gradient descent to estimate the latent variables. We also establish non-asymptotic error rates for the estimates, which are further verified through simulation studies. In addition, we apply the proposed approach to an international relation network, which provides an informative and interpretable model-based visualization of countries during World War II.

stat.ME↗

arXiv2023

Fair Information Spread on Social Networks with Community Structure

Information spread through social networks is ubiquitous. Influence maximiza- tion (IM) algorithms aim to identify individuals who will generate the greatest spread through the social network if provided with information, and have been largely devel- oped with marketing in mind. In social networks with community structure, which are very common, IM algorithms focused solely on maximizing spread may yield signifi- cant disparities in information coverage between communities, which is problematic in settings such as public health messaging. While some IM algorithms aim to remedy disparity in information coverage using node attributes, none use the empirical com- munity structure within the network itself, which may be beneficial since communities directly affect the spread of information. Further, the use of empirical network struc- ture allows us to leverage community detection techniques, making it possible to run fair-aware algorithms when there are no relevant node attributes available, or when node attributes do not accurately capture network community structure. In contrast to other fair IM algorithms, this work relies on fitting a model to the social network which is then used to determine a seed allocation strategy for optimal fair information spread. We develop an algorithm to determine optimal seed allocations for expected fair coverage, defined through maximum entropy, provide some theoretical guarantees under appropriate conditions, and demonstrate its empirical accuracy on both simu- lated and real networks. Because this algorithm relies on a fitted network model and not on the network directly, it is well-suited for partially observed and noisy social networks.

stat.ML↗

arXiv2023

Conformal Prediction for Network-Assisted Regression

An important problem in network analysis is predicting a node attribute using both network covariates, such as graph embedding coordinates or local subgraph counts, and conventional node covariates, such as demographic characteristics. While standard regression methods that make use of both types of covariates may be used for prediction, statistical inference is complicated by the fact that the nodal summary statistics are often dependent in complex ways. We show that under a mild joint exchangeability assumption, a network analog of conformal prediction achieves finite sample validity for a wide range of network covariates. We also show that a form of asymptotic conditional validity is achievable. The methods are illustrated on both simulated networks and a citation network dataset.

stat.ME↗

arXiv2023

Incorporating External Risk Information with the Cox Model under Population Heterogeneity: Applications to Trans-Ancestry Polygenic Hazard Scores

Polygenic hazard score (PHS) models designed for European ancestry (EUR) individuals provide ample information regarding survival risk discrimination. Incorporating such information can improve the performance of risk discrimination in an internal small-sized non-EUR cohort. However, given that external EUR-based model and internal individual-level data come from different populations, ignoring population heterogeneity can introduce substantial bias. In this paper, we develop a Kullback-Leibler-based Cox model (CoxKL) to integrate internal individual-level time-to-event data with external risk scores derived from published prediction models, accounting for population heterogeneity. Partial-likelihood-based KL information is utilized to measure the discrepancy between the external risk information and the internal data. We establish the asymptotic properties of the CoxKL estimator. Simulation studies show that the integration model by the proposed CoxKL method achieves improved estimation efficiency and prediction accuracy. We applied the proposed method to develop a trans-ancestry PHS model for prostate cancer and found that integrating a previously published EUR-based PHS with an internal genotype data of African ancestry (AFR) males yielded considerable improvement on the prostate cancer risk discrimination.

stat.ME↗

arXiv2022

Wilks' theorems in the $β$-model

Likelihood ratio tests and the Wilks theorems have been pivotal in statistics but have rarely been explored in network models with an increasing dimension. We are concerned here with likelihood ratio tests in the $β$-model for undirected graphs. For two growing dimensional null hypotheses including a specified null $H_0: β_i=β_i^0$ for $i=1,\ldots, r$ and a homogenous null $H_0: β_1=\cdots=β_r$, we reveal high dimensional Wilks' phenomena that the normalized log-likelihood ratio statistic, $[2\{\ell(\widehat{\boldsymbolβ}) - \ell(\widehat{\boldsymbolβ}^0)\} - r]/(2r)^{1/2}$, converges in distribution to the standard normal distribution as $r$ goes to infinity. Here, $\ell( \boldsymbolβ)$ is the log-likelihood function on the vector parameter $\boldsymbolβ=(β_1, \ldots, β_n)^\top$, $\widehat{\boldsymbolβ}$ is its maximum likelihood estimator (MLE) under the full parameter space, and $\widehat{\boldsymbolβ}^0$ is the restricted MLE under the null parameter space. For the corresponding fixed dimensional null $H_0: β_i=β_i^0$ for $i=1,\ldots, r$ and the homogenous null $H_0: β_1=\cdots=β_r$ with a fixed $r$, we establish Wilks type of results that $2\{\ell(\widehat{\boldsymbolβ}) - \ell(\widehat{\boldsymbolβ}^0)\}$ converges in distribution to a Chi-square distribution with respective $r$ and $r-1$ degrees of freedom, as the total number of parameters, $n$, goes to infinity. The Wilks type of results are further extended into a closely related Bradley--Terry model for paired comparisons, where we discover a different phenomenon that the log-likelihood ratio statistic under the fixed dimensional specified null asymptotically follows neither a Chi-square nor a rescaled Chi-square distribution. Simulation studies and an application to NBA data illustrate the theoretical results.

math.ST↗