Searcharxiv⌕ Search

arXiv subjects

Hanyu Zhang

Publications and source records attributed to Hanyu Zhang.

At least 55 records · Page 3Linked to original sources

Transfer Learning for Causal Effect Estimation

We present a Transfer Causal Learning (TCL) framework when target and source domains share the same covariate/feature spaces, aiming to improve causal effect estimation accuracy in limited data. Limited data is very common in medical applications, where some rare medical conditions, such as sepsis, are of interest. Our proposed method, named \texttt{$\ell_1$-TCL}, incorporates $\ell_1$ regularized TL for nuisance models (e.g., propensity score model); the TL estimator of the nuisance parameters is plugged into downstream average causal/treatment effect estimators (e.g., inverse probability weighted estimator). We establish non-asymptotic recovery guarantees for the \texttt{$\ell_1$-TCL} with generalized linear model (GLM) under the sparsity assumption in the high-dimensional setting, and demonstrate the empirical benefits of \texttt{$\ell_1$-TCL} through extensive numerical simulation for GLM and recent neural network nuisance models. Our method is subsequently extended to real data and generates meaningful insights consistent with medical literature, a case where all baseline methods fail.

cs.LG↗

Multi-resolution spatio-temporal prediction with application to wind power generation

Wind energy is becoming an increasingly crucial component of a sustainable grid, but its inherent variability and limited predictability present challenges for grid operators. The energy sector needs novel forecasting techniques that can precisely predict the generation of renewable power and offer precise quantification of prediction uncertainty. This will facilitate well-informed decision-making by operators who wish to integrate renewable energy into the power grid. This paper presents a novel approach to wind speed prediction with uncertainty quantification using a multi-resolution spatio-temporal Gaussian process. By leveraging information from multiple sources of predictions with varying accuracies and uncertainties, the joint framework provides a more accurate and robust prediction of wind speed while measuring the uncertainty in these predictions. We assess the effectiveness of our proposed framework using real-world wind data obtained from the Midwest region of the United States. Our results demonstrate that the framework enables predictors with varying data resolutions to learn from each other, leading to an enhancement in overall predictive performance. The proposed framework shows a superior performance compared to other state-of-the-art methods. The goal of this research is to improve grid operation and management by aiding system operators and policymakers in making better-informed decisions related to energy demand management, energy storage system deployment, and energy supply scheduling. This results in potentially further integration of renewable energy sources into the existing power systems.

stat.AP↗

The DESI One-Percent Survey: Exploring A Generalized SHAM for Multiple Tracers with the UNIT Simulation

We perform SubHalo Abundance Matching (SHAM) studies on UNIT simulations with \{$σ, V_{\rm ceil}, v_{\rm smear}$\}-SHAM and \{$σ, V_{\rm ceil},f_{\rm sat}$\}-SHAM. They are designed to reproduce the clustering on 5--30$\,\hmpc$ of Luminous Red Galaxies (LRGs), Emission Line Galaxies (ELGs) and Quasi-Stellar Objects (QSOs) at $0.4<z<3.5$ from DESI One Percent Survey. $V_{\rm ceil}$ is the incompleteness of the massive host (sub)haloes and is the key to the generalized SHAM. $v_{\rm smear}$ models the clustering effect of redshift uncertainties, providing measurments consistent with those from repeat observations. A free satellite fraction $f_{\rm sat}$ is necessary to reproduce the clustering of ELGs. We find ELGs present a more complex galaxy--halo mass relation than LRGs reflected in their weak constraints on $σ$. LRGs, QSOs and ELGs show increasing $V_{\rm ceil}$ values, corresponding to the massive galaxy incompleteness of LRGs, the quenched star formation of ELGs and the quenched black hole accretion of QSOs. For LRGs, a Gaussian $v_{\rm smear}$ presents a better profile for sub-samples at redshift bins than a Lorentzian profile used for other tracers. The impact of the statistical redshift uncertainty on ELG clustering is negligible. The best-fitting satellite fraction for DESI ELGs is around 4 per cent, lower than previous estimations for ELGs. The mean halo mass log$_{10}(\langle M_{\rm vir}\rangle)$ in $\Msun{}$ for LRGs, ELGs and QSOs are ${13.16\pm0.01}$, ${11.90\pm0.06}$ and ${12.66\pm0.45}$ respectively. Our generalized SHAM algorithms facilitate the production of mult-tracer galaxy mocks for cosmological tests.

astro-ph.CO↗

Manifold learning: what, how, and why

Manifold learning (ML), known also as non-linear dimension reduction, is a set of methods to find the low dimensional structure of data. Dimension reduction for large, high dimensional data is not merely a way to reduce the data; the new representations and descriptors obtained by ML reveal the geometric shape of high dimensional point clouds, and allow one to visualize, de-noise and interpret them. This survey presents the principles underlying ML, the representative methods, as well as their statistical foundations from a practicing statistician's perspective. It describes the trade-offs, and what theory tells us about the parameter and algorithmic choices we make in order to obtain reliable conclusions.

stat.ML↗

Asset Bundling for Wind Power Forecasting

The growing penetration of intermittent, renewable generation in US power grids, especially wind and solar generation, results in increased operational uncertainty. In that context, accurate forecasts are critical, especially for wind generation, which exhibits large variability and is historically harder to predict. To overcome this challenge, this work proposes a novel Bundle-Predict-Reconcile (BPR) framework that integrates asset bundling, machine learning, and forecast reconciliation techniques. The BPR framework first learns an intermediate hierarchy level (the bundles), then predicts wind power at the asset, bundle, and fleet level, and finally reconciles all forecasts to ensure consistency. This approach effectively introduces an auxiliary learning task (predicting the bundle-level time series) to help the main learning tasks. The paper also introduces new asset-bundling criteria that capture the spatio-temporal dynamics of wind power time series. Extensive numerical experiments are conducted on an industry-size dataset of 283 wind farms in the MISO footprint. The experiments consider short-term and day-ahead forecasts, and evaluates a large variety of forecasting models that include weather predictions as covariates. The results demonstrate the benefits of BPR, which consistently and significantly improves forecast accuracy over baselines, especially at the fleet level.

stat.ME↗

The DESI One-Percent Survey: Exploring the Halo Occupation Distribution of Luminous Red Galaxies and Quasi-Stellar Objects with AbacusSummit

We present the first comprehensive Halo Occupation Distribution (HOD) analysis of the DESI One-Percent survey Luminous Red Galaxy (LRG) and Quasi-Stellar Object (QSO) samples. We constrain the HOD of each sample and test possible HOD extensions by fitting the redshift-space galaxy 2-point correlation functions in 0.15 < r < 32 Mpc/h in a set of fiducial redshift bins. We use AbacusSummit cubic boxes at Planck 2018 cosmology as model templates and forward model galaxy clustering with the AbacusHOD package. We achieve good fits with a standard HOD model with velocity bias, and we find no evidence for galaxy assembly bias or satellite profile modulation at the current level of statistical uncertainty. For LRGs in 0.4 < z < 0.6, we infer a satellite fraction of fsat = 11+-1%, a mean halo mass of log10 Mh = 13.40+0.02-0.02, and a linear bias of blin = 1.93+0.06-0.04. For LRGs in 0.6 < z < 0.8, we find fsat = 14+-1%, log10 Mh = 13.24+0.02-0.02, and blin = 2.08+0.03-0.03. For QSOs, we infer fsat = 3+8-2%, log10 Mh = 12.65+0.09-0.04, and blin = 2.63+0.37-0.26 in redshift range 0.8 < z < 2.1. Using these fits, we generate a large suite of high-fidelity galaxy mocks. We also study the redshift-evolution of the DESI LRG sample from z = 0.4 up to z = 1.1, revealing significant and interesting trends in mean halo mass, linear bias, and satellite fraction.

astro-ph.CO↗

Changes in Commuter Behavior from COVID-19 Lockdowns in the Atlanta Metropolitan Area

This paper analyzes the impact of COVID-19 related lockdowns in the Atlanta, Georgia metropolitan area by examining commuter patterns in three periods: prior to, during, and after the pandemic lockdown. A cellular phone location dataset is utilized in a novel pipeline to infer the home and work locations of thousands of users from the Density-based Spatial Clustering of Applications with Noise (DBSCAN) algorithm. The coordinates derived from the clustering are put through a reverse geocoding process from which word embeddings are extracted in order to categorize the industry of each work place based on the workplace name and Point of Interest (POI) mapping. Frequencies of commute from home locations to work locations are analyzed in and across all three time periods. Public health and economic factors are discussed to explain potential reasons for the observed changes in commuter patterns.

cs.LG↗

Dictionary-based Manifold Learning

We propose a paradigm for interpretable Manifold Learning for scientific data analysis, whereby we parametrize a manifold with $d$ smooth functions from a scientist-provided dictionary of meaningful, domain-related functions. When such a parametrization exists, we provide an algorithm for finding it based on sparse non-linear regression in the manifold tangent bundle, bypassing more standard manifold learning algorithms. We also discuss conditions for the existence of such parameterizations in function space and for successful recovery from finite samples. We demonstrate our method with experimental results from a real scientific domain.

cs.LG↗

Distribution free optimality intervals for clustering

We address the problem of validating the ouput of clustering algorithms. Given data $\mathcal{D}$ and a partition $\mathcal{C}$ of these data into $K$ clusters, when can we say that the clusters obtained are correct or meaningful for the data? This paper introduces a paradigm in which a clustering $\mathcal{C}$ is considered meaningful if it is good with respect to a loss function such as the K-means distortion, and stable, i.e. the only good clustering up to small perturbations. Furthermore, we present a generic method to obtain post-inference guarantees of near-optimality and stability for a clustering $\mathcal{C}$. The method can be instantiated for a variety of clustering criteria (also called loss functions) for which convex relaxations exist. Obtaining the guarantees amounts to solving a convex optimization problem. We demonstrate the practical relevance of this method by obtaining guarantees for the K-means and the Normalized Cut clustering criteria on realistic data sets. We also prove that asymptotic instability implies finite sample instability w.h.p., allowing inferences about the population clusterability from a sample. The guarantees do not depend on any distributional assumptions, but they depend on the data set $\mathcal{D}$ admitting a stable clustering.

stat.ML↗

The Parametric Stability of Well-separated Spherical Gaussian Mixtures

We quantify the parameter stability of a spherical Gaussian Mixture Model (sGMM) under small perturbations in distribution space. Namely, we derive the first explicit bound to show that for a mixture of spherical Gaussian $P$ (sGMM) in a pre-defined model class, all other sGMM close to $P$ in this model class in total variation distance has a small parameter distance to $P$. Further, this upper bound only depends on $P$. The motivation for this work lies in providing guarantees for fitting Gaussian mixtures; with this aim in mind, all the constants involved are well defined and distribution free conditions for fitting mixtures of spherical Gaussians. Our results tighten considerably the existing computable bounds, and asymptotically match the known sharp thresholds for this problem.

stat.ML↗

Constraining galaxy-halo connection with high-order statistics

We investigate using three-point statistics in constraining the galaxy-halo connection. We show that for some galaxy samples, the constraints on the halo occupation distribution parameters are dominated by the three-point function signal (over its two-point counterpart). We demonstrate this on mock catalogs corresponding to the Luminous Red Galaxies (LRGs), Emission-Line Galaxies (ELG), and quasars (QSOs) targeted by the Dark Energy Spectroscopic Instrument (DESI) Survey. The projected three-point function for triangle sides less up to 20$h^{-1}$ Mpc measured from a cubic Gpc of data can constrain the characteristic minimum mass of the LRGs with a precision of $0.46$ %. For comparison, similar constraints from the projected two-point function are $1.55$ %. The improvements for the ELGs and QSOs targets are more modest. In the case of the QSOs it is caused by the high shot-noise of the sample, and in the case of the ELGs, this is caused by the range of halo masses of the host halos. The most time-consuming part of our pipeline is the measurement of the three-point functions. We adopt a tabulation method, proposed in earlier works for the two-point function, to reduce significantly the required compute time for the three-point analysis.

astro-ph.CO↗

Feasibility Analysis and Proof of Model Predictive Control

Sequential and terminal constraint feasibility of the model predictive control (MPC) play important roles in ensuring MPC control continuity. This study thus investigates these two properties theoretically using an MPC model for vehicle platooning and eco-driving strategy at signalized intersections. This study is served as a supplement material for our paper coming soon.

math.OC↗

The DESI $N$-body Simulation Project -- II. Suppressing sample variance with fast simulations

Dark Energy Spectroscopic Instrument (DESI) will construct a large and precise three-dimensional map of our Universe. The survey effective volume reaches $\sim20\Gpchcube$. It is a great challenge to prepare high-resolution simulations with a much larger volume for validating the DESI analysis pipelines. \textsc{AbacusSummit} is a suite of high-resolution dark-matter-only simulations designed for this purpose, with $200\Gpchcube$ (10 times DESI volume) for the base cosmology. However, further efforts need to be done to provide a more precise analysis of the data and to cover also other cosmologies. Recently, the CARPool method was proposed to use paired accurate and approximate simulations to achieve high statistical precision with a limited number of high-resolution simulations. Relying on this technique, we propose to use fast quasi-$N$-body solvers combined with accurate simulations to produce accurate summary statistics. This enables us to obtain 100 times smaller variance than the expected DESI statistical variance at the scales we are interested in, e.g. $k < 0.3\hMpc$ for the halo power spectrum. In addition, it can significantly suppress the sample variance of the halo bispectrum. We further generalize the method for other cosmologies with only one realization in \textsc{AbacusSummit} suite to extend the effective volume $\sim 20$ times. In summary, our proposed strategy of combining high-fidelity simulations with fast approximate gravity solvers and a series of variance suppression techniques sets the path for a robust cosmological analysis of galaxy survey data.

astro-ph.CO↗

Ultrastrong Coupling of Band-Nested Excitons in Few-Layer Molybdenum Disulphide

The two-dimensional transition-metal dichalcogenides (2D TMDCs) are an intriguing platform for studying light-matter interactions because they combine the electronic properties of conventional semiconductors with the optical resonances found in organic systems. However, the coupling strengths demonstrated in strong exciton-polariton coupling remain much lower than those found in organic systems. In this paper, we take on a new approach by utilizing the large oscillator strength of the above-band gap C exciton in few-layer molybdenum disulphide ($\text{FL-MoS}_2$). We show a k-space Rabi splitting of 293 meV when coupling $\text{FL-MoS}_2$ C excitons to surface plasmon polaritons at room temperature. This value is 11% of the uncoupled exciton energy (2.67 eV or 464 nm), ~2x what is typically seen in the TMDCs, placing the system in the ultrastrong coupling regime. Our results take a step towards finally achieving the efficient quantum coherent processes of ultrastrong coupling in a CMOS-compatible system -- the 2D TMDCs -- in the visible spectrum.

cond-mat.mes-hall↗

Risk-Aware Control and Optimization for High-Renewable Power Grids

The transition of the electrical power grid from fossil fuels to renewable sources of energy raises fundamental challenges to the market-clearing algorithms that drive its operations. Indeed, the increased stochasticity in load and the volatility of renewable energy sources have led to significant increases in prediction errors, affecting the reliability and efficiency of existing deterministic optimization models. The RAMC project was initiated to investigate how to move from this deterministic setting into a risk-aware framework where uncertainty is quantified explicitly and incorporated in the market-clearing optimizations. Risk-aware market-clearing raises challenges on its own, primarily from a computational standpoint. This paper reviews how RAMC approaches risk-aware market clearing and presents some of its innovations in uncertainty quantification, optimization, and machine learning. Experimental results on real networks are presented.

math.OC↗

The DESI $N$-body Simulation Project I: Testing the Robustness of Simulations for the DESI Dark Time Survey

Analysis of large galaxy surveys requires confidence in the robustness of numerical simulation methods. The simulations are used to construct mock galaxy catalogs to validate data analysis pipelines and identify potential systematics. We compare three $N$-body simulation codes, ABACUS, GADGET, and SWIFT, to investigate the regimes in which their results agree. We run $N$-body simulations at three different mass resolutions, $6.25\times10^{8}$, $2.11\times10^{9}$, and $5.00\times10^{9}~h^{-1}$M$_{\odot}$, matching phases to reduce the noise within the comparisons. We find systematic errors in the halo clustering between different codes are smaller than the DESI statistical error for $s > 20\, h^{-1}$Mpc in the correlation function in redshift space. Through the resolution comparison we find that simulations run with a mass resolution of $2.1\times10^{9}~h^{-1}$M$_{\odot}$ are sufficiently converged for systematic effects in the halo clustering to be smaller than the DESI statistical error at scales larger than $20 \, h^{-1}$Mpc. These findings show that the simulations are robust for extracting cosmological information from large scales which is the key goal of the DESI survey. Comparing matter power spectra, we find the codes agree to within 1% for $k \leq 10~h$Mpc$^{-1}$. We also run a comparison of three initial condition generation codes and find good agreement. In addition, we include a quasi-$N$-body code, FastPM, since we plan use it for certain DESI analyses. The impact of the halo definition and galaxy-halo relation will be presented in a follow up study.

astro-ph.CO↗

Resiliency of On-Demand Multimodal Transit Systems During a Pandemic

During the COVID-19 pandemic, the collapse of the public transit ridership led to significant budget deficits due to dramatic decreases in fare revenues. Additionally, public transit agencies are facing challenges of reduced vehicle capacity due to social distancing requirements, additional costs of cleaning and protective equipment, and increased downtime for vehicle cleaning. Due to these constraints on resources and budgets, many transit agencies have adopted essential service plans with reduced service hours, number of routes, or frequencies. This paper studies the resiliency during a pandemic of On-Demand Multimodal Transit Systems (ODMTS), a new generation of transit systems that combine a network of high-frequency trains and buses with on-demand shuttles to serve the first and last miles and act as feeders to the fixed network. It presents a case study for the city of Atlanta and evaluates ODMTS for multiple scenarios of depressed demand and social distancing representing various stages of the pandemic. The case study relies on an optimization pipeline that provides an end-to-end ODMTS solution by bringing together methods for demand estimation, network design, fleet sizing, and real-time dispatching. These methods are adapted to work in a multimodal setting and to satisfy practical constraints. In particular, a limit is imposed on the number of passenger transfers, and a new network design model is introduced to avoid the computational burden stemming from this constraint. Real data from the Metropolitan Atlanta Rapid Transit Authority (MARTA) is used to conduct the case study, and the results are evaluated with a high-fidelity simulation. The case study demonstrates how ODMTS provide a resilient solution in terms of cost, convenience, and accessibility for this wide range of scenarios.

math.OC↗

Manifold Coordinates with Physical Meaning

Manifold embedding algorithms map high-dimensional data down to coordinates in a much lower-dimensional space. One of the aims of dimension reduction is to find intrinsic coordinates that describe the data manifold. The coordinates returned by the embedding algorithm are abstract, and finding their physical or domain-related meaning is not formalized and often left to domain experts. This paper studies the problem of recovering the meaning of the new low-dimensional representation in an automatic, principled fashion. We propose a method to explain embedding coordinates of a manifold as non-linear compositions of functions from a user-defined dictionary. We show that this problem can be set up as a sparse linear Group Lasso recovery problem, find sufficient recovery conditions, and demonstrate its effectiveness on data.

stat.ML↗