arXiv subjects

Wei Ma

Publications and source records attributed to Wei Ma.

At least 145 records · Page 8Linked to original sources

arXiv2021

Short-term origin-destination demand prediction in urban rail transit systems: A channel-wise attentive split-convolutional neural network method

Short-term origin-destination (OD) flow prediction in urban rail transit (URT) plays a crucial role in smart and real-time URT operation and management. Different from other short-term traffic forecasting methods, the short-term OD flow prediction possesses three unique characteristics: (1) data availability: real-time OD flow is not available during the prediction; (2) data dimensionality: the dimension of the OD flow is much higher than the cardinality of transportation networks; (3) data sparsity: URT OD flow is spatiotemporally sparse. There is a great need to develop novel OD flow forecasting method that explicitly considers the unique characteristics of the URT system. To this end, a channel-wise attentive split-convolutional neural network (CAS-CNN) is proposed. The proposed model consists of many novel components such as the channel-wise attention mechanism and split CNN. In particular, an inflow/outflow-gated mechanism is innovatively introduced to address the data availability issue. We further originally propose a masked loss function to solve the data dimensionality and data sparsity issues. The model interpretability is also discussed in detail. The CAS-CNN model is tested on two large-scale real-world datasets from Beijing Subway, and it outperforms the rest of benchmarking methods. The proposed model contributes to the development of short-term OD flow prediction, and it also lays the foundations of real-time URT operation and management.

eess.SP↗

arXiv2020

A general theory of regression adjustment for covariate-adaptive randomization: OLS, Lasso, and beyond

We consider the problem of estimating and inferring treatment effects in randomized experiments. In practice, stratified randomization, or more generally, covariate-adaptive randomization, is routinely used in the design stage to balance the treatment allocations with respect to a few variables that are most relevant to the outcomes. Then, regression is performed in the analysis stage to adjust the remaining imbalances to yield more efficient treatment effect estimators. Building upon and unifying the recent results obtained for ordinary least squares adjusted estimators under covariate-adaptive randomization, this paper presents a general theory of regression adjustment that allows for arbitrary model misspecification and the presence of a large number of baseline covariates. We exemplify the theory on two Lasso-adjusted treatment effect estimators, both of which are optimal in their respective classes. In addition, nonparametric consistent variance estimators are proposed to facilitate valid inferences, which work irrespective of the specific randomization methods used. The robustness and improved efficiency of the proposed estimators are demonstrated through a simulation study and a clinical trial example. This study sheds light on improving treatment effect estimation efficiency by implementing machine learning methods in covariate-adaptive randomized experiments.

stat.ME↗

arXiv2020

Statistical Inference for Covariate-Adaptive Randomization Procedures

Covariate-adaptive randomization (CAR) procedures are frequently used in comparative studies to increase the covariate balance across treatment groups. However, because randomization inevitably uses the covariate information when forming balanced treatment groups, the validity of classical statistical methods after such randomization is often unclear. In this article, we derive the theoretical properties of statistical methods based on general CAR under the linear model framework. More importantly, we explicitly unveil the relationship between covariate-adaptive and inference properties by deriving the asymptotic representations of the corresponding estimators. We apply the proposed general theory to various randomization procedures such as complete randomization, rerandomization, pairwise sequential randomization, and Atkinson's $D_A$-biased coin design and compare their performance analytically. Based on the theoretical results, we then propose a new approach to obtain valid and more powerful tests. These results open a door to understand and analyze experiments based on CAR. Simulation studies provide further evidence of the advantages of the proposed framework and the theoretical results. Supplementary materials for this article are available online.

math.ST↗

arXiv2020

Regression analysis for covariate-adaptive randomization: A robust and efficient inference perspective

Linear regression is arguably the most fundamental statistical model; however, the validity of its use in randomized clinical trials, despite being common practice, has never been crystal clear, particularly when stratified or covariate-adaptive randomization is used. In this paper, we investigate several of the most intuitive and commonly used regression models for estimating and inferring the treatment effect in randomized clinical trials. By allowing the regression model to be arbitrarily misspecified, we demonstrate that all these regression-based estimators robustly estimate the treatment effect, albeit with possibly different efficiency. We also propose consistent non-parametric variance estimators and compare their performances to those of the model-based variance estimators that are readily available in standard statistical software. Based on the results and taking into account both theoretical efficiency and practical feasibility, we make recommendations for the effective use of regression under various scenarios. For equal allocation, it suffices to use the regression adjustment for the stratum covariates and additional baseline covariates, if available, with the usual ordinary-least-squares variance estimator. For unequal allocation, regression with treatment-by-covariate interactions should be used, together with our proposed variance estimators. These recommendations apply to simple and stratified randomization, and minimization, among others. We hope this work helps to clarify and promote the usage of regression in randomized clinical trials.

stat.ME↗

arXiv2020

Efficient Estimation of Mixture Cure Frailty Model for Clustered Current Status Data

Current status data abounds in the field of epidemiology and public health, where the only observable data for a subject is the random inspection time and the event status at inspection. Motivated by such a current status data from a periodontal study where data are inherently clustered, we propose a unified methodology to analyze such complex data. We allow the time-to-event to follow the semiparametric GOR model with a cure fraction, and develop a unified estimation scheme powered by the EM algorithm. The within-subject correlation is accounted for by a random (frailty) effect, and the non-parametric component of the GOR model is approximated via penalized splines, with a set of knot points that increases with the sample size. Proposed methodology is accompanied by a rigorous asymptotic theory, and the related semiparametric efficiency. The finite sample performance of our model parameters are assessed via simulation studies. Furthermore, the proposed methodology is illustrated via application to the oral health data, accompanied by diagnostic checks to identify influential observations. An easy to use R package CRFCSD is also available for implementation.

stat.ME↗

arXiv2019

Measuring and reducing the disequilibrium levels of dynamic networks through ride-sourcing vehicle data

Transportation systems are being reshaped by ride-sourcing and shared mobility services in recent years. The transportation network companies (TNCs) have been collecting high-granular ride-sourcing vehicle (RV) trajectory data over the past decade, while it is still unclear how the RV data can improve current dynamic network modeling for network traffic management. This paper proposes to statistically estimate network disequilibrium level (NDL), namely to what extent the dynamic user equilibrium (DUE) conditions are deviated in real-world networks. Using the data based on RV trajectories, we present a novel method to estimate the real-world NDL measure. More importantly, we present a method to compute zone-to-zone travel time data from trajectory-level RV data. This would become a data-sharing scheme for TNCs such that, while being used to effectively estimate and reduce NDL, the zone-to-zone data reveals neither personally identifiable information nor trip-level business information if shared with the public. In addition, we present an NDL based traffic management method to perform user optimal routing on a small fraction of vehicles in the network. The NDL measures and NDL-based routing are examined on two real-world large-scale networks: the City of Chengdu with trajectory-level RV data and the City of Pittsburgh with zone-to-zone travel time data. We found that, on weekdays in each city, NDLs are likely high when travel demand is high (thus when congestion is mild or heavy).

physics.soc-ph↗

arXiv2019

Missing Not at Random in Matrix Completion: The Effectiveness of Estimating Missingness Probabilities Under a Low Nuclear Norm Assumption

Matrix completion is often applied to data with entries missing not at random (MNAR). For example, consider a recommendation system where users tend to only reveal ratings for items they like. In this case, a matrix completion method that relies on entries being revealed at uniformly sampled row and column indices can yield overly optimistic predictions of unseen user ratings. Recently, various papers have shown that we can reduce this bias in MNAR matrix completion if we know the probabilities of different matrix entries being missing. These probabilities are typically modeled using logistic regression or naive Bayes, which make strong assumptions and lack guarantees on the accuracy of the estimated probabilities. In this paper, we suggest a simple approach to estimating these probabilities that avoids these shortcomings. Our approach follows from the observation that missingness patterns in real data often exhibit low nuclear norm structure. We can then estimate the missingness probabilities by feeding the (always fully-observed) binary matrix specifying which entries are revealed or missing to an existing nuclear-norm-constrained matrix completion algorithm by Davenport et al. [2014]. Thus, we tackle MNAR matrix completion by solving a different matrix completion problem first that recovers missingness probabilities. We establish finite-sample error bounds for how accurate these probability estimates are and how well these estimates debias standard matrix completion losses for the original matrix to be completed. Our experiments show that the proposed debiasing strategy can improve a variety of existing matrix completion algorithms, and achieves downstream matrix completion accuracy at least as good as logistic regression and naive Bayes debiasing baselines that require additional auxiliary information.

stat.ML↗

arXiv2019

High-Resolution Traffic Sensing with Autonomous Vehicles

The last decades have witnessed the breakthrough of autonomous vehicles (AVs), and the perception capabilities of AVs have been dramatically improved. Various sensors installed on AVs, including, but are not limited to, LiDAR, radar, camera and stereovision, will be collecting massive data and perceiving the surrounding traffic states continuously. In fact, a fleet of AVs can serve as floating (or probe) sensors, which can be utilized to infer traffic information while cruising around the roadway networks. In contrast, conventional traffic sensing methods rely on fixed traffic sensors such as loop detectors, cameras and microwave vehicle detectors. Due to the high cost of conventional traffic sensors, traffic state data are usually obtained in a low-frequency and sparse manner. In view of this, this paper leverages rich data collected through AVs to propose the high-resolution traffic sensing framework. The proposed framework estimates the fundamental traffic state variables, namely, flow, density and speed in high spatio-temporal resolution, and it is developed under different levels of AV perception capabilities and low AV market penetration rate. The Next Generation Simulation (NGSIM) data is adopted to examine the accuracy and robustness of the proposed framework. Experimental results show that the proposed estimation framework achieves high accuracy even with low AV market penetration rate. Sensitivity analysis regarding AV penetration rate, sensor configuration, and perception accuracy will also be studied. This study will help policymakers and private sectors (e.g Uber, Waymo) to understand the values of AVs, especially the values of massive data collected by AVs, in traffic operation and management.

eess.SP↗

arXiv2019

Tannin-controlled micelles and fibrils of $κ$-casein

Effects of green tea tannin epigallocatechin-gallate (EGCG) on thermal-stress-induced amyloid fibril formation of reduced carboxymethylated bovine milk protein $κ$-casein (RCMK) were studied by dynamical light scattering (DLS) and small angle x-rays scattering (SAXS). Two populations of aggregates, micelles and fibrils, dominated the time evolution of light scattering intensity and of effective hydrodynamic diameter. SAXS experiments allowed to resolve micelles and fibrils so that the time dependence of scattering profile revealed structural evolution of the two populations. The low-Q scattering intensity prior to an expected increase with time due to fibril growth, shows an intriguing rapid decrease which is interpreted as the release of monomers from micelles. This phenomenon, observed both in the absence and in the presence of EGCG, indicates that under thermal stress free native monomers are converted to amyloid-prone monomers that do not form micelles. The consumption of free native monomers results in a release of native monomers from micelles, because only native protein participate in micelle-monomer (quasi-)equilibrium. This release is reversible, indicating also that native-to-amyloid-prone monomers conversion is reversible as well. We show that EGCG does not bind to protein in fibrils, neither does it affect/prevent the pro-amyloid conversion of monomers. EGCG hinders the addition of monomers to growing fibrils. These facts allowed us to propose kinetics model for EGCG-controlled amyloid aggregation of micellar proteins. Therein, we introduced the growth-rate inhibition function which quantitatively accounts for the effect of EGCG on the fibril growth at any degree of thermal stress.

physics.chem-ph↗

arXiv2019

A deep learning approach to real-time parking occupancy prediction in spatio-temporal networks incorporating multiple spatio-temporal data sources

A deep learning model is applied for predicting block-level parking occupancy in real time. The model leverages Graph-Convolutional Neural Networks (GCNN) to extract the spatial relations of traffic flow in large-scale networks, and utilizes Recurrent Neural Networks (RNN) with Long-Short Term Memory (LSTM) to capture the temporal features. In addition, the model is capable of taking multiple heterogeneously structured traffic data sources as input, such as parking meter transactions, traffic speed, and weather conditions. The model performance is evaluated through a case study in Pittsburgh downtown area. The proposed model outperforms other baseline methods including multi-layer LSTM and Lasso with an average testing MAPE of 10.6\% when predicting block-level parking occupancies 30 minutes in advance. The case study also shows that, in generally, the prediction model works better for business areas than for recreational locations. We found that incorporating traffic speed and weather information can significantly improve the prediction performance. Weather data is particularly useful for improving predicting accuracy in recreational areas.

cs.LG↗

arXiv2019

Test Selection for Deep Learning Systems

Testing of deep learning models is challenging due to the excessive number and complexity of computations involved. As a result, test data selection is performed manually and in an ad hoc way. This raises the question of how we can automatically select candidate test data to test deep learning models. Recent research has focused on adapting test selection metrics from code-based software testing (such as coverage) to deep learning. However, deep learning models have different attributes from code such as spread of computations across the entire network reflecting training data properties, balance of neuron weights and redundancy (use of many more neurons than needed). Such differences make code-based metrics inappropriate to select data that can challenge the models (can trigger misclassification). We thus propose a set of test selection metrics based on the notion of model uncertainty (model confidence on specific inputs). Intuitively, the more uncertain we are about a candidate sample, the more likely it is that this sample triggers a misclassification. Similarly, the samples for which we are the most uncertain, are the most informative and should be used to improve the model by retraining. We evaluate these metrics on two widely-used image classification problems involving real and artificial (adversarial) data. We show that uncertainty-based metrics have a strong ability to select data that are misclassified and lead to major improvement in classification accuracy during retraining: up to 80% more gain than random selection and other state-of-the-art metrics on one dataset and up to 29% on the other.

cs.LG↗

arXiv2019

Estimating multi-class dynamic origin-destination demand through a forward-backward algorithm on computational graphs

Transportation networks are unprecedentedly complex with heterogeneous vehicular flow. Conventionally, vehicle classes are considered by vehicle classifications (such as standard passenger cars and trucks). However, vehicle flow heterogeneity stems from many other aspects in general, e.g., ride-sourcing vehicles versus personal vehicles, human driven vehicles versus connected and automated vehicles. Provided with some observations of vehicular flow for each class in a large-scale transportation network, how to estimate the multi-class spatio-temporal vehicular flow, in terms of time-varying Origin-Destination (OD) demand and path/link flow, remains a big challenge. This paper presents a solution framework for multi-class dynamic OD demand estimation (MCDODE) in large-scale networks. The proposed framework is built on a computational graph with tensor representations of spatio-temporal flow and all intermediate features involved in the MCDODE formulation. A forward-backward algorithm is proposed to efficiently solve the MCDODE formulation on computational graphs. In addition, we propose a novel concept of tree-based cumulative curves to estimate the gradient of OD demand. A Growing Tree algorithm is developed to construct tree-based cumulative curves. The proposed framework is examined on a small network as well as a real-world large-scale network. The experiment results indicate that the proposed framework is compelling, satisfactory and computationally plausible.

eess.SY↗

arXiv2019

Probabilistic representation and inverse design of metamaterials based on a deep generative model with semi-supervised learning strategy

The research of metamaterials has achieved enormous success in the manipulation of light in an artificially prescribed manner using delicately designed sub-wavelength structures, so-called meta-atoms. Even though modern numerical methods allow to accurately calculate the optical response of complex structures, the inverse design of metamaterials is still a challenging task due to the non-intuitive and non-unique relationship between physical structures and optical responses. To better unveil this implicit relationship and thus facilitate metamaterial design, we propose to represent metamaterials and model the inverse design problem in a probabilistically generative manner. By employing an encoder-decoder configuration, our deep generative model compresses the meta-atom design and optical response into a latent space, where similar designs and similar optical responses are automatically clustered together. Therefore, by sampling in the latent space, the stochastic latent variables function as codes, from which the candidate designs are generated upon given requirements in a decoding process. With the effective latent representation of metamaterials, we can elegantly model the complex structure-performance relationship in an interpretable way, and solve the one-to-many mapping issue that is intractable in a deterministic model. Moreover, to alleviate the burden of numerical calculation in data collection, we develop a semi-supervised learning strategy that allows our model to utilize unlabeled data in addition to labeled data during training, simultaneously optimizing the generative inverse design and deterministic forward prediction in an end-to-end manner. On a data-driven basis, the proposed model can serve as a comprehensive and efficient tool that accelerates the design, characterization and even new discovery in the research domain of metamaterials and photonics in general.

physics.optics↗

arXiv2019

Statistical inference of probabilistic origin-destination demand using day-to-day traffic data

Recent transportation network studies on uncertainty and reliability call for modeling the probabilistic O-D demand and probabilistic network flow. Making the best use of day-to-day traffic data collected over many years, this paper develops a novel theoretical framework for estimating the mean and variance/covariance matrix of O-D demand considering the day-to-day variation induced by travelers' independent route choices. It also estimates the probability distributions of link/path flow and their travel cost where the variance stems from three sources, O-D demand, route choice and unknown errors. The framework estimates O-D demand mean and variance/covariance matrix iteratively, also known as iterative generalized least squares (IGLS) in statistics. Lasso regularization is employed to obtain sparse covariance matrix for better interpretation and computational efficiency. Though the probabilistic O-D estimation (ODE) works with a much larger solution space than the deterministic ODE, we show that its estimator for O-D demand mean is no worse than the best possible estimator by an error that reduces with the increase in sample size. The probabilistic ODE is examined on two small networks and two real-world large-scale networks. The solution converges quickly under the IGLS framework. In all those experiments, the results of the probabilistic ODE are compelling, satisfactory and computationally plausible. Lasso regularization on the covariance matrix estimation leans to underestimate most of variance/covariance entries. A proper Lasso penalty ensures a good trade-off between bias and variance of the estimation.

stat.ME↗

arXiv2019

Estimating multi-year 24/7 origin-destination demand using high-granular multi-source traffic data

Dynamic origin-destination (OD) demand is central to transportation system modeling and analysis. The dynamic OD demand estimation problem (DODE) has been studied for decades, most of which solve the DODE problem on a typical day or several typical hours. There is a lack of methods that estimate high-resolution dynamic OD demand for a sequence of many consecutive days over several years (referred to as 24/7 OD in this research). Having multi-year 24/7 OD demand would allow a better understanding of characteristics of dynamic OD demands and their evolution/trends over the past few years, a critical input for modeling transportation system evolution and reliability. This paper presents a data-driven framework that estimates day-to-day dynamic OD using high-granular traffic counts and speed data collected over many years. The proposed framework statistically clusters daily traffic data into typical traffic patterns using t-Distributed Stochastic Neighbor Embedding (t-SNE) and k-means methods. A GPU-based stochastic projected gradient descent method is proposed to efficiently solve the multi-year 24/7 DODE problem. It is demonstrated that the new method efficiently estimates the 5-minute dynamic OD demand for every single day from 2014 to 2016 on I-5 and SR-99 in the Sacramento region. The resultant multi-year 24/7 dynamic OD demand reveals the daily, weekly, monthly, seasonal and yearly change in travel demand in a region, implying intriguing demand characteristics over the years.

eess.SY↗

arXiv2018

An Interpretable Produce Price Forecasting System for Small and Marginal Farmers in India using Collaborative Filtering and Adaptive Nearest Neighbors

Small and marginal farmers, who account for over 80% of India's agricultural population, often sell their harvest at low, unfavorable prices before spoilage. These farmers often lack access to either cold storage or market forecasts. In particular, by having access to cold storage, farmers can store their produce for longer and thus have more flexibility as to when they should sell their harvest by. Meanwhile, by having access to market forecasts, farmers can more easily identify which markets to sell at and when. While affordable cold storage solutions have become more widely available, there has been less work on produce price forecasting. A key challenge is that in many regions of India, predominantly in rural and remote areas, we have either very limited or no produce pricing data available from public online sources. In this paper, we present a produce price forecasting system that pulls data from the Indian Ministry of Agriculture and Farmers Welfare's website Agmarknet, trains a model of prices using over a thousand markets, and displays interpretable price forecasts in a web application viewable from a mobile phone. Due to the pricing data being extremely sparse, our method first imputes missing entries using collaborative filtering to obtain a dense dataset. Using this imputed dense dataset, we then train a decision-tree-based classifier to predict produce prices at different markets. In terms of interpretability, we display the most relevant historical pricing data that drive each forecasted price, where we take advantage of the fact that a wide family of decision-tree-based ensemble learning methods are adaptive nearest neighbor methods. We show how to construct heuristic price uncertainty intervals based on nearest neighbors. We validate forecast accuracy on data from Agmarknet and a small field survey of a few markets in Odisha.

cs.CY↗

arXiv2018

Photoexcitation Induced Quantum Dynamics of Charge Density Wave and Emergence of a Collective Mode in 1T-TaS$_2$

Photoexcitation is a powerful means in distinguishing different interactions and manipulating the states of matter, especially in complex quantum systems. As a well-known charge density wave (CDW) material, 1T-TaS2 has been widely studied experimentally thanks to its intriguing photoexcited responses. However, the microscopic atomic dynamics and underlying mechanism are still under debate. Here, we demonstrate photoexcitation induced ultrafast dynamics in 1T-TaS2 using time-dependent density functional theory molecular dynamics. We discover a novel collective mode induced by photodoping, which is significantly different from thermally-induced phonon mode in TaS2. In addition, our finding validates nonthermal melting of CDW induced at low light intensities, supporting that conventional hot electron model is inadequate to explain photoinduced dynamics. Our results provide a deep insight on coherent electron and lattice quantum dynamics during the formation and excitation of CDW in 1T-TaS2.

cond-mat.str-el↗

arXiv2018

Pairwise Sequential Randomization and Its Properties

In comparative studies, such as in causal inference and clinical trials, balancing important covariates is often one of the most important concerns for both efficient and credible comparison. However, chance imbalance still exists in many randomized experiments. This phenomenon of covariate imbalance becomes much more serious as the number of covariates $p$ increases. To address this issue, we introduce a new randomization procedure, called pairwise sequential randomization (PSR). The proposed method allocates the units sequentially and adaptively, using information on the current level of imbalance and the incoming unit's covariate. With a large number of covariates or a large number of units, the proposed method shows substantial advantages over the traditional methods in terms of the covariate balance, estimation accuracy, and computational time, making it an ideal technique in the era of big data. The proposed method attains the optimal covariate balance, in the sense that the estimated treatment effect under the proposed method attains its minimum variance asymptotically. Also the proposed method is widely applicable in both causal inference and clinical trials. Numerical studies and real data analysis provide further evidence of the advantages of the proposed method.

stat.ME↗