arXiv subjects

Henry Lam

Publications and source records attributed to Henry Lam.

At least 91 records · Page 5Linked to original sources

arXiv2019

Combating Conservativeness in Data-Driven Optimization under Uncertainty: A Solution Path Approach

In data-driven optimization, solution feasibility is often ensured through a "safe" reformulation of the uncertain constraints, such that an obtained data-driven solution is guaranteed to be feasible for the oracle formulation with high statistical confidence. Such approaches generally involve an implicit estimation of the whole feasible set that can scale rapidly with the problem dimension, in turn leading to over-conservative solutions. In this paper, we investigate a validation-based strategy to avoid set estimation by exploiting the intrinsic low dimensionality among all possible solutions output from a given reformulation. We demonstrate how our obtained solutions satisfy statistical feasibility guarantees with light dimension dependence, and how they are asymptotically optimal and thus regarded as the least conservative with respect to the considered reformulation classes. We apply this strategy to several data-driven optimization paradigms including (distributionally) robust optimization, sample average approximation and scenario optimization. Numerical experiments show encouraging performances of our strategy compared to established benchmarks.

math.OC↗

arXiv2019

Evaluation Uncertainty in Data-Driven Self-Driving Testing

Safety evaluation of self-driving technologies has been extensively studied. One recent approach uses Monte Carlo based evaluation to estimate the occurrence probabilities of safety-critical events as safety measures. These Monte Carlo samples are generated from stochastic input models constructed based on real-world data. In this paper, we propose an approach to assess the impact on the probability estimates from the evaluation procedures due to the estimation error caused by data variability. Our proposed method merges the classical bootstrap method for estimating input uncertainty with a likelihood ratio based scheme to reuse experiment outputs. This approach is economical and efficient in terms of implementation costs in assessing input uncertainty for the evaluation of self-driving technology. We use an example in autonomous vehicle (AV) safety evaluation to demonstrate the proposed approach as a diagnostic tool for the quality of the fitted input model.

stat.ME↗

arXiv2019

Optimization-based Quantification of Simulation Input Uncertainty via Empirical Likelihood

We study an optimization-based approach to construct statistically accurate confidence intervals for simulation performance measures under nonparametric input uncertainty. This approach computes confidence bounds from simulation runs driven by probability weights defined on the data, which are obtained from solving optimization problems under suitably posited averaged divergence constraints. We illustrate how this approach offers benefits in computational efficiency and finite-sample performance compared to the bootstrap and the delta method. While resembling robust optimization, we explain the procedural design and develop tight statistical guarantees of this approach via a generalization of the empirical likelihood method.

stat.ME↗

arXiv2019

Enhanced Balancing of Bias-Variance Tradeoff in Stochastic Estimation: A Minimax Perspective

Biased stochastic estimators, such as finite-differences for noisy gradient estimation, often contain parameters that need to be properly chosen to balance impacts from the bias and the variance. While the optimal order of these parameters in terms of the simulation budget can be readily established, the precise best values depend on model characteristics that are typically unknown in advance. We introduce a framework to construct new classes of estimators, based on judicious combinations of simulation runs on sequences of tuning parameter values, such that the estimators consistently outperform a given tuning parameter choice in the conventional approach, regardless of the unknown model characteristics. We argue the outperformance via what we call the asymptotic minimax risk ratio, obtained by minimizing the worst-case asymptotic ratio between the mean square errors of our estimators and the conventional one, where the worst case is over any possible values of the model unknowns. In particular, when the minimax ratio is less than 1, the calibrated estimator is guaranteed to perform better asymptotically. We identify this minimax ratio for general classes of weighted estimators, and the regimes where this ratio is less than 1. Moreover, we show that the best weighting scheme is characterized by a sum of two components with distinct decay rates. We explain how this arises from bias-variance balancing that combats the adversarial selection of the model constants, which can be analyzed via a tractable reformulation of a non-convex optimization problem.

stat.ME↗

arXiv2018

Synthesis of Different Autonomous Vehicles Test Approaches

Currently, the most prevalent way to evaluate an autonomous vehicle is to directly test it on the public road. However, because of recent accidents caused by autonomous vehicles, it becomes controversial about whether on-road tests should be the best approach. Alternatively, people use test tracks or simulation to assess the safety of autonomous vehicles. These approaches are time-efficient and less costly, however, their credibility varies. In this paper, we propose to use a co-Kriging model to synthesize the results from different evaluation approaches, which allows us to fully utilize the information and provides an accurate, affordable, and safe way to assess a design of an autonomous vehicle.

stat.AP↗

arXiv2018

Robust Analysis in Stochastic Simulation: Computation and Performance Guarantees

Any performance analysis based on stochastic simulation is subject to the errors inherent in misspecifying the modeling assumptions, particularly the input distributions. In situations with little support from data, we investigate the use of worst-case analysis to analyze these errors, by representing the partial, nonparametric knowledge of the input models via optimization constraints. We study the performance and robustness guarantees of this approach. We design and analyze a numerical scheme for solving a general class of simulation objectives and uncertainty specifications. The key steps involve a randomized discretization of the probability spaces, a simulable unbiased gradient estimator using a nonparametric analog of the likelihood ratio method, and a Frank-Wolfe (FW) variant of the stochastic approximation (SA) method (which we call FWSA) run on the space of input probability distributions. A convergence analysis for FWSA on non-convex problems is provided. We test the performance of our approach via several numerical examples.

math.PR↗

arXiv2018

Robust and Parallel Bayesian Model Selection

Effective and accurate model selection is an important problem in modern data analysis. One of the major challenges is the computational burden required to handle large data sets that cannot be stored or processed on one machine. Another challenge one may encounter is the presence of outliers and contaminations that damage the inference quality. The parallel "divide and conquer" model selection strategy divides the observations of the full data set into roughly equal subsets and perform inference and model selection independently on each subset. After local subset inference, this method aggregates the posterior model probabilities or other model/variable selection criteria to obtain a final model by using the notion of geometric median. This approach leads to improved concentration in finding the "correct" model and model parameters and also is provably robust to outliers and data contamination.

stat.ML↗

arXiv2018

Optimization-based Calibration of Simulation Input Models

Studies on simulation input uncertainty often built on the availability of input data. In this paper, we investigate an inverse problem where, given only the availability of output data, we nonparametrically calibrate the input models and other related performance measures of interest. We propose an optimization-based framework to compute statistically valid bounds on input quantities. The framework utilizes constraints that connect the statistical information of the real-world outputs with the input-output relation via a simulable map. We analyze the statistical guarantees of this approach from the view of data-driven robust optimization, and show how the guarantees relate to the function complexity of the constraints arising in our framework. We investigate an iterative procedure based on a stochastic quadratic penalty method to approximately solve the resulting optimization. We conduct numerical experiments to demonstrate our performance in bounding the input models and related quantities.

math.OC↗

arXiv2018

Tail Analysis without Parametric Models: A Worst-case Perspective

A common bottleneck in evaluating extremal performance measures is that, due to their very nature, tail data are often very limited. The conventional approach selects the best probability distribution from tail data using parametric fitting, but the validity of the parametric choice can be difficult to verify. This paper describes an alternative based on the computation of worst-case bounds under the geometric premise of tail convexity, a feature shared by all common parametric tail distributions. We characterize the optimality structure of the resulting optimization problem, and demonstrate that the worst-case convex tail behavior is in a sense either extremely light-tailed or extremely heavy-tailed. We develop low-dimensional nonlinear programs that distinguish between the two cases and compute the worst-case bound. We numerically illustrate how the proposed approach can give more reliable performances than conventional parametric methods.

stat.CO↗

arXiv2017

On Optimization over Tail Distributions

We investigate the use of optimization to compute bounds for extremal performance measures. This approach takes a non-parametric viewpoint that aims to alleviate the issue of model misspecification possibly encountered by conventional methods in extreme event analysis. We make two contributions towards solving these formulations, paying especial attention to the arising tail issues. First, we provide a technique in parallel to Choquet's theory, via a combination of integration by parts and change of measures, to transform shape constrained problems (e.g., monotonicity of derivatives) into families of moment problems. Second, we show how a moment problem cast over infinite support can be reformulated into a problem over compact support with an additional slack variable. In the context of optimization over tail distributions, the latter helps resolve the issue of non-convergence of solutions when using algorithms such as generalized linear programming. We further demonstrate the applicability of this result to problems with infinite-value constraints, which can arise in modeling heavy tails.

stat.ME↗

arXiv2017

A Versatile Approach to Evaluating and Testing Automated Vehicles based on Kernel Methods

Evaluation and validation of complicated control systems are crucial to guarantee usability and safety. Usually, failure happens in some very rarely encountered situations, but once triggered, the consequence is disastrous. Accelerated Evaluation is a methodology that efficiently tests those rarely-occurring yet critical failures via smartly-sampled test cases. The distribution used in sampling is pivotal to the performance of the method, but building a suitable distribution requires case-by-case analysis. This paper proposes a versatile approach for constructing sampling distribution using kernel method. The approach uses statistical learning tools to approximate the critical event sets and constructs distributions based on the unique properties of Gaussian distributions. We applied the method to evaluate the automated vehicles. Numerical experiments show proposed approach can robustly identify the rare failures and significantly reduce the evaluation time.

cs.LG↗

arXiv2017

An Uncertainty Quantification Method for Inexact Simulation Models

The vast majority of stochastic simulation models are imperfect in that they fail to exactly emulate real system dynamics. The inexactness of the simulation model, or model discrepancy, can impact the predictive accuracy and usefulness of the simulation for decision-making. This paper proposes a systematic framework to integrate data from both the simulation responses and the real system responses to learn this discrepancy and quantify the resulting uncertainty. Our framework addresses the theoretical and computational requirements for stochastic estimation in a Bayesian setting. It involves an optimization-based procedure to compute confidence bounds on the target outputs that elicit desirable large-sample statistical properties. We illustrate the practical value of our framework with a call center example and a manufacturing line case study.

stat.ME↗

arXiv2017

An Accelerated Testing Approach for Automated Vehicles with Background Traffic Described by Joint Distributions

This paper proposes a new framework based on joint statistical models for evaluating risks of automated vehicles in a naturalistic driving environment. The previous studies on the Accelerated Evaluation for automated vehicles are extended from multi-independent-variate models to joint statistics. The proposed toolkit includes exploration of the rare event (e.g. crash) sets and construction of accelerated distributions for Gaussian Mixture models using Importance Sampling techniques. Furthermore, the monotonic property is used to avoid the curse of dimensionality introduced by the joint distributions. Simulation results show that the procedure is effective and has a great potential to reduce the test cost for automated vehicles.

eess.SY↗

arXiv2017

Towards Affordable On-track Testing for Autonomous Vehicle - A Kriging-based Statistical Approach

This paper discusses the use of Kriging model in Automated Vehicle evaluation. We explore how a Kriging model can help reduce the number of experiments or simulations in the Accelerated Evaluation procedure. We also propose an adaptive sampling scheme for selecting samples to construct the Kriging model. Application examples in the lane change scenario are presented to illustrate the proposed methods.

eess.SY↗

arXiv2017

Sequential Experimentation to Efficiently Test Automated Vehicles

Automated vehicles have been under heavy developments in major auto and tech companies and are expected to release into market in the foreseeable future. However, the road safety of these vehicles remains a concern. One approach to evaluate their safety is via on-track experimentation, but this requires gigantic costs and time investments. This paper discusses a sequential learning approach based on kriging models to reduce the experimental runs and economize on-track experimentation. The approach relies on a heuristic simulation-based gradient descent procedure to search for the best next test scenario. We demonstrate our approach with some numerical test cases.

eess.SY↗

arXiv2017

Accelerated Evaluation of Automated Vehicles in Car-Following Maneuvers

The safety of Automated Vehicles (AVs) must be assured before their release and deployment. The current approach to evaluation relies primarily on (i) testing AVs on public roads or (ii) track testing with scenarios defined in a test matrix. These two methods have completely opposing drawbacks: the former, while offering realistic scenarios, takes too much time to execute; the latter, though it can be completed in a short amount of time, has no clear correlation to safety benefits in the real world. To avoid the aforementioned problems, we propose Accelerated Evaluation, focusing on the car-following scenario. The stochastic human-controlled vehicle (HV) motions are modeled based on 1.3 million miles of naturalistic driving data collected by the University of Michigan Safety Pilot Model Deployment Program. The statistics of the HV behaviors are then modified to generate more intense interactions between HVs and AVs to accelerate the evaluation procedure. The Importance Sampling theory was used to ensure that the safety benefits of AVs are accurately assessed under accelerated tests. Crash, injury and conflict rates for a simulated AV are simulated to demonstrate the proposed approach. Results show that test duration is reduced by a factor of 300 to 100,000 compared with the non-accelerated (naturalistic) evaluation. In other words, the proposed techniques have great potential for accelerating the AV evaluation process.

cs.OH↗

arXiv2017

Accelerated Evaluation of Automated Vehicles Using Piecewise Mixture Models

The process to certify highly Automated Vehicles has not yet been defined by any country in the world. Currently, companies test Automated Vehicles on public roads, which is time-consuming and inefficient. We proposed the Accelerated Evaluation concept, which uses a modified statistics of the surrounding vehicles and the Importance Sampling theory to reduce the evaluation time by several orders of magnitude, while ensuring the evaluation results are statistically accurate. In this paper, we further improve the accelerated evaluation concept by using Piecewise Mixture Distribution models, instead of Single Parametric Distribution models. We developed and applied this idea to forward collision control system reacting to vehicles making cut-in lane changes. The behavior of the cut-in vehicles was modeled based on more than 403,581 lane changes collected by the University of Michigan Safety Pilot Model Deployment Program. Simulation results confirm that the accuracy and efficiency of the Piecewise Mixture Distribution method outperformed single parametric distribution methods in accuracy and efficiency, and accelerated the evaluation process by almost four orders of magnitude.

eess.SY↗

arXiv2017

Evaluation of Automated Vehicles in the Frontal Cut-in Scenario - an Enhanced Approach using Piecewise Mixture Models

Evaluation and testing are critical for the development of Automated Vehicles (AVs). Currently, companies test AVs on public roads, which is very time-consuming and inefficient. We proposed the Accelerated Evaluation concept which uses a modified statistics of the surrounding vehicles and the Importance Sampling theory to reduce the evaluation time by several orders of magnitude, while ensuring the final evaluation results are accurate. In this paper, we further extend this idea by using Piecewise Mixture Distribution models instead of Single Distribution models. We demonstrate this idea to evaluate vehicle safety in lane change scenarios. The behavior of the cut-in vehicles was modeled based on more than 400,000 naturalistic driving lane changes collected by the University of Michigan Safety Pilot Model Deployment Program. Simulation results confirm that the accuracy and efficiency of the Piecewise Mixture Distribution method are better than the single distribution.

eess.SY↗