Aurélien Ouattara | Searcharxiv

arXiv2021

Scalable Econometrics on Big Data -- The Logistic Regression on Spark

Extra-large datasets are becoming increasingly accessible, and computing tools designed to handle huge amount of data efficiently are democratizing rapidly. However, conventional statistical and econometric tools are still lacking fluency when dealing with such large datasets. This paper dives into econometrics on big datasets, specifically focusing on the logistic regression on Spark. We review the robustness of the functions available in Spark to fit logistic regression and introduce a package that we developed in PySpark which returns the statistical summary of the logistic regression, necessary for statistical inference.

stat.CO↗

arXiv2017

Duality Approach to Bilevel Programs with a Convex Lower Level

Bilevel programs are optimization problems where some variables are solutions to optimization problems themselves, and they arise in a variety of control applications, including: control of vehicle traffic networks, inverse reinforcement learning and inverse optimization, and robust control for human-automation systems. This paper develops a duality-based approach to solving bilevel programs where the lower level problem is convex. Our approach is to use partial dualization to construct a new dual function that is differentiable, unlike the Lagrangian dual that is only directionally differentiable. We use our dual to define a duality-based reformulation of bilevel programs, prove equivalence of our reformulation with the original bilevel program, and then introduce regularization to ensure constraint qualification holds. These technical results about our new dual and regularized duality-based reformulation are used to provide theoretical justification for an algorithm we construct for solving bilevel programs with a convex lower level, and we conclude by demonstrating the efficacy of our algorithm by solving two practical instances of bilevel programs.

math.OC↗