arXiv subjects

Sebastian Lerch

Publications and source records attributed to Sebastian Lerch.

At least 37 records · Page 2Linked to original sources

arXiv2023

Direction Augmentation in the Evaluation of Armed Conflict Predictions

In many forecasting settings, there is a specific interest in predicting the sign of an outcome variable correctly in addition to its magnitude. For instance, when forecasting armed conflicts, positive and negative log-changes in monthly fatalities represent escalation and de-escalation, respectively, and have very different implications. In the ViEWS forecasting challenge, a prediction competition on state-based violence, a novel evaluation score called targeted absolute deviation with direction augmentation (TADDA) has therefore been suggested, which accounts for both for the sign and magnitude of log-changes. While it has a straightforward intuitive motivation, the empirical results of the challenge show that a no-change model always predicting a log-change of zero outperforms all submitted forecasting models under the TADDA score. We provide a statistical explanation for this phenomenon. Analyzing the properties of TADDA, we find that in order to achieve good scores, forecasters often have an incentive to predict no or only modest log-changes. In particular, there is often an incentive to report conservative point predictions considerably closer to zero than the forecaster's actual predictive median or mean. In an empirical application, we demonstrate that a no-change model can be improved upon by tailoring predictions to the particularities of the TADDA score. We conclude by outlining some alternative scoring concepts.

stat.AP↗

arXiv2023

Learning to forecast: The probabilistic time series forecasting challenge

We report on a course project in which students submit weekly probabilistic forecasts of two weather variables and one financial variable. This real-time format allows students to engage in practical forecasting, which requires a diverse set of skills in data science and applied statistics. We describe the context and aims of the course, and discuss design parameters like the selection of target variables, the forecast submission process, the evaluation of forecast performance, and the feedback provided to students. Furthermore, we describe empirical properties of students' probabilistic forecasts, as well as some lessons learned on our part.

stat.OT↗

arXiv2022

Comparison of multivariate post-processing methods using global ECMWF ensemble forecasts

An influential step in weather forecasting was the introduction of ensemble forecasts in operational use due to their capability to account for the uncertainties in the future state of the atmosphere. However, ensemble weather forecasts are often underdispersive and might also contain bias, which calls for some form of post-processing. A popular approach to calibration is the ensemble model output statistics (EMOS) approach resulting in a full predictive distribution for a given weather variable. However, this form of univariate post-processing may ignore the prevailing spatial and/or temporal correlation structures among different dimensions. Since many applications call for spatially and/or temporally coherent forecasts, multivariate post-processing aims to capture these possibly lost dependencies. We compare the forecast skill of different nonparametric multivariate approaches to modeling temporal dependence of ensemble weather forecasts with different forecast horizons. The focus is on two-step methods, where after univariate post-processing, the EMOS predictive distributions corresponding to different forecast horizons are combined to a multivariate calibrated prediction using an empirical copula. Based on global ensemble predictions of temperature, wind speed and precipitation accumulation of the European Centre for Medium-Range Weather Forecasts from January 2002 to March 2014, we investigate the forecast skill of different versions of Ensemble Copula Coupling (ECC) and Schaake Shuffle (SSh). In general, compared with the raw and independently calibrated forecasts, multivariate post-processing substantially improves the forecast skill. While even the simplest ECC approach with low computational cost provides a powerful benchmark method, recently proposed advanced extensions of the ECC and the SSh are found to not provide any significant improvements over their basic counterparts.

stat.AP↗

arXiv2022

Evaluating Ensemble Post-Processing for Wind Power Forecasts

Capturing the uncertainty in probabilistic wind power forecasts is challenging, especially when uncertain input variables, such as the weather, play a role. Since ensemble weather predictions aim to capture the uncertainty in the weather system, they can be used to propagate this uncertainty through to subsequent wind power forecasting models. However, as weather ensemble systems are known to be biased and underdispersed, meteorologists post-process the ensembles. This post-processing can successfully correct the biases in the weather variables but has not been evaluated thoroughly in the context of subsequent forecasts, such as wind power generation forecasts. The present paper evaluates multiple strategies for applying ensemble post-processing to probabilistic wind power forecasts. We use Ensemble Model Output Statistics (EMOS) as the post-processing method and evaluate four possible strategies: only using the raw ensembles without post-processing, a one-step strategy where only the weather ensembles are post-processed, a one-step strategy where we only post-process the power ensembles, and a two-step strategy where we post-process both the weather and power ensembles. Results show that post-processing the final wind power ensemble improves forecast performance regarding both calibration and sharpness, whilst only post-processing the weather ensembles does not necessarily lead to increased forecast performance.

stat.AP↗

arXiv2022

Convolutional autoencoders for spatially-informed ensemble post-processing

Ensemble weather predictions typically show systematic errors that have to be corrected via post-processing. Even state-of-the-art post-processing methods based on neural networks often solely rely on location-specific predictors that require an interpolation of the physical weather model's spatial forecast fields to the target locations. However, potentially useful predictability information contained in large-scale spatial structures within the input fields is potentially lost in this interpolation step. Therefore, we propose the use of convolutional autoencoders to learn compact representations of spatial input fields which can then be used to augment location-specific information as additional inputs to post-processing models. The benefits of including this spatial information is demonstrated in a case study of 2-m temperature forecasts at surface stations in Germany.

cs.LG↗

arXiv2021

Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison

Postprocessing ensemble weather predictions to correct systematic errors has become a standard practice in research and operations. However, only few recent studies have focused on ensemble postprocessing of wind gust forecasts, despite its importance for severe weather warnings. Here, we provide a comprehensive review and systematic comparison of eight statistical and machine learning methods for probabilistic wind gust forecasting via ensemble postprocessing, that can be divided in three groups: State of the art postprocessing techniques from statistics (ensemble model output statistics (EMOS), member-by-member postprocessing, isotonic distributional regression), established machine learning methods (gradient-boosting extended EMOS, quantile regression forests) and neural network-based approaches (distributional regression network, Bernstein quantile network, histogram estimation network). The methods are systematically compared using six years of data from a high-resolution, convection-permitting ensemble prediction system that was run operationally at the German weather service, and hourly observations at 175 surface weather stations in Germany. While all postprocessing methods yield calibrated forecasts and are able to correct the systematic errors of the raw ensemble predictions, incorporating information from additional meteorological predictor variables beyond wind gusts leads to significant improvements in forecast skill. In particular, we propose a flexible framework of locally adaptive neural networks with different probabilistic forecast types as output, which not only significantly outperform all benchmark postprocessing methods but also learn physically consistent relations associated with the diurnal cycle, especially the evening transition of the planetary boundary layer.

stat.ML↗

arXiv2021

From Photometric Redshifts to Improved Weather Forecasts: machine learning and proper scoring rules as a basis for interdisciplinary work

The amount, size, and complexity of astronomical data-sets and databases are growing rapidly in the last decades, due to new technologies and dedicated survey telescopes. Besides dealing with poly-structured and complex data, sparse data has become a field of growing scientific interest. A specific field of Astroinformatics research is the estimation of redshifts of extra-galactic sources by using sparse photometric observations. Many techniques have been developed to produce those estimates with increasing precision. In recent years, models have been favored which instead of providing a point estimate only, are able to generate probabilistic density functions (PDFs) in order to characterize and quantify the uncertainties of their estimates. Crucial to the development of those models is a proper, mathematically principled way to evaluate and characterize their performances, based on scoring functions as well as on tools for assessing calibration. Still, in literature inappropriate methods are being used to express the quality of the estimates that are often not sufficient and can potentially generate misleading interpretations. In this work we summarize how to correctly evaluate errors and forecast quality when dealing with PDFs. We describe the use of the log-likelihood, the continuous ranked probability score (CRPS) and the probability integral transform (PIT) to characterize the calibration as well as the sharpness of predicted PDFs. We present what we achieved when using proper scoring rules to train deep neural networks as well as to evaluate the model estimates and how this work led from well calibrated redshift estimates to improvements in probabilistic weather forecasting. The presented work is an example of interdisciplinarity in data-science and illustrates how methods can help to bridge gaps between different fields of application.

astro-ph.IM↗

arXiv2021

Post-processing numerical weather prediction ensembles for probabilistic solar irradiance forecasting

In order to enable the transition towards renewable energy sources, probabilistic energy forecasting is of critical importance for incorporating volatile power sources such as solar energy into the electrical grid. Solar energy forecasting methods often aim to provide probabilistic predictions of solar irradiance. In particular, many hybrid approaches combine physical information from numerical weather prediction models with statistical methods. Even though the physical models can provide useful information at intra-day and day-ahead forecast horizons, ensemble weather forecasts from multiple model runs are often not calibrated and show systematic biases. We propose a post-processing model for ensemble weather predictions of solar irradiance at temporal resolutions between 30 minutes and 6 hours. The proposed models provide probabilistic forecasts in the form of a censored logistic probability distribution for lead times up to 5 days and are evaluated in two case studies covering distinct physical models, geographical regions, temporal resolutions, and types of solar irradiance. We find that post-processing consistently and significantly improves the forecast performance of the ensemble predictions for lead times up to at least 48 hours and is well able to correct the systematic lack of calibration.

stat.AP↗

arXiv2020

Predictive Inference Based on Markov Chain Monte Carlo Output

In Bayesian inference, predictive distributions are typically in the form of samples generated via Markov chain Monte Carlo (MCMC) or related algorithms. In this paper, we conduct a systematic analysis of how to make and evaluate probabilistic forecasts from such simulation output. Based on proper scoring rules, we develop a notion of consistency that allows to assess the adequacy of methods for estimating the stationary distribution underlying the simulation output. We then provide asymptotic results that account for the salient features of Bayesian posterior simulators, and derive conditions under which choices from the literature satisfy our notion of consistency. Importantly, these conditions depend on the scoring rule being used, such that the choices of approximation method and scoring rule are intertwined. While the logarithmic rule requires fairly stringent conditions, the continuous ranked probability score (CRPS) yields consistent approximations under minimal assumptions. These results are illustrated in a simulation study and an economic data example. Overall, mixture-of-parameters approximations which exploit the parametric structure of Bayesian models perform particularly well. Under the CRPS, the empirical distribution function is a simple and appealing alternative option.

stat.ME↗

arXiv2020

Statistical Postprocessing for Weather Forecasts -- Review, Challenges and Avenues in a Big Data World

Statistical postprocessing techniques are nowadays key components of the forecasting suites in many National Meteorological Services (NMS), with for most of them, the objective of correcting the impact of different types of errors on the forecasts. The final aim is to provide optimal, automated, seamless forecasts for end users. Many techniques are now flourishing in the statistical, meteorological, climatological, hydrological, and engineering communities. The methods range in complexity from simple bias corrections to very sophisticated distribution-adjusting techniques that incorporate correlations among the prognostic variables. The paper is an attempt to summarize the main activities going on this area from theoretical developments to operational applications, with a focus on the current challenges and potential avenues in the field. Among these challenges is the shift in NMS towards running ensemble Numerical Weather Prediction (NWP) systems at the kilometer scale that produce very large datasets and require high-density high-quality observations; the necessity to preserve space time correlation of high-dimensional corrected fields; the need to reduce the impact of model changes affecting the parameters of the corrections; the necessity for techniques to merge different types of forecasts and ensembles with different behaviors; and finally the ability to transfer research on statistical postprocessing to operations. Potential new avenues will also be discussed.

physics.ao-ph↗

arXiv2020

Machine learning for total cloud cover prediction

Accurate and reliable forecasting of total cloud cover (TCC) is vital for many areas such as astronomy, energy demand and production, or agriculture. Most meteorological centres issue ensemble forecasts of TCC, however, these forecasts are often uncalibrated and exhibit worse forecast skill than ensemble forecasts of other weather variables. Hence, some form of post-processing is strongly required to improve predictive performance. As TCC observations are usually reported on a discrete scale taking just nine different values called oktas, statistical calibration of TCC ensemble forecasts can be considered a classification problem with outputs given by the probabilities of the oktas. This is a classical area where machine learning methods are applied. We investigate the performance of post-processing using multilayer perceptron (MLP) neural networks, gradient boosting machines (GBM) and random forest (RF) methods. Based on the European Centre for Medium-Range Weather Forecasts global TCC ensemble forecasts for 2002-2014 we compare these approaches with the proportional odds logistic regression (POLR) and multiclass logistic regression (MLR) models, as well as the raw TCC ensemble forecasts. We further assess whether improvements in forecast skill can be obtained by incorporating ensemble forecasts of precipitation as additional predictor. Compared to the raw ensemble, all calibration methods result in a significant improvement in forecast skill. RF models provide the smallest increase in predictive performance, while MLP, POLR and GBM approaches perform best. The use of precipitation forecast data leads to further improvements in forecast skill and except for very short lead times the extended MLP model shows the best overall performance.

stat.ML↗

arXiv2018

Desublimation Frosting on Nanoengineered Surfaces

Ice nucleation from vapor presents a variety of challenges across a wide range of industries and applications including refrigeration, transportation, and energy generation. However, a rational comprehensive approach to fabricating intrinsically icephobic surfaces for frost formation-both from water condensation (followed by freezing) and in particular from desublimation (direct growth of ice crystals from vapor)-remains elusive. Here, guided by nucleation physics, we investigate the effect of material composition and surface texturing (atomically smooth to nanorough) on the nucleation and growth mechanism of frost for a range of conditions within the sublimation domain (0 C to -55C; partial water vapor pressures 6 to 0.02 mbar). Surprisingly, we observe that on silicon at very cold temperatures-below the homogeneous ice solidification nucleation limit (< -46 C)-desublimation does not become the favorable pathway to frosting. Furthermore we show that surface nanoroughness makes frost formation on silicon more probable. We experimentally demonstrate at temperatures between -48 C and -55 C that nanotexture with radii of curvature within one order of magnitude of the critical radius of nucleation favors frost growth, facilitated by capillary condensation, consistent with Kelvin's Equation. Our findings show that such nanoscale surface morphology imposed by design to impart desired functionalities-such as superhydrophobicity-or from defects, can be highly detrimental for frost icephobicity at low temperatures and water vapor partial pressures (< 0.05 mbar). Our work contributes to the fundamental understanding of heterogeneous phase transitions well within the equilibrium sublimation domain and has implications for applications such as travel, power generation, and refrigeration.

physics.app-ph↗

arXiv2018

Evaluating probabilistic forecasts with scoringRules

Probabilistic forecasts in the form of probability distributions over future events have become popular in several fields including meteorology, hydrology, economics, and demography. In typical applications, many alternative statistical models and data sources can be used to produce probabilistic forecasts. Hence, evaluating and selecting among competing methods is an important task. The scoringRules package for R provides functionality for comparative evaluation of probabilistic models based on proper scoring rules, covering a wide range of situations in applied work. This paper discusses implementation and usage details, presents case studies from meteorology and economics, and points to the relevant background literature.

stat.CO↗

arXiv2018

Neural networks for post-processing ensemble weather forecasts

Ensemble weather predictions require statistical post-processing of systematic errors to obtain reliable and accurate probabilistic forecasts. Traditionally, this is accomplished with distributional regression models in which the parameters of a predictive distribution are estimated from a training period. We propose a flexible alternative based on neural networks that can incorporate nonlinear relationships between arbitrary predictor variables and forecast distribution parameters that are automatically learned in a data-driven way rather than requiring pre-specified link functions. In a case study of 2-meter temperature forecasts at surface stations in Germany, the neural network approach significantly outperforms benchmark post-processing methods while being computationally more affordable. Key components to this improvement are the use of auxiliary predictor variables and station-specific information with the help of embeddings. Furthermore, the trained neural network can be used to gain insight into the importance of meteorological variables thereby challenging the notion of neural networks as uninterpretable black boxes. Our approach can easily be extended to other statistical post-processing and forecasting problems. We anticipate that recent advances in deep learning combined with the ever-increasing amounts of model and observation data will transform the post-processing of numerical weather forecasts in the coming decade.

stat.ML↗

arXiv2017

Combining predictive distributions for statistical post-processing of ensemble forecasts

Statistical post-processing techniques are now widely used to correct systematic biases and errors in calibration of ensemble forecasts obtained from multiple runs of numerical weather prediction models. A standard approach is the ensemble model output statistics (EMOS) method, a distributional regression approach where the forecast distribution is given by a single parametric law with parameters depending on the ensemble members. Choosing an appropriate parametric family for the weather variable of interest is a critical, however, often non-trivial task, and has been the focus of much recent research. In this article, we assess the merits of combining predictive distributions from multiple EMOS models based on different parametric families. In four case studies with wind speed and precipitation forecasts from two ensemble prediction systems, we study whether state of the art forecast combination methods are able to improve forecast skill.

stat.ME↗

arXiv2015

Forecaster's Dilemma: Extreme Events and Forecast Evaluation

In public discussions of the quality of forecasts, attention typically focuses on the predictive performance in cases of extreme events. However, the restriction of conventional forecast evaluation methods to subsets of extreme observations has unexpected and undesired effects, and is bound to discredit skillful forecasts when the signal-to-noise ratio in the data generating process is low. Conditioning on outcomes is incompatible with the theoretical assumptions of established forecast evaluation methods, thereby confronting forecasters with what we refer to as the forecaster's dilemma. For probabilistic forecasts, proper weighted scoring rules have been proposed as decision theoretically justifiable alternatives for forecast evaluation with an emphasis on extreme events. Using theoretical arguments, simulation experiments, and a real data study on probabilistic forecasts of U.S. inflation and gross domestic product growth, we illustrate and discuss the forecaster's dilemma along with potential remedies.

stat.ME↗

arXiv2015

Mixture EMOS model for calibrating ensemble forecasts of wind speed

Ensemble model output statistics (EMOS) is a statistical tool for post-processing forecast ensembles of weather variables obtained from multiple runs of numerical weather prediction models in order to produce calibrated predictive probability density functions (PDFs). The EMOS predictive PDF is given by a parametric distribution with parameters depending on the ensemble forecasts. We propose an EMOS model for calibrating wind speed forecasts based on weighted mixtures of truncated normal (TN) and log-normal (LN) distributions where model parameters and component weights are estimated by optimizing the values of proper scoring rules over a rolling training period. The new model is tested on wind speed forecasts of the 50 member European Centre for Medium-Range Weather Forecasts ensemble, the 11 member Aire Limitée Adaptation dynamique Développement International-Hungary Ensemble Prediction System ensemble of the Hungarian Meteorological Service and the eight-member University of Washington mesoscale ensemble, and its predictive performance is compared to that of various benchmark EMOS models based on single parametric families and combinations thereof. The results indicate improved calibration of probabilistic and accuracy of point forecasts in comparison with the raw ensemble and climatological forecasts. The mixture EMOS model significantly outperforms the TN and LN EMOS methods, moreover, it provides better calibrated forecasts than the TN-LN combination model and offers an increased flexibility while avoiding covariate selection problems.

stat.AP↗

arXiv2015

Similarity-based semi-local estimation of EMOS models

Weather forecasts are typically given in the form of forecast ensembles obtained from multiple runs of numerical weather prediction models with varying initial conditions and physics parameterizations. Such ensemble predictions tend to be biased and underdispersive and thus require statistical postprocessing. In the ensemble model output statistics (EMOS) approach, a probabilistic forecast is given by a single parametric distribution with parameters depending on the ensemble members. This article proposes two semi-local methods for estimating the EMOS coefficients where the training data for a specific observation station are augmented with corresponding forecast cases from stations with similar characteristics. Similarities between stations are determined using either distance functions or clustering based on various features of the climatology, forecast errors, ensemble predictions and locations of the observation stations. In a case study on wind speed over Europe with forecasts from the Grand Limited Area Model Ensemble Prediction System, the proposed similarity-based semi-local models show significant improvement in predictive performance compared to standard regional and local estimation methods. They further allow for estimating complex models without numerical stability issues and are computationally more efficient than local parameter estimation.

stat.AP↗