arXiv subjects

Mohammadreza Khanarmuei

Publications and source records attributed to Mohammadreza Khanarmuei.

4 recordsLinked to original sources

arXiv2024

The Jive Verification System and its Transformative Impact on Weather Forecasting Operations

Forecast verification is critical for continuous improvement in meteorological organizations. The Jive verification system was originally developed to assess the accuracy of public weather forecasts issued by the Australian Bureau of Meteorology. It started as a research project in 2015 and gradually evolved to be a Bureau operational verification system in 2022. The system includes daily verification dashboards for forecasters to visualize recent forecast performance and "Evidence Targeted Automation" dashboards for exploring the performance of competing forecast systems. Additionally, Jive includes a Jupyter Notebook server with the Jive Python library which supports research experiments, case studies, and the development of new verification metrics and tools. This paper describes the Jive verification system and how it helped bring verification to the forefront at the Bureau of Meteorology, leading to more accurate, streamlined forecasts. Jive has provided evidence to support forecast automation decisions and has helped to understand the evolving role of meteorologists in the forecast process. It has given operational meteorologists tools for evaluating forecast processes, including identifying when and how manual interventions lead to superior predictions. Work on Jive led to new verification science, including novel metrics that are decision-focused, including diagnostics for extreme conditions. Jive also provided the Bureau with an enterprise-wide data analysis environment and has prompted a clarification of forecast definitions. These collective impacts have resulted in more accurate forecasts, ultimately benefiting society, and building trust with forecast users. These positive outcomes highlight the importance of meteorological organizations investing in verification science and technology.

physics.ao-ph↗

arXiv2024

scores: A Python package for verifying and evaluating models and predictions with xarray

`scores` is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models. It supports labelled n-dimensional (multidimensional) data, which is used in many scientific fields and in machine learning. At present, `scores` primarily supports the geoscience communities; in particular, the meteorological, climatological and oceanographic communities. `scores` not only includes common scores (e.g., Mean Absolute Error), it also includes novel scores not commonly found elsewhere (e.g., FIxed Risk Multicategorical (FIRM) score, Flip-Flop Index), complex scores (e.g., threshold-weighted continuous ranked probability score), and statistical tests (such as the Diebold Mariano test). It also contains isotonic regression which is becoming an increasingly important tool in forecast verification and can be used to generate stable reliability diagrams. Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). At the time of writing, `scores` includes over 50 metrics, statistical techniques and data processing tools. All of the scores and statistical techniques in this package have undergone a thorough scientific and software review. Every score has a companion Jupyter Notebook tutorial that demonstrates its use in practice. `scores` supports `xarray` datatypes, allowing it to work with Earth system data in a range of formats including NetCDF4, HDF5, Zarr and GRIB among others. `scores` uses Dask for scaling and performance. Support for `pandas` is being introduced. The `scores` software repository can be found at https://github.com/nci/scores/

physics.ao-ph↗

arXiv2024

A User-Focused Approach to Evaluating Probabilistic and Categorical Forecasts

A user-focused verification approach for evaluating probability forecasts of binary outcomes (also known as probabilistic classifiers) is demonstrated that is (i) based on proper scoring rules, (ii) focuses on user decision thresholds, and (iii) provides actionable insights. It is argued that when categorical performance diagrams and the critical success index are used to evaluate overall predictive performance, rather than the discrimination ability of probabilistic forecasts, they may produce misleading results. Instead, Murphy diagrams are shown to provide better understanding of overall predictive performance as a function of user probabilistic decision threshold. It is illustrated how to select a proper scoring rule, based on the relative importance of different user decision thresholds, and how this choice impacts scores of overall predictive performance and supporting measures of discrimination and calibration. These approaches and ideas are demonstrated using several probabilistic thunderstorm forecast systems as well as synthetic forecast data. Furthermore, a fair method for comparing the performance of probabilistic and categorical forecasts is illustrated using the FIxed Risk Multicategorical (FIRM) score, which is a proper scoring rule directly connected to values on the Murphy diagram. While the methods are illustrated using thunderstorm forecasts, they are applicable for evaluating probabilistic forecasts for any situation with binary outcomes.

stat.AP↗

arXiv2020

Material and debris transport patterns in Moreton Bay, Australia: The influence of Lagrangian coherent structures

Coastal tidal estuaries are vital to the exchange of energy and material between inland waters and the open ocean. Debris originating from the land and ocean enter this environment and are transported by currents (river outflow and tide), wind, waves and density gradients. Understanding and predicting the source and fate of such debris has considerable environmental, economic and visual importance. We show that this issue can be addressed using the Lagrangian coherent structures (LCS) technique which is highly robust to hydrodynamic model uncertainties. Here we present a comprehensive study showing the utility of this approach to describe the fate of floating material in a coastal tidal embayment. An example is given from Moreton Bay, a semi-enclosed subtropical embayment with high morphologic, ecological and economic significance to Southeast Queensland, Australia. Transport barriers visualised by the LCS create pathways and barriers for material transport in the embayment. It was found that the wind field modified both the rate attraction and location of the transport barriers. One of the key outcomes is the demonstration of the significant role of islands in partitioning the transport of material and mixing within the embayment. The distribution of the debris sources along the shoreline are explained by the relative location of the LCS to the shoreline. Therefore, extraction of LCS can help to predict sources and fate of anthropogenic marine debris and thus, serve as a useful way for effective management of vulnerable regions and marine protected areas.

physics.ao-ph↗