Pervasive Label Errors in Seismological Machine Learning Datasets

Abstract

The recent boom in artificial intelligence and machine learning has been powered by large datasets with accurate labels, combined with algorithmic advances and efficient computing. The quality of data can be a major factor in determining model performance. Here, we detail observations of commonly occurring errors in popular seismological machine learning datasets. We used an ensemble of available deep learning models PhaseNet and EQTransformer to evaluate the dataset labels and found four types of errors ranked from most prevalent to least prevalent: (1) unlabeled earthquakes; (2) noise samples that contain earthquakes; (3) inaccurately labeled arrival times, and (4) absent earthquake signals. We checked a total of 8.6 million examples from the following datasets: Iquique, ETHZ, PNW, TXED, STEAD, INSTANCE, AQ2009, and CEED. The average error rate across all datasets is 3.9 %, ranging from nearly zero to 8 % for individual datasets. These faulty data and labels are likely to degrade model training and performance. By flagging these errors, we aim to increase the quality of the data used to train machine learning models, especially for the measurement of arrival times, and thereby to improve the reliability of the models. We present a companion list of examples that contain problems, aiming to integrate them into training routines so that only the reliable data is used for training.