Impact of Validation Strategy on Machine Learning Performance in EEG-Based Alcoholism Classification

Abstract

Electroencephalography provides a non-invasive and cost-effective approach for analyzing neural patterns associated with alcohol dependence. However, reported classification performance in EEG-based alcoholism studies varies considerably, often due to differences in validation strategies rather than intrinsic model capability. This study presents a validation-aware machine learning framework to assess the impact of evaluation methodology on classification performance. A balanced multi-channel EEG dataset of 300 trials (150 alcoholic, 150 control) was analyzed using a structured feature representation combining statistical descriptors and spectral band interactions. Five classifiers, including support vector machines (linear and radial basis function kernels), random forest, k-nearest neighbors, and AdaBoost, were evaluated under standard and nested cross-validation protocols. Results show that conventional validation with global hyperparameter tuning introduces optimistic bias. In particular, SVM with radial basis function kernel exhibited a performance decrease of approximately 5\% under nested cross-validation, indicating overestimation. Ensemble-based methods showed more stable generalization, with AdaBoost achieving the highest performance, reaching 78.3\% accuracy ($\pm$4.25), an AUC of 0.868, and balanced sensitivity (78.67\%) and specificity (81.33\%). These findings highlight that validation strategy is a primary determinant of perceived model performance. Statistical analysis using McNemar's test further shows that most performance differences between models are not statistically significant, emphasizing careful interpretation of classification results. The proposed framework provides a reproducible and robust basis for evaluating machine learning models in biomedical signal analysis.