William Roden | Searcharxiv

arXiv2023

A Controlled Experiment on the Impact of Intrusion Detection False Alarm Rate on Analyst Performance

Organizations use intrusion detection systems (IDSes) to identify harmful activity among millions of computer network events. Cybersecurity analysts review IDS alarms to verify whether malicious activity occurred and to take remedial action. However, IDS systems exhibit high false alarm rates. This study examines the impact of IDS false alarm rate on human analyst sensitivity (probability of detection), precision (positive predictive value), and time on task when evaluating IDS alarms. A controlled experiment was conducted with participants divided into two treatment groups, 50% IDS false alarm rate and 86% false alarm rate, who classified whether simulated IDS alarms were true or false alarms. Results show statistically significant differences in precision and time on task. The median values for the 86% false alarm rate group were 47% lower precision and 40% slower time on task than the 50% false alarm rate group. No significant difference in analyst sensitivity was observed.

cs.CR↗

arXiv2020

Cry Wolf: Toward an Experimentation Platform and Dataset for Human Factors in Cyber Security Analysis

Computer network defense is a partnership between automated systems and human cyber security analysts. The system behaviors, for example raising a high proportion of false alarms, likely impact cyber analyst performance. Experimentation in the analyst-system domain is challenging due to lack of access to security experts, the usability of attack datasets, and the training required to use security analysis tools. This paper describes Cry Wolf, an open source web application for user studies of cyber security analysis tasks. This paper also provides an open-access dataset of 73 true and false Intrusion Detection System (IDS) alarms derived from real-world examples of "impossible travel" scenarios. Cry Wolf and the impossible travel dataset were used in an experiment on the impact of IDS false alarm rate on analysts' abilities to correctly classify IDS alerts as true or false alarms. Results from that experiment are used to evaluate the quality of the dataset using difficulty and discrimination index measures drawn from classical test theory. Many alerts in the dataset provide good discrimination for participants' overall task performance.

cs.CR↗