arXiv subjects

Alex Kantchelian

Publications and source records attributed to Alex Kantchelian.

5 recordsLinked to original sources

arXiv2026

Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations

This paper presents AuditBench, a new benchmark dataset for evaluating the capabilities of LLMs at investigating security-related system audit logs. We design and use this benchmark to explore the performance of LLMs on four log-investigation tasks that incident response teams commonly perform, ranging from triaging alerts generated by detectors to identifying persistence mechanisms on compromised systems. AuditBench consists of system audit logs collected from Linux and Windows machines, and spans over 50 different security investigation scenarios, including both malicious and benign activity. Using our benchmark, we evaluate and analyze the performance of five frontier LLMs at analyzing audit logs for attack investigations. Our analysis illuminates how LLM performance and error profiles vary according to different design choices, such as differences in model size, data representation, prompt construction, and specific investigation tasks. Additionally, we characterize the quality of the explanations produced by LLMs and the types of errors that models make across our benchmark. Collectively, our work provides a foundation for assessing the capabilities of LLMs for investigating security logs, novel insights for practitioners using LLMs in security operations, and important directions for future research.

cs.CR↗

arXiv2024

Facade: High-Precision Insider Threat Detection Using Deep Contextual Anomaly Detection

Insiders with privileged access have the power to cause great harm to their organization. Even a single insider threat incident can be catastrophic, resulting in both financial losses and reputation damage. These threats are some of the most difficult to detect, as attack activity is interspersed in large volumes of legitimate activity. Although it is a serious threat, the literature is sparse aside from a few studies with various limitations, preventing their practical deployment in large-scale organizations. We present Facade: fast and accurate contextual anomaly detection, a high-precision, deep-learning system that has served as the last line of defense against insider threats at Google since 2018. Facade is an innovative self-supervised system that detects suspicious events by considering the context surrounding each event, including relevant facts about the user and resource involved. It is built around a new multi-modal model that is trained on corporate document access, SQL table access, and HTTP/RPC request logs. To overcome the scarcity of labeled incident data, Facade employs a novel contrastive learning strategy trained exclusively on benign activity. Facade detects insider attackers with an extremely low false-positive rate, lower than 0.01%. For single rogue events, such as the illegitimate access to a sensitive document, the false-positive rate is as low as 0.0003%. To the best of our knowledge, Facade is the only proposed insider risk anomaly detection system with a false positive rate low enough for use in large corporate environments.

cs.CR↗

arXiv2024

Fine Grained Insider Risk Detection

We present a method to detect departures from business-justified workflows among support agents. Our goal is to assist auditors in identifying agent actions that cannot be explained by the activity within their surrounding context, where normal activity patterns are established from historical data. We apply our method to help audit millions of actions of over three thousand support agents. We collect logs from the tools used by support agents and construct a bipartite graph of Actions and Entities representing all the actions of the agents, as well as background information about entities. From this graph, we sample subgraphs rooted on security-significant actions taken by the agents. Each subgraph captures the relevant context of the root action in terms of other actions, entities and their relationships. We then prioritize the rooted-subgraphs for auditor review using feed-forward and graph neural networks, as well as nearest neighbors techniques. To alleviate the issue of scarce labeling data, we use contrastive learning and domain-specific data augmentations. Expert auditors label the top ranked subgraphs as ``worth auditing" or ``not worth auditing" based on the company's business policies. This system finds subgraphs that are worth auditing with high enough precision to be used in production.

cs.CR↗

arXiv2016

Evasion and Hardening of Tree Ensemble Classifiers

Classifier evasion consists in finding for a given instance $x$ the nearest instance $x'$ such that the classifier predictions of $x$ and $x'$ are different. We present two novel algorithms for systematically computing evasions for tree ensembles such as boosted trees and random forests. Our first algorithm uses a Mixed Integer Linear Program solver and finds the optimal evading instance under an expressive set of constraints. Our second algorithm trades off optimality for speed by using symbolic prediction, a novel algorithm for fast finite differences on tree ensembles. On a digit recognition task, we demonstrate that both gradient boosted trees and random forests are extremely susceptible to evasions. Finally, we harden a boosted tree model without loss of predictive accuracy by augmenting the training set of each boosting round with evading instances, a technique we call adversarial boosting.

cs.LG↗

arXiv2016

Reviewer Integration and Performance Measurement for Malware Detection

We present and evaluate a large-scale malware detection system integrating machine learning with expert reviewers, treating reviewers as a limited labeling resource. We demonstrate that even in small numbers, reviewers can vastly improve the system's ability to keep pace with evolving threats. We conduct our evaluation on a sample of VirusTotal submissions spanning 2.5 years and containing 1.1 million binaries with 778GB of raw feature data. Without reviewer assistance, we achieve 72% detection at a 0.5% false positive rate, performing comparable to the best vendors on VirusTotal. Given a budget of 80 accurate reviews daily, we improve detection to 89% and are able to detect 42% of malicious binaries undetected upon initial submission to VirusTotal. Additionally, we identify a previously unnoticed temporal inconsistency in the labeling of training datasets. We compare the impact of training labels obtained at the same time training data is first seen with training labels obtained months later. We find that using training labels obtained well after samples appear, and thus unavailable in practice for current training data, inflates measured detection by almost 20 percentage points. We release our cluster-based implementation, as well as a list of all hashes in our evaluation and 3% of our entire dataset.

cs.CR↗