arXiv subjects

Ronal Singh

Publications and source records attributed to Ronal Singh.

14 recordsLinked to original sources

arXiv2026

From Frontier to Shadow AI: A Simmering Threat to Assurance and Security in Critical Infrastructure

Frontier AI systems, including large language models and emerging agentic AI tools, offer significant operational benefits but present unique challenges to critical infrastructure (CI) environments due to their non-deterministic and emergent properties. While formal adoption is inherently cautious and tightly controlled due to strict regulatory oversight, widespread accessibility has catalysed shadow AI: the unsanctioned use of frontier AI outside established organisational controls. In CI settings, shadow AI bypasses established assurance and oversight mechanisms, amplifying risks to data protection, decision reliability, and regulatory compliance, with potential consequences for essential service delivery. We present the first empirical study of shadow AI in CI environments, characterising it as a systemic socio-technical condition of assurance erosion. Drawing on semi-structured interviews with senior executives and functional leaders across 27 Australian CI organisations (Communications, Energy, and Water and Sewerage sectors), we analyse how shadow AI manifests in practice, how it interacts with existing technical and governance controls, and the resulting security, assurance, and compliance risks. We develop an empirically derived threat model identifying three primary mechanisms of security degradation: (i) boundary bypass, where data flows circumvent established perimeters; (ii) unassessed capability expansion, where embedded AI features introduce latent risks; and (iii) loss of observability via governance circumvention, undermining forensic auditability and least-privilege enforcement. Our findings demonstrate that shadow AI introduces unmanaged risks that fundamentally challenge existing security and compliance frameworks, necessitating tailored, pathway-aligned governance and control strategies.

cs.CR↗

arXiv2025

LLMs in the SOC: An Empirical Study of Human-AI Collaboration in Security Operations Centres

The integration of Large Language Models (LLMs) into Security Operations Centres (SOCs) presents a transformative, yet still evolving, opportunity to reduce analyst workload through human-AI collaboration. However, their real-world application in SOCs remains underexplored. To address this gap, we present a longitudinal study of 3,090 analyst queries from 45 SOC analysts over 10 months. Our analysis reveals that analysts use LLMs as on-demand aids for sensemaking and context-building, rather than for making high-stakes determinations, preserving analyst decision authority. The majority of queries are related to interpreting low-level telemetry (e.g., commands) and refining technical communication through short (1-3 turn) interactions. Notably, 93% of queries align with established cybersecurity competencies (NICE Framework), underscoring the relevance of LLM use for SOC-related tasks. Despite variations in tasks and engagement, usage trends indicate a shift from occasional exploration to routine integration, with growing adoption and sustained use among a subset of analysts. We find that LLMs function as flexible, on-demand cognitive aids that augment, rather than replace, SOC expertise. Our study provides actionable guidance for designing context-aware, human-centred AI assistance in security operations, highlighting the need for further in-the-wild research on real-world analyst-LLM collaboration, challenges, and impacts.

cs.CR↗

arXiv2025

ContextBuddy: AI-Enhanced Contextual Insights for Security Alert Investigation (Applied to Intrusion Detection)

Modern Security Operations Centres (SOCs) integrate diverse tools, such as SIEM, IDS, and XDR systems, offering rich contextual data, including alert enrichments, flow features, and similar case histories. Yet, analysts must still manually determine which of these contextual cues are most relevant when validating specific alerts. We introduce ContextBuddy, an AI assistant that learns from analysts' prior investigations to help them identify the most relevant context for new alerts. Rather than providing enrichments, ContextBuddy models how analysts have previously selected context and suggests tailored cues based on the characteristics of each alert. We formulate context selection as a sequential decision-making problem and apply imitation learning (IL) to capture analysts' strategies, evaluating multiple IL approaches. Through staged evaluation, we validate ContextBuddy using two intrusion detection datasets (HIKARI-2021, UNSW-NB15). In simulation-based experiments, ContextBuddy helped simulated reinforcement learning analysts improve classification accuracy (p < 0.001) (increasing F1 by 2.5% for HIKARI and 9% for UNSW), reducing false negatives (1.5% for HIKARI and 10% for UNSW), and keeping false positives below 1%. Decision confidence among agents also improved by 2-3% (p < 0.001). In a within-subject user study (N=13; power = 0.8), non-experts using ContextBuddy improved classification accuracy by 21.1% (p = 0.008) and reduced alert validation time by 24% (p = 0.01). These results demonstrate that by learning context-selection patterns from analysts, ContextBuddy can yield notable improvements in investigation effectiveness and efficiency.

cs.CR↗

arXiv2025

Bridging Expertise Gaps: The Role of LLMs in Human-AI Collaboration for Cybersecurity

This study investigates whether large language models (LLMs) can function as intelligent collaborators to bridge expertise gaps in cybersecurity decision-making. We examine two representative tasks-phishing email detection and intrusion detection-that differ in data modality, cognitive complexity, and user familiarity. Through a controlled mixed-methods user study, n = 58 (phishing, n = 34; intrusion, n = 24), we find that human-AI collaboration improves task performance,reducing false positives in phishing detection and false negatives in intrusion detection. A learning effect is also observed when participants transition from collaboration to independent work, suggesting that LLMs can support long-term skill development. Our qualitative analysis shows that interaction dynamics-such as LLM definitiveness, explanation style, and tone-influence user trust, prompting strategies, and decision revision. Users engaged in more analytic questioning and showed greater reliance on LLM feedback in high-complexity settings. These results provide design guidance for building interpretable, adaptive, and trustworthy human-AI teaming systems, and demonstrate that LLMs can meaningfully support non-experts in reasoning through complex cybersecurity problems.

cs.CR↗

arXiv2024

An Actionability Assessment Tool for Explainable AI

In this paper, we introduce and evaluate a tool for researchers and practitioners to assess the actionability of information provided to users to support algorithmic recourse. While there are clear benefits of recourse from the user's perspective, the notion of actionability in explainable AI research remains vague, and claims of `actionable' explainability techniques are based on the researchers' intuition. Inspired by definitions and instruments for assessing actionability in other domains, we construct a seven-question tool and evaluate its effectiveness through two user studies. We show that the tool discriminates actionability across explanation types and that the distinctions align with human judgements. We show the impact of context on actionability assessments, suggesting that domain-specific tool adaptations may foster more human-centred algorithmic systems. This is a significant step forward for research and practices into actionable explainability and algorithmic recourse, providing the first clear human-centred definition and tool for assessing actionability in explainable AI.

cs.HC↗

arXiv2024

Visual Evaluative AI: A Hypothesis-Driven Tool with Concept-Based Explanations and Weight of Evidence

This paper presents Visual Evaluative AI, a decision aid that provides positive and negative evidence from image data for a given hypothesis. This tool finds high-level human concepts in an image and generates the Weight of Evidence (WoE) for each hypothesis in the decision-making process. We apply and evaluate this tool in the skin cancer domain by building a web-based application that allows users to upload a dermatoscopic image, select a hypothesis and analyse their decisions by evaluating the provided evidence. Further, we demonstrate the effectiveness of Visual Evaluative AI on different concept-based explanation approaches.

cs.CV↗

arXiv2024

Explainable agency: human preferences for simple or complex explanations

Research in cognitive psychology has established that whether people prefer simpler explanations to complex ones is context dependent, but the question of `simple vs. complex' becomes critical when an artificial agent seeks to explain its decisions or predictions to humans. We present a model for abstracting causal reasoning chains for the purpose of explanation. This model uses a set of rules to progressively abstract different types of causal information in causal proof traces. We perform online studies using 123 Amazon MTurk participants and with five industry experts over two domains: maritime patrol and weather prediction. We found participants' satisfaction with generated explanations was based on the consistency of relationships among the causes (coherence) that explain an event; and that the important question is not whether people prefer simple or complex explanations, but what types of causal information are relevant to individuals in specific contexts.

cs.HC↗

arXiv2024

From Evidence to Decision: Exploring Evaluative AI

This paper presents a hypothesis-driven approach to improve AI-supported decision-making that is based on the Evaluative AI paradigm - a conceptual framework that proposes providing users with evidence for or against a given hypothesis. We propose an implementation of Evaluative AI by extending the Weight of Evidence framework, leading to hypothesis-driven models that support both tabular and image data. We demonstrate the application of the new decision-support approach in two domains: housing price prediction and skin cancer diagnosis. The findings show promising results in improving human decisions, as well as providing insights on the strengths and weaknesses of different decision-support approaches.

cs.AI↗

arXiv2023

Explaining Model Confidence Using Counterfactuals

Displaying confidence scores in human-AI interaction has been shown to help build trust between humans and AI systems. However, most existing research uses only the confidence score as a form of communication. As confidence scores are just another model output, users may want to understand why the algorithm is confident to determine whether to accept the confidence score. In this paper, we show that counterfactual explanations of confidence scores help study participants to better understand and better trust a machine learning model's prediction. We present two methods for understanding model confidence using counterfactual explanation: (1) based on counterfactual examples; and (2) based on visualisation of the counterfactual space. Both increase understanding and trust for study participants over a baseline of no explanation, but qualitative results show that they are used quite differently, leading to recommendations of when to use each one and directions of designing better explanations.

cs.AI↗

arXiv2022

Improving Model Understanding and Trust with Counterfactual Explanations of Model Confidence

In this paper, we show that counterfactual explanations of confidence scores help users better understand and better trust an AI model's prediction in human-subject studies. Showing confidence scores in human-agent interaction systems can help build trust between humans and AI systems. However, most existing research only used the confidence score as a form of communication, and we still lack ways to explain why the algorithm is confident. This paper also presents two methods for understanding model confidence using counterfactual explanation: (1) based on counterfactual examples; and (2) based on visualisation of the counterfactual space.

cs.LG↗

arXiv2021

Collaborative Human-Agent Planning for Resilience

Intelligent agents powered by AI planning assist people in complex scenarios, such as managing teams of semi-autonomous vehicles. However, AI planning models may be incomplete, leading to plans that do not adequately meet the stated objectives, especially in unpredicted situations. Humans, who are apt at identifying and adapting to unusual situations, may be able to assist planning agents in these situations by encoding their knowledge into a planner at run-time. We investigate whether people can collaborate with agents by providing their knowledge to an agent using linear temporal logic (LTL) at run-time without changing the agent's domain model. We presented 24 participants with baseline plans for situations in which a planner had limitations, and asked the participants for workarounds for these limitations. We encoded these workarounds as LTL constraints. Results show that participants' constraints improved the expected return of the plans by 10% ($p < 0.05$) relative to baseline plans, demonstrating that human insight can be used in collaborative planning for resilience. However, participants used more declarative than control constraints over time, but declarative constraints produced plans less similar to the expectation of the participants, which could lead to potential trust issues.

cs.AI↗

arXiv2021

LEx: A Framework for Operationalising Layers of Machine Learning Explanations

Several social factors impact how people respond to AI explanations used to justify AI decisions affecting them personally. In this position paper, we define a framework called the \textit{layers of explanation} (LEx), a lens through which we can assess the appropriateness of different types of explanations. The framework uses the notions of \textit{sensitivity} (emotional responsiveness) of features and the level of \textit{stakes} (decision's consequence) in a domain to determine whether different types of explanations are \textit{appropriate} in a given context. We demonstrate how to use the framework to assess the appropriateness of different types of explanations in different domains.

cs.LG↗

arXiv2021

Directive Explanations for Actionable Explainability in Machine Learning Applications

This paper investigates the prospects of using directive explanations to assist people in achieving recourse of machine learning decisions. Directive explanations list which specific actions an individual needs to take to achieve their desired outcome. If a machine learning model makes a decision that is detrimental to an individual (e.g. denying a loan application), then it needs to both explain why it made that decision and also explain how the individual could obtain their desired outcome (if possible). At present, this is often done using counterfactual explanations, but such explanations generally do not tell individuals how to act. We assert that counterfactual explanations can be improved by explicitly providing people with actions they could use to achieve their desired goal. This paper makes two contributions. First, we present the results of an online study investigating people's perception of directive explanations. Second, we propose a conceptual model to generate such explanations. Our online study showed a significant preference for directive explanations ($p<0.001$). However, the participants' preferred explanation type was affected by multiple factors, such as individual preferences, social factors, and the feasibility of the directives. Our findings highlight the need for a human-centred and context-specific approach for creating directive explanations.

cs.LG↗

arXiv2018

Interaction Design for Explainable AI: Workshop Proceedings

As artificial intelligence (AI) systems become increasingly complex and ubiquitous, these systems will be responsible for making decisions that directly affect individuals and society as a whole. Such decisions will need to be justified due to ethical concerns as well as trust, but achieving this has become difficult due to the `black-box' nature many AI models have adopted. Explainable AI (XAI) can potentially address this problem by explaining its actions, decisions and behaviours of the system to users. However, much research in XAI is done in a vacuum using only the researchers' intuition of what constitutes a `good' explanation while ignoring the interaction and the human aspect. This workshop invites researchers in the HCI community and related fields to have a discourse about human-centred approaches to XAI rooted in interaction and to shed light and spark discussion on interaction design challenges in XAI.

cs.AI↗