arXiv subjects

Jeremy E. Block

Publications and source records attributed to Jeremy E. Block.

4 recordsLinked to original sources

arXiv2024

From Data Dump to Digestible Chunks: Automated Segmentation and Summarization of Provenance Logs for Communication

Communicating one's sensemaking during a complex analysis session to explain thought processes is hard, yet most intelligence occurs in collaborative settings. Team members require a deeper understanding of the work being completed by their peers and subordinates, but little research has fully articulated best practices for analytic provenance consumers. This work proposes an automatic summarization technique that separates an analysis session and summarizes interaction provenance as textual blurbs to allow for meta-analysis of work done. Focusing on the domain of intelligence analysis, we demonstrate our segmentation technique using five datasets, including both publicly available and classified interaction logs. We shared our demonstration with a notoriously inaccessible population of expert reviewers with experience as United States Department of Defense analysts. Our findings indicate that the proposed pipeline effectively generates cards that display key events from interaction logs, facilitating the sharing of analysis progress. Yet, we also hear that there is a need for more prominent justifications and pattern elicitation controls to communicate analysis summaries more effectively. The expert review highlights the potential of automated approaches in addressing the challenges of provenance information in complex domains. We'd like to emphasize the need for further research into provenance communication in other domains. A free copy of this paper and all supplemental materials are available at https://osf.io/j4bxt

cs.HC↗

arXiv2022

The Influence of Visual Provenance Representations on Strategies in a Collaborative Hand-off Data Analysis Scenario

Conducting data analysis tasks rarely occur in isolation. Especially in intelligence analysis scenarios where different experts contribute knowledge to a shared understanding, members must communicate how insights develop to establish common ground among collaborators. The use of provenance to communicate analytic sensemaking carries promise by describing the interactions and summarizing the steps taken to reach insights. Yet, no universal guidelines exist for communicating provenance in different settings. Our work focuses on the presentation of provenance information and the resulting conclusions reached and strategies used by new analysts. In an open-ended, 30-minute, textual exploration scenario, we qualitatively compare how adding different types of provenance information (specifically data coverage and interaction history) affects analysts' confidence in conclusions developed, propensity to repeat work, filtering of data, identification of relevant information, and typical investigation strategies. We see that data coverage (i.e., what was interacted with) provides provenance information without limiting individual investigation freedom. On the other hand, while interaction history (i.e., when something was interacted with) does not significantly encourage more mimicry, it does take more time to comfortably understand, as represented by less confident conclusions and less relevant information-gathering behaviors. Our results contribute empirical data towards understanding how provenance summarizations can influence analysis behaviors.

cs.HC↗

arXiv2020

Micro-entries: Encouraging Deeper Evaluation of Mental Models Over Time for Interactive Data Systems

Many interactive data systems combine visual representations of data with embedded algorithmic support for automation and data exploration. To effectively support transparent and explainable data systems, it is important for researchers and designers to know how users understand the system. We discuss the evaluation of users' mental models of system logic. Mental models are challenging to capture and analyze. While common evaluation methods aim to approximate the user's final mental model after a period of system usage, user understanding continuously evolves as users interact with a system over time. In this paper, we review many common mental model measurement techniques, discuss tradeoffs, and recommend methods for deeper, more meaningful evaluation of mental models when using interactive data analysis and visualization systems. We present guidelines for evaluating mental models over time that reveal the evolution of specific model updates and how they may map to the particular use of interface features and data queries. By asking users to describe what they know and how they know it, researchers can collect structured, time-ordered insight into a user's conceptualization process while also helping guide users to their own discoveries.

cs.HC↗

arXiv2020

A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning

Research in interpretable machine learning proposes different computational and human subject approaches to evaluate model saliency explanations. These approaches measure different qualities of explanations to achieve diverse goals in designing interpretable machine learning systems. In this paper, we propose a human attention benchmark for image and text domains using multi-layer human attention masks aggregated from multiple human annotators. We then present an evaluation study to evaluate model saliency explanations obtained using Grad-cam and LIME techniques. We demonstrate our benchmark's utility for quantitative evaluation of model explanations by comparing it with human subjective ratings and ground-truth single-layer segmentation masks evaluations. Our study results show that our threshold agnostic evaluation method with the human attention baseline is more effective than single-layer object segmentation masks to ground truth. Our experiments also reveal user biases in the subjective rating of model saliency explanations.

cs.HC↗