Biblio

Gartner, a large research and advisory company, anticipates that by 2024 80% of security operation centers (SOCs) will use machine learning (ML) based solutions to enhance their operations.11https://www.ciodive.com/news/how-data-science-tools-can-lighten-the-load-for-cybersecurity-teams/572209/ In light of such widespread adoption, it is vital for the research community to identify and address usability concerns. This work presents the results of the first in situ usability assessment of ML-based tools. With the support of the US Navy, we leveraged the national cyber range-a large, air-gapped cyber testbed equipped with state-of-the-art network and user emulation capabilities-to study six US Naval SOC analysts' usage of two tools. Our analysis identified several serious usability issues, including multiple violations of established usability heuristics for user interface design. We also discovered that analysts lacked a clear mental model of how these tools generate scores, resulting in mistrust \$a\$ and/or misuse of the tools themselves. Surprisingly, we found no correlation between analysts' level of education or years of experience and their performance with either tool, suggesting that other factors such as prior background knowledge or personality play a significant role in ML-based tool usage. Our findings demonstrate that ML-based security tool vendors must put a renewed focus on working with analysts, both experienced and inexperienced, to ensure that their systems are usable and useful in real-world security operations settings.

Situational awareness during sophisticated cyber attacks on the power grid is critical for the system operator to perform suitable attack response and recovery functions to ensure grid reliability. The overall theme of this paper is to identify existing practical issues and challenges that utilities face while monitoring substations, and to suggest potential approaches to enhance the situational awareness for the grid operators. In this paper, we provide a broad discussion about the various gaps that exist in the utility industry today in monitoring substations, and how those gaps could be addressed by identifying the various data sources and monitoring tools to improve situational awareness. The paper also briefly describes the advantages of contextualizing and correlating substation monitoring alerts using expert systems at the control center to obtain a holistic systems-level view of potentially malicious cyber activity at the substations before they cause impacts to grid operation.