Data-Driven Model-Based Decision-Making - January 2017

Public Audience
Purpose: To highlight project progress. Information is generally at a higher level which is accessible to the interested public. All information contained in the report (regions 1-3) is a Government Deliverable/CDRL.

PI(s): William Sanders, Masooda Bashir, David Nicol, and Aad Van Moorsel*

Researchers: Ken Keefe, Mohamad Noureddine, Charles Morriset* and Rob Cain* (*Newcastle Univ., UK)

HARD PROBLEM(S) ADDRESSED
This refers to Hard Problems, released November 2012.

• Predictive Security Metrics - System security analysis requires a holistic approach that considers the behavior of non-human subsystem, bad actors or adversaries, and expected human participants such as users and system administrators. We are developing the HITOP modeling formalism to formally describe the behavior of human participants and how their decisions affect overall system performance and security. With this modeling methodology and the tool support we are developing, we will produce quantitative security metrics for cyber-human systems.
• Human Behavior - Modeling and evaluating human behavior is challenging, but it is an imperative component in security analysis. Stochastic modeling serves as a good approximation of human behavior, but we intend to do more with the HITOP method, which considers a task based process modeling language that evaluates a human's opportunity, willingness, and capability to perform individual tasks in their daily behavior. Partnered with an effective data collection strategy to validate model parameters, we are working to provide a sound model of human behavior.

PUBLICATIONS
Papers published in this quarter as a result of this research. Include title, author(s), venue published/presented, and a short description or abstract. Identify which hard problem(s) the publication addressed. Papers that have not yet been published should be reported in region 2 below.

No new publications this quarter.

ACCOMPLISHMENT HIGHLIGHTS

We continue with the problem of optimizing data collection for parameterized probabilistic models. Collecting data for parameters is costly and is often constrained therefore it is important to identify how data collection should be best spread across parameter data sources for accurate model checking. The optimization technique has to be model independent. Our objective is to identify the most optimum data collection strategy for a given model within an allowed budget.

We have developed and tested an extension for the probabilistic model checker PRISM we call Data Collection Optimization (DCO). DCO analyses various data collection strategies for a probabilistic model encoded in PRISM and finds the best strategy by using the optimization algorithm defined in [1]. The algorithm generates a set of strategies, which satisfy a given budget, each stating a number of samples to be collected from each of the model parameter data sources. Monte Carlo simulation is performed on the model with each of these strategies. The optimum strategy is identified using the variance method.

We have also developed and tested a PRISM extension, which analyzes the sensitivity of a model's input parameters. Sensitivity analysis of a probabilistic model's inputs is performed using the differential method. Using this extension a user can manually create data collection strategies which can be optimized using our DCO extension.

Internal testing of the HITOP prototype is proceeding on schedule for a public release. HITOP seeks to model human behavior in the terms of a graphical task/process formalism, which can be nested and connected to other useful models, like attacker models and performance models. We continue to refine a test model that can bring together the data collection strategies described earlier and the human behavior models that HITOP enables.

We started a new effort on a different approach for human behavior modeling. The aim is to create an easy to use, powerful, and flexible modeling formalism that would allow a security modeler to model multiple defenders and users in a system as well as the attacker. An engineer using the modeling formalism could make more informed design decisions to improve the security of cyber systems. We started exploring the existing literature, developing the theory behind the new modeling formalism, and laying the groundwork for a case study which will exercise both established and novel modeling formalisms.