Multi-model Testbed for the Simulation-based Evaluation of Resilience (April '20)
PI(s), Co-PI(s), Researchers:
- Peter Volgyesi (PI)
- Himanshu Neema (Co-PI)
HARD PROBLEM(S) ADDRESSED
This refers to Hard Problems, released in November 2012.
- Security Metrics Driven Evaluation, Design, Development, and Deployment
- Resilient Architectures
The goal of the Multi-model Testbed is to provide a collaborative design tool for evaluating various cyber-attack / defense strategies and their effects on the physical infrastructure. The web-based, cloud-hosted environment integrates state-of-the-art simulation engines for the different CPS domains and presents interesting research challenges as ready to use scenarios. Input data, model parameters, and simulation results are archived, versioned with a strong emphasis on repeatability and provenance.
PUBLICATIONS
Himanshu Neema, Peter Volgyesi, Xenofon Koutsoukos, Thomas Roth, Cuong Nguyen, "Online Testbed for Evaluating Vulnerability of Deep Learning Based Power Grid Load Forecasters", 8th Workshop on Modeling and Simulation of Cyber-Physical Energy Systems, Cyber-Physical Systems and Internet-of-Things Week 2020. (accepted paper, video presentation)
KEY HIGHLIGHTS
Due to the COVID-19 pandemic, the MSCPES 2020 workshop is hosted as a virtual event. Our paper and the pre-recorded video presentation are posted at http://palensky.org/mscpes/2020/videos.php. Our paper and the presentation focus on the development and architecture of the Multi-model Testbed, providing a good overview of this effort since the beginning of the project.
| Figure 1: MSCPES Overview: Machine Learning in Power Grid Load Forecasting |
 |
Scikit-learn for DeepForge
On the technical side, we created an initial prototype extension for DeepForge to integrate the scikit-learn library. The core DeepForge modeling framework does not prescribe any particular machine learning library but provides an extension mechanism to adopt such toolboxes. Indeed, this is how the current TensorFlow/Keras backend has been implemented in DF.
The Scikit-learn library is one of the most well-known packages containing a rich set of algorithms and efficient tools for predictive data analysis. It follows a complementary approach to deep AI libraries - such as TensorFlow, Pytorch, Caffe - as it covers almost all aspects of machine-learning except for deep networks.
Beyond the actual algorithms, the most significant benefit of scikit-learn is a unified and simple API, based on the fit-transform-predict operations. The Keras library was heavily influenced by this API pattern. Thus, integrating several scikit-learn algorithms in DeepForge is a relatively straightforward task with an immediate benefit of providing access to a rich set of well-known machine learning algorithms (linear models, SVM, PCA, ensemble methods, etc.). Figure 2 shows a simple regression model using and comparing an SVM-based regressor to Gradient Boosting.
| Figure 2: Scikit-learn pipelines in DeepForge |
 |
The scikit-learn library already provides a flexible and efficient pipeline mechanism (not to be confused with DeepForge pipelines), which can be leveraged in the code generation phase. Note, that the current early prototype implementation does not have an end-to-end code generation capability, yet.
FUTURE PLANS
We plan to do further experimentation with scikit-learn inside DeepForge, and will start the development of a complete DeepForge extension supporting this library. We also plan to provide example models - relevant to the science of security - either from scratch or by applying more traditional ML-methods to our existing testbed models (e.g. load forecasting)
Since the release of the TensorFlow 2 framework from Google the current TensorFlow/Keras extension became somewhat out-of-date. Fortunately, this extension primarily targets the Keras API, which did not change as significantly as TensorFlow. However, since TensorFlow 2 integrates its own optimized version of the Keras library (tf.keras), it would be beneficial to bring this extension up-to-date. See the corresponding issue on GitHub for details.
Our long-term open issue is a tighter integration of the GridLAB-D studio with DeepForge. Currently, these models are stored in separate projects with lightweight cross-references. Developing a more unified modeling architecture is a longer-term goal for the next three quarters.
COMMUNITY ENGAGEMENTS, TECHNOLOGY TRANSFER
None, during this reporting period.
EDUCATIONAL ADVANCES:
None, during this reporting period.