SoS Lablet Annual Report - NCSU

Lablet Annual Report
Purpose: To highlight progress made within the first base year (March 2014 to Present). Information is generally at a higher level which is accessible to the interested public. This will be published in an overall SoS Annual Report to be shared with stakeholders to highlight the accomplishments the Lablets have made over the past year.

A). Lablet Introduction 
Please include each of the following:

• General introduction about the Lablet - 1 paragraph

North Carolina State University’s (NCSU) Science of Security Lablet (SoSL) has embraced and helped build a foundation for NSA’s vision of the Science of Security (SoS) and of a SoS community.  We have emphasized data-driven discovery and analytics to formulate, validate, evolve, and solidify the theory and practice of security. Efforts in our current lablet have yielded significant findings, providing a deeper understanding of users’ susceptibility to deception, developers’ adoption of security tools, how trust between people relates to their commitments.  Motivated by NSA’s overarching vision for SoS and building on our experience and accomplishments, we are (1) developing a science-based foundation for the five hard problems that we previously helped formulate; and (2) fostering a SoS community with high standards for reproducible research. Our approach involves a comprehensive, rigorous perspective on SoS, including an integrated treatment of technical artifacts, humans (both stakeholders and adversaries) along with relationships and processes relevant to the hard problems.  Continual evaluation of our research and community development efforts is key to our approach.

• Team description (universities that are Lablets, Sub-Lablets, and any collaborators) - 1 paragraph

We have formed teams to conduct scientific research and evaluate progress on hard problems: Security Metrics and Models; Humans; Policy; and Resilient Architectures. The Scalability and Composability hard problem has no explicit team since we address it as a secondary hard problem in several of our projects.  Each Hard Problem team is composed of three or four projects researching complimentary aspects of the Hard Problem. We also have additional teams for Research Methods, Community Development and Support, and for Evaluation. 

• Security Metrics and Models

Attack Surface and Defense-in-Depth Metrics:  Rochester Institute of Technology:  Andy Meneely, NC State University:  Laurie Williams

Systematization of Knowledge from Intrusion Detection Models:  NC State University:  Huaiyu Dai, Rochester Institute of Technology:  Andy Meneely

Vulnerability and Resilience Prediction Models:  NC State University:  Mladen Vouk, Laurie Williams

• Humans

Warning of Phishing Attacks: Supporting Human Information Processing, Identifying Phishing Deception Indicators, and Reducing Vulnerability:  NC State University:  Christopher B. Mayhorn, Emerson Murphy-Hill

A Human Information-Processing Analysis of Online Deception Detection:  Purdue University:  Robert W. Proctor, Ninghu Li

Leveraging the Effects of Cognitive Function on Input Device Analytics to Improve Security:  NC State University:  David L. Roberts, Robert St. Amant

• Policy

Understanding Effects of Norms and Policies on the Robustness, Liveness, and Resilience of Systems:  NC State University:  Emily Berglund, Jon Doyle, Munindar Singh

Formal Specification and Analysis of Security- Critical Norms and Policies:  NC State University:  Jon Doyle, Munindar Singh, Rada Chirkova

Scientific Understanding of Policy Complexity:  Purdue University:  Ninghui Li, Robert Proctor, NC State University:  Emerson Murphy-Hill

• Resilient Architectures

Resilience Requirements, Design, and Testing:  University of Virginia:  Kevin Sullivan, NC State University:  Mladen Vouk, University of North Carolina at Charlotte:  Ehab Al-Shaer

Redundancy for Network Intrusion Prevention Systems (NIPS):  University of North Carolina:  Mike Reiter

Smart Isolation in Large-Scale Production Computing Infrastructures:  NC State University:  Xiaohui (Helen) Gu, William Enck

Automated Synthesis of Resilient Architectures:  University of North Carolina at Charlotte:  Ehab Al-Shaer

• Research Methods, Community Development and Support:  University of Alabama:  Jeff Carver, NC State University:  Lindsey McGowen, Jon Stallings, Laurie Williams, David Wright
• Evaluation:  NC State University:  Lindsey McGowen, Jon Stallings, David Wright, University of Alabama:  Jeff Carver

• Overall viewpoint of the progress made over the past year - 1-2 paragraphs

Our collective efforts have advanced the science of security by bring forth best practices and research methodologies to the hard problems in security.

• We are developing an agent-based simulation methodology for secure collaboration. Agent-based simulation is an established methodology for understanding complex systems formed of the interactions of autonomous parties and where analytical solutions are not expected to be found. It has been used in areas such as epidemiology and, given the complexity of systems in connection with cybersecurity, should be a component of security research.
• We are advancing the nature and rigor of empirical studies of end users via lab experiments and surveys.
• We are advancing the empirical study of system administration via evaluations of artifacts such as software and policies through tools as well as through experimental studies in which participants apply competing approaches to create specifications (thereby helping us address insidious security errors through errors in modeling or configuration).
• Through example, we are advancing the systematic development of survey papers in the science of security.

B). Fundamental Research
High level report of results for each project that helped move security science forward -- in most cases it should point to a "hard problem". - 1 paragraph per project
 

*** Hard problem progress summaries are from the other report.  We also need a breif summary of progress for each of the projects.  ***

• Security Metrics and Models

We have results related to attack surface metrics and vulnerabilities. Many organizations prioritize security efforts around the general idea of attack surface (entry and exit points of a software program), considering areas of the code not reachable by an attacker to be a lower priority. However, the process for practically identifying what part of the code is on the attack surface and specific attack surface metrics have not been validated. We have results (weakly) associating our attack surface metrics with vulnerabilities. Additionally, our analysis of crash dumps at Microsoft indicates that the code identified in crash dumps accounts for most (94.6%) of the vulnerabilities, indicating that crash dumps may be used to indicate whether a piece of code is on the attack surface or not.

Attack Surface and Defense-in-Depth Metrics: 

Systematization of Knowledge from Intrusion Detection Models: 

Vulnerability and Resilience Prediction Models: 

• Humans

We have developed a cognitive model of users based on the well-known ACT-R framework. We have developed an understanding of observable human behaviors that indicates the level of thought a user puts into an action (as a measure of the naturalness of the action). These contributions provide some of the bases of the science underlying the humans hard problem by leading us to an understanding of (1) how users process information and make security-relevant mistakes and the bases on which we may identify such mistakes; (2) how to distinguish potential deceptive user behaviors through largely unobtrusive observations of users; and (3) how to generate cognitively and ecologically relevant warnings to users to assist them in their security-relevant decision making.

Warning of Phishing Attacks: Supporting Human Information Processing, Identifying Phishing Deception Indicators, and Reducing Vulnerability: 

A Human Information-Processing Analysis of Online Deception Detection: 

Leveraging the Effects of Cognitive Function on Input Device Analytics to Improve Security: 

• Policy

We have developed metrics of policy complexity that capture how difficult a set of policies is for people to comprehend, which are indicative of configuration errors that lead to vulnerabilities. We have identified such errors in practical enterprise policies. We have developed and evaluated an argumentation-based approach for capturing firewall requirements that reduces errors and improves comprehensibility over traditional methods. We are studying policy errors in software and how to ameliorate such errors based on principles pertaining to software analytics. We have developed a normative formulation of accountability that captures its essential features separately from traceability; we have additionally developed a (partial) approach that relates normative relationships to data representations as a basis for logging and analytics. We have developed a simulation framework in which to study the robustness and resilience of norms that modulate the security-relevant behaviors and interactions of users, as a basis for understanding and exploring potential norms.

Understanding Effects of Norms and Policies on the Robustness, Liveness, and Resilience of Systems: 

Formal Specification and Analysis of Security- Critical Norms and Policies: 

Scientific Understanding of Policy Complexity: 

• Resilient Architectures

We have created a classification scheme of existing isolation techniques. The purpose of the scheme is to enable the identification of underlying design principles the tradeoffs between them. Discovering these principles will aid in the design of the next generation of smart isolation techniques to support resilient architectures. Similarly, we have created a taxonomy of resiliency metrics. The purpose of the metrics are to estimate the resiliency level that a system exhibit given specific attack model or scenario, and the implied recommended actions to improved resiliency.

Resilience Requirements, Design, and Testing: 

Redundancy for Network Intrusion Prevention Systems (NIPS): 

Smart Isolation in Large-Scale Production Computing Infrastructures: 

Automated Synthesis of Resilient Architectures: 

C). Publications
Please list all publications published in the base year starting in March 2014 to present.

• Amant, R. S. & Goodwin, P. R. & Dominguez, I. & Roberts, D. L. (2015) "Toward Expert Typing in ACT-R." Proceedings of the 2015 International Conference on Cognitive Modeling (ICCM 15)
• Chopra, A. K. & Singh, M. P. (2015) "Cupid: Commitments in Relational Algebra." Proceedings of the 23rd Conference on Artificial Intelligence (AAAI)
• Chopra, A. K. & Singh, M. P. (2014) "The Thing Itself Speaks: Accountability as a Foundation for Requirements in Sociotechnical Systems." Requirements Engineering and Law (RELAW), 2014 IEEE 7th International Workshop on
• Dominguez, I. X. & Goel, A. & Roberts, D. L. & Amant, R. S. (2015) "Detecting Abnormal User Behavior Through Pattern-mining Input Device Analytics." Proceedings of the 2015 Symposium and Bootcamp on the Science of Security (HotSoS-15)
• Dou, K. & Wang, X. & Tang, C. & Ross, A. & Sullivan, K. (2015) "An Evolutionary Theory-Systems Approach to a Science of the Ilities." Proceedings of the Conference on Systems Engineering Research (CSER 2015)
• Du, H. & Narron, B. Y. & Ajmeri, N. & Berglund, E. & Doyle, J. & Singh, M. P. (2015) "ENGMAS textendash Understanding Sanction under Variable Observability in a Secure Environment." Proceedings of the 2nd International Workshop on Agents and CyberSecurity (ACySE)
• Du, H. & Narron, B. Y. & Ajmeri, N. & Berglund, E. & Doyle, J. & Singh, M. P. (2015) "Understanding Sanction under Variable Observability in a Secure, Collaborative Environment." Proceedings of the International Symposium and Bootcamp on the Science of Security (HotSoS)
• Heorhiadi, V. & Fayaz, S. & Reiter, M. K. & Sekar, V. (2014) "SNIPS: A Software-Defined Approach for Scaling Intrusion Prevention Systems via Offloading." 10th International Conference on Information Systems Security, ICISS 2014
• Huang, Y. & He, X. & Dai, H. (2015) "Poster: Systematization of Metrics in Intrusion Detection Systems." ACM Proc. Of the Symposium and Bootcamp on the Science of Security (HotSoS), University of Illinois at Urbana-Champaign, IL
• Kim, D. & Vouk, M. (2014) "A survey of common security vulnerabilities and corresponding countermeasures for SaaS." Second IEEE International workshop on Cloud Computing Systems, Networks, and Applications (CCSNA-2014)
• Mayhorn, C. B. & Welk, A. K. & Zielinska, O. A. & Murphy-Hill, E. (2015) "Assessing individual differences in a phishing detection task." Proceedings of the Annual International Ergonomics Association Conference
• Rahman, M. A. & Al-Shaer, E. & Bobba, R. B. (2014) "Moving Target Defense for Hardening the Security of the Power System State Estimation." First ACM Workshop on Moving Target Defense
• Rahman, M. A. & Al-Shaer, E. & Kavasseri, R. G. (2014) "Impact Analysis of Topology Poisoning Attacks on Economic Operation of the Smart Power Grid." Proceedings of the 34th International Conference on Distributed Computing Systems (ICDCS)
• Rivers, A. T. & Vouk, M. A. & Williams, L. A. (2014) "On Coverage-Based Attack Profiles." Software Security and Reliability-Companion (SERE-C), 2014 IEEE Eighth International Conference on
• Singh, M. P. (2015) "Norms as a Basis for Governing Sociotechnical Systems: Extended Abstract." Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI)
• Singh, M. P. (2015) "Cybersecurity as an Application Domain for Multiagent Systems." Proceedings of the 14th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)
• Subramani, S. (2014) "A Study of Fedora Security Profile."
• Venkatakrishnan, R. (2014) "Redundancy-Based Detection of Security Anomalies in Web-Server Environments."
• Welk, A. K. & Mayhorn, C. B. (2015) "All Signals Go: Investigating How Individual Differences Affect Performance on a Medical Diagnosis Task Designed to Parallel a Signal Intelligence Analyst Task." Symposium and Bootcamp on the Science of Security (HotSoS)
• Zielinska, O. A. & Tembe, R. & Hong, K. W. & Ge, X. & Murphy-Hill, E. & Mayhorn, C. B. (2014) "One Phish, Two Phish, How to Avoid the Internet Phish: Analysis of Training Strategies to Detect Phishing Emails." Human Factors and Ergonomics Society Annual Meeting
• Zielinska, O. & Welk, A. & Mayhorn, C. B. & Murphy-Hill, E. (2015) "Exploring expert and novice mental models of phishing." Proceedings of the 2nd HotSoS: Symposium and Bootcamp on the Science of Security

D). Community Engagements
Briefly describe your Lablets community outreach efforts to extend scientific rigor in the community/culture. For example, list workshops, seminars, competitions, etc. that your Lablet has accomplished since March 2014 to present.

• In April 2014, we organized the first HotSoS in Raleigh.  Over 130 leaders from government, industry, and the academic community met to discuss new and ongoing programs in security science. The presentations emphasized a broad range of topics including computing architectures, networks, software engineering practices, models of human interaction and behavior, organizational models, and evaluation methodologies.
• In Summer 2014, we conducted a two-day research workshop for lablet participants on best practices for the science of security. This workshop included sessions conducted by a statistician on experimental design; by a "science of science" methodologist on the ways in which we can collectively advance the science of security; by a computer scientist on conducting empirical software engineering research. A similar workshop is planned for late May 2015.
• In October 2014, we conducted an Industry Day workshop whose first part involved Pecha Kucha presentations by all lablet projects; presentations by invited industry speakers; and a poster session by students. We use this workshop as a way to engage more closely with industry colleagues both in advancing cybersecurity and in promoting the science of security. 
• We are developing a rubric for reviewing research publications in a way that seeks to bring out and evaluate their core scientific claims and findings. We offered a workshop at the January 2015 quarterly meeting based on this rubric. The idea is that this rubric would sensitize researchers to the scientific aspects of security research and thereby lead to papers and peer reviews that are more clearly scientific and thus lead to improved scientific research overall.
• We have taken numerous opportunities to give keynote and other invited lectures on the science of security to broader computer science communities as a way to bring them into the fold.  Need to provide details here, e.g., a list of keynotes & invited talks.

SoS Quarterly Meetings

Science of Security Quarterly Lablet PI Meeting, July 2014: 

Lindsey McGowen, "Evaluating the Development of a Science of Security: A Plan for Measuring and Demonstrating Lablet Contributions."

Science of Security Quarterly Lablet PI Meeting, October 2014: 

Andrew Meneely, "Developing Security Metrics." 

Munindar Singh, "Survey on Policy-Governed Secure Collaboration."

Science of Security Quarterly Lablet PI Meeting, January 2015: 

William Enck, "Systematizing Isolation Techniques." 

Lindsey McGowen, "Customized Bibliometrics for Evaluating Computer Science Research." 

Ehab Al-Shaer, "On Objective Resiliency Analysis of Smart Grid Energy Management Systems."

E). Educational
Briefly describe any changes to curriculum at your school or elsewhere that indicates an increased training or rigor in security research that your Lablet has accomplished since March 2014 to present.

Just out of curiosity, has any of the lablet research found its way into CSC courses? 

We have continued to hold (approximately) bi-weekly research seminars in the Fall 2014 and Spring 2015 academic semesters.  These seminars consist of two main types of discussions and are attended by NCSU as well as remote participants at our collaborating institutions.

• In research design seminars, students present their designs for their proposed study, including not only the motivation and existing theoretical frameworks, but also details of the theoretical or empirical investigations they plan to carry out. Our motivation for discussing research designs is, first, to reflect on the nature of an investigation before launching into the effort and, second, to vet the proposed design in consultation with peers in the lablet. The intended benefit is in strengthening the scientific basis of the research by improving clarity of the hypotheses and metrics underlying the research as well as ensuring the design would help evaluate those hypotheses.

• In manuscript review seminars, students make a presentation about a manuscript they are preparing for submission for peer review. The intended benefit is in strengthening the positioning of the research with respect to the literature and in discussing the robustness of the claimed evaluation of the hypotheses.