Science of Human Circumvention of Security - October 2015

Public Audience
Purpose: To highlight project progress. Information is generally at a higher level which is accessible to the interested public. All information contained in the report (regions 1-3) is a Government Deliverable/CDRL.

PI(s): Tao Xie

Co-PI(s): Jim Blythe (USC), Ross Koppel (UPenn), and Sean Smith (Dartmouth)

HARD PROBLEM(S) ADDRESSED
This refers to Hard Problems, released November 2012.

Our project most closely aligns with problem 5, "Understanding and Accounting for Human Behavior", however it also pertains to problems 1, 2, and 3:

• Scalability and Composability: We want to understand not just the drivers of individual incidents of human circumvention, but also the net effect of these incidents on security, usefulness of the software, integration into workflow, learnability, and user frustrations. Included here are measures of the environment (physical, organizational, hierarchical, embeddedness within larger systems.)
• Policy-Governed Secure Collaboration: In order to create policies that in reality actually enable secure collaboration among users in varying domains, we need to understand and predict the de facto consequences of policies, not just the de juro ones.
• Security-Metrics-Driven Evaluation, Design, Development, and Deployment: Making sane decisions about what security controls to deploy requires understanding the de facto consequences of these deployments---instead of just pretending that circumvention by honest users never happens.

PUBLICATIONS
Papers published in this quarter as a result of this research. Include title, author(s), venue published/presented, and a short description or abstract. Identify which hard problem(s) the publication addressed. Papers that have not yet been published should be reported in region 2 below.

[1] R. Koppel, S. Smith, J. Blythe, and V. Kothari, "Workarounds to Computer Access in Healthcare Organizations: You Want My Password or a Dead Patient?" Driving Quality in Informatics: Fulfilling the Promise. Karen L. Courtney, Alex Kuo, Omid Shabestari, Eds. Series on Technology and Informatics, 209. Amsterdam, Netherlands: IOS Press, 2015

Abstract: Workarounds to computer access in healthcare are sufficiently common that they often go unnoticed. Clinicians focus on patient care, not cybersecurity. We argue and demonstrate that understanding workarounds to healthcare workers' computer access requires not only analyses of computer rules, but also interviews and observations with clinicians. In addition, we illustrate the value of shadowing clinicians and conducing focus groups to understand their motivations and tradeoffs for circumvention. Ethnographic investigation of the medical workplace emerges as a critical method of research because in the inevitable conflict between even well-intended people versus the machines, it's the people who are the more creative, flexible, and motivated. We conducted interviews and observations with hundreds of medical workers and with 19 cybersecurity experts, CIOs, CMIOs, CTO, and IT workers to obtain their perceptions of computer security. We also shadowed clinicians as they worked. We present dozens of ways workers ingeniously circumvent security rules. The clinicians we studied were not "black hat" hackers, but just professionals seeking to accomplish their work despite the security technologies and regulations.

This publication addresses Problems 5,1,2,3.

[2] R. Koppel, S. Smith, J. Blythe, and V. Kothari, "Workarounds to Computer Access in Healthcare Organizations: You Want My Password or a Dead Patient?" Presentation by Koppel at International Conference Addressing Information Technology and Communications In Health, 2015. Victoria, BC, Canada. February/March 2015. Published in Driving Quality in Informatics: Fulfilling the Promise. IOS Press, Studies In Health Technology and Informatics, Volume 208, pp 215-20, 2015.

[3] S.W. Smith, R. Koppel, J. Blythe, V. Kothari. Mismorphism: A Semiotic Model of Computer Security Circumvention (Extended Version). Computer Science Technical Report TR2015-768. Dartmouth College. March 2015.

Abstract: In real world domains, from healthcare to power to finance, we deploy computer systems intended to streamline and improve the activities of human agents in the corresponding non-cyber worlds. However, talking to actual users (instead of just computer security experts) reveals endemic circumvention of the computer-embedded rules. Good-intentioned users, trying to get their jobs done, systematically work around security and other controls embedded in their IT systems. This paper reports on our work compiling a large corpus of such incidents and developing a model based on semiotic triads to examine security circumvention. This model suggests that mismorphisms---mappings that fail to preserve structure---lie at the heart of circumvention scenarios; differential perceptions and needs explain users' actions. We support this claim with empirical data from the corpus.

This submission addresses Problems 5,1,2,3.

[4] Mismorphism: a Semiotic Model of Computer Security Circumvention, Smith, Koppel, Blythe and Kothari, 9th International Symposium on Human Aspects of Information Security and Assurance, 2015.

Smith Presented this in July 2015

Addresses 5,1,2,3; shorter, revised version of [3] above.

[5] Koppel gave keynote presentation at Royal College of Physicians (Edinburgh) on healthcare software usability and the influence on compliance with cyber security rules February 2015 (Co-presented with Professor Harold Thimbleby, Computer Science Department, Swansea University, Wales, UK. "Dangers and Frustrations of Poorly Designed and Badly Implemented Healthcare IT: Implications for Medication Errors"

This submission addresses Problems 5 and 3

[6] Koppel gave presentation to Wales Health Trust at Prince of Wales Hospital, Swansea, Wales, UK. February 2015.

This submission addresses Problems 5.

[7] V. Kothari, J. Blythe, S.W. Smith, R. Koppel. "Measuring the Security Impacts of Password Policies Using Cognitive Behavioral Agent-Based Modeling." Symposium and Bootcamp on the Science of Security (HotSoS 2015). ACM. April 2015.

Abstract: Agent-based modeling can serve as a valuable asset to security personnel who wish to better understand the security landscape within their organization, especially as it relates to user behavior and circumvention. In this paper, we argue in favor of cognitive behavioral agent-based modeling for usable security and report on our work on developing an agent-based model for a password management scenario. We perform a number of trials and a sensitivity analysis that provide valuable insights into improving security (e.g., an organization that wishes to suppress one form of circumvention may want to endorse another form of circumvention).

This publication addresses Problems 5,1,2,3.

[8] S.W. Smith, R. Koppel, J. Blythe, V. Kothari. "Mismorphism: A Semiotic Model of Computer Security Circumvention (Poster Abstract)." Symposium and Bootcamp on the Science of Security (HotSoS 2015). ACM. Accepted for publication; to appear April 2015. See [3 and 4] above.

[9] J. Blythe, R. Koppel, V. Kothari, S. Smith, "The Computer Security Perils of Reuse", Submitted for publication, March 2015.

Abstract: When developing new systems and components, designers routinely reuse existing policies, technologies, and architectures--frequently with little or no changes. Standard software engineering practice advocates the reuse of reliable components. However, our findings reveal that careless reuse in a different or even similar domain can introduce failures and new challenges that subvert security goals and impede organizational objectives. In this paper, we enumerate and analyze examples of reuse in various settings. We examine the motivations for reuse including its advantages, its disadvantages, human biases, and the real and the false economies it provides. We also study the factors and conditions that affect the success of reuse to provide recommendations to security personnel.

This submission addresses Problems 5,1,2,3.

[10] W. Yang, X. Xiao, B Andow, S. Li, T. Xie, and W. Enck. "AppContext: Differentiating Malicious and Benign Mobile App Behavior Under Contexts." In Proceedings of the 37th International Conference on Software Engineering (ICSE 2015), Florence, Italy, May 2015.

Abstract: Mobile malware attempts to evade detection during app analysis by mimicking security-sensitive behaviors of benign apps that provide similar functionality (e.g., sending SMS messages), and suppressing their payload to reduce the chance of being observed (e.g., executing only its payload at night). Since current approaches focus their analyses on the types of security-sensitive resources being accessed (e.g., network), these evasive techniques in malware make differentiating between malicious and benign app behaviors a difficult task during app analysis. We propose that the malicious and benign behaviors within apps can be differentiated based on the contexts that trigger security-sensitive behaviors, i.e., the events and conditions that cause the security-sensitive behaviors to occur. In this work, we introduce AppContext, an approach of static program analysis that extracts the contexts of security-sensitive behaviors to assist app analysis in differentiating between malicious and benign behaviors. We implement a prototype of AppContext and evaluate AppContext on 202 malicious apps from various malware datasets, and 633 benign apps from the Google Play Store. AppContext correctly identifies 192 malicious apps with 87.7% precision and 95% recall. Our evaluation results suggest that the maliciousness of a security-sensitive behavior is more closely related to the intention of the behavior (reflected via contexts) than the type of the security-sensitive resources that the behavior accesses.

This paper addresses Problems 5,1,3.

PhD student Wei Yang presented this in May 2015.

[11] T. Xie, J. Bishop, N. Tillmann, and J. de Halleux. "Gamifying Software Security Education and Training via Secure Coding Duels in Code Hunt". In Proceedings of Symposium and Bootcamp on the Science of Security (HotSoS 2015), Urbana, IL, April 2015.

Abstract: Sophistication and flexibility of software development make it easy to leave security vulnerabilities in software applications for attackers. It is critical to educate and train software engineers to avoid introducing vulnerabilities in software applications in the first place such as adopting secure coding mechanisms and conducting security testing. A number of websites provide training grounds to train people's hacking skills, which are highly related to security testing skills, and train people's secure coding skills. However, there exists no interactive gaming platform for instilling gaming aspects into the education and training of secure coding. To address this issue, we propose to construct secure coding duels in Code Hunt, a high-impact serious gaming platform released by Microsoft Research. In Code Hunt, a coding duel consists of two code segments: a secret code segment and a player-visible code segment. To solve a coding duel, a player iteratively modifies the player-visible code segment to match the functional behaviors of the secret code segment. During the duel-solving process, the player is given clues as a set of automatically generated test cases to characterize sample functional behaviors of the secret code segment. The game aspect in Code Hunt is to recognize a pattern from the test cases, and to re-engineer the player-visible code segment to exhibit the expected behaviors. Secure coding duels proposed in this work are coding duels that are carefully designed to train players' secure coding skills, such as sufficient input validation and access control.

This paper addresses Problems 5,1,3.

Xie presented this in April 2015.

[12] Xusheng Xiao, Nikolai Tillmann, Manuel Fahndrich, Jonathan de Halleux, Michal Moskal, and Tao Xie. User-Aware Privacy Control via Extended Static-Information-Flow Analysis. Automated Software Engineering Journal, 22(3), pages 333-366, 2015.

Abstract: Applications in mobile marketplaces may leak private user information without notification. Existing mobile platforms provide little information on how applications use private user data, making it difficult for experts to validate applications and for users to grant applications access to their private data. We propose a user-aware-privacy-control approach, which reveals how private information is used inside applications. We compute static information flows and classify them as safe/unsafe based on a tamper analysis that tracks whether private data is obscured before escaping through output channels. This flow information enables platforms to provide default settings that expose private data for only safe flows, thereby preserving privacy and minimizing decisions required from users. We build our approach into TouchDevelop, an application-creation environment that allows users to write scripts on mobile devices and install scripts published by other users. We evaluate our approach by studying 546 scripts published by 194 users, and the results show that our approach effectively reduces the need to make access-granting choices to only 10.1 % (54) of all scripts. We also conduct a user survey that involves 50 TouchDevelop users to assess the effectiveness and usability of our approach. The results show that 90 % of the users consider our approach useful in protecting their privacy, and 54 % prefer our approach over other privacy-control approaches.

This paper addresses Problems 5,1,3.

ACCOMPLISHMENT HIGHLIGHTS

Via fieldwork in real-world enterprises, we have been identifying and cataloging types and causes of circumvention by well-intentioned users. We are using help desk logs, records security-related computer changes, analysis of user behavior in situ, and surveys--in addition to interviews and observations. We then began to build and validate models of usage and circumvention behavior, for individuals and then for populations within an enterprise--as well as developing some typologies of the deeper patterns and causes.