Science of Human Cirumvention of Science - October 2016

Public Audience
Purpose: To highlight project progress. Information is generally at a higher level which is accessible to the interested public. All information contained in the report (regions 1-3) is a Government Deliverable/CDRL.

PI(s): Tao Xie

Co-PI(s): Jim Blythe (USC), Ross Koppel (UPenn), and Sean Smith (Dartmouth)

HARD PROBLEM(S) ADDRESSED
This refers to Hard Problems, released November 2012.

Our project most closely aligns with problem 5, "Understanding and Accounting for Human Behavior." However, it also pertains to problems 1, 2, and 3:

• Scalability and Composability: We want to understand not just the drivers of individual incidents of human circumvention, but also the net effect of these incidents.Included here are measures of the environment (physical, organizational, hierarchical, embeddedness within larger systems.)
• Policy-Governed Secure Collaboration: In order to create policies that in reality actually enable secure collaboration among users in varying domains, we need to understand and predict the de facto consequences of policies, not just the de juro ones.
• Security-Metrics-Driven Evaluation, Design, Development, and Deployment:Making sane decisions about what security controls to deploy requires understanding the de facto consequences of these deployments---instead of just pretending that circumvention by honest users never happens.

PUBLICATIONS
Papers published in this quarter as a result of this research. Include title, author(s), venue published/presented, and a short description or abstract. Identify which hard problem(s) the publication addressed. Papers that have not yet been published should be reported in region 2 below.

[1] R. Koppel, S. Smith, J. Blythe, and V. Kothari, "Workarounds to Computer Access in Healthcare Organizations: You Want My Password or a Dead Patient?", Driving Quality in Informatics: Fulfilling the Promise, Karen L. Courtney, Alex Kuo, Omid Shabestari, Eds. Series on Technology and Informatics, Volume 208, pp 215-20, 2015. Amsterdam, Netherlands: IOS Press, 2015.

Abstract: Workarounds to computer access in healthcare are sufficiently common that they often go unnoticed. Clinicians focus on patient care, not cybersecurity. We argue and demonstrate that understanding workarounds to healthcare workers' computer access requires not only analyses of computer rules, but also interviews and observations with clinicians. In addition, we illustrate the value of shadowing clinicians and conducing focus groups to understand their motivations and tradeoffs for circumvention. Ethnographic investigation of the medical workplace emerges as a critical method of research because in the inevitable conflict between even well-intended people versus the machines, it's the people who are the more creative, flexible, and motivated. We conducted interviews and observations with hundreds of medical workers and with 19 cybersecurity experts, CIOs, CMIOs, CTO, and IT workers to obtain their perceptions of computer security. We also shadowed clinicians as they worked. We present dozens of ways workers ingeniously circumvent security rules. The clinicians we studied were not "black hat" hackers, but just professionals seeking to accomplish their work despite the security technologies and regulations.

This publication addresses Problems 5,1,2,3.

[2] S.W. Smith, R. Koppel, J. Blythe, and V. Kothari. Mismorphism: A Semiotic Model of Computer Security Circumvention (Extended Version). Computer Science Technical Report TR2015-768. Dartmouth College. March 2015.

Abstract: In real world domains, from healthcare to power to finance, we deploy computer systems intended to streamline and improve the activities of human agents in the corresponding non-cyber worlds. However, talking to actual users (instead of just computer security experts) reveals endemic circumvention of the computer-embedded rules. Good-intentioned users, trying to get their jobs done, systematically work around security and other controls embedded in their IT systems. This paper reports on our work compiling a large corpus of such incidents and developing a model based on semiotic triads to examine security circumvention. This model suggests that mismorphisms---mappings that fail to preserve structure---lie at the heart of circumvention scenarios; differential perceptions and needs explain users' actions. We support this claim with empirical data from the corpus.

This submission addresses Problems 5,1,2,3.

[3] Koppel gave keynote presentation at Royal College of Physicians (Edinburgh) on healthcare software usability and the influence on compliance with cyber security rules February 2015 (Co-presented with Professor Harold Thimbleby, Computer Science Department, Swansea University, Wales, UK. "Dangers and Frustrations of Poorly Designed and Badly Implemented Healthcare IT: Implications for Medication Errors".

This submission addresses Problems 5 and 3

[4] Koppel gave presentation to Wales Health Trust at Prince of Wales Hospital, Swansea, Wales, UK. February 2015.

This submission addresses Problems 5.

[5] V. Kothari, J. Blythe, S.W. Smith, and R. Koppel, "Measuring the Security Impacts of Password Policies Using Cognitive Behavioral Agent-Based Modeling", Symposium and Bootcamp on the Science of Security (HotSoS 2015), ACM. April 2015.

Abstract: Agent-based modeling can serve as a valuable asset to security personnel who wish to better understand the security landscape within their organization, especially as it relates to user behavior and circumvention. In this paper, we argue in favor of cognitive behavioral agent-based modeling for usable security and report on our work on developing an agent-based model for a password management scenario. We perform a number of trials and a sensitivity analysis that provide valuable insights into improving security (e.g., an organization that wishes to suppress one form of circumvention may want to endorse another form of circumvention).

This publication addresses Problems 5,1,2,3.

[6] S.W. Smith, R. Koppel, J. Blythe, and V. Kothari, "Mismorphism: A Semiotic Model of Computer Security Circumvention (Poster Abstract)", Symposium and Bootcamp on the Science of Security (HotSoS 2015), ACM, April 2015. See [2] above and [11] below.

[7] J. Blythe, R. Koppel, V. Kothari, and S. Smith, "The Computer Security Perils of Reuse", Submitted for publication, March 2015.

Abstract: When developing new systems and components, designers routinely reuse existing policies, technologies, and architectures--frequently with little or no changes. Standard software engineering practice advocates the reuse of reliable components. However, our findings reveal that careless reuse in a different or even similar domain can introduce failures and new challenges that subvert security goals and impede organizational objectives. In this paper, we enumerate and analyze examples of reuse in various settings. We examine the motivations for reuse including its advantages, its disadvantages, human biases, and the real and the false economies it provides. We also study the factors and conditions that affect the success of reuse to provide recommendations to security personnel.

This submission addresses Problems 5,1,2,3.

[8] T. Xie, J. Bishop, N. Tillmann, and J. de Halleux. "Gamifying Software Security Education and Training via Secure Coding Duels in Code Hunt". In Proceedings of Symposium and Bootcamp on the Science of Security (HotSoS 2015), Urbana, IL, April 2015.

Abstract: Sophistication and flexibility of software development make it easy to leave security vulnerabilities in software applications for attackers. It is critical to educate and train software engineers to avoid introducing vulnerabilities in software applications in the first place such as adopting secure coding mechanisms and conducting security testing. A number of websites provide training grounds to train people's hacking skills, which are highly related to security testing skills, and train people's secure coding skills. However, there exists no interactive gaming platform for instilling gaming aspects into the education and training of secure coding. To address this issue, we propose to construct secure coding duels in Code Hunt, a high-impact serious gaming platform released by Microsoft Research. In Code Hunt, a coding duel consists of two code segments: a secret code segment and a player-visible code segment. To solve a coding duel, a player iteratively modifies the player-visible code segment to match the functional behaviors of the secret code segment. During the duel-solving process, the player is given clues as a set of automatically generated test cases to characterize sample functional behaviors of the secret code segment. The game aspect in Code Hunt is to recognize a pattern from the test cases, and to re-engineer the player-visible code segment to exhibit the expected behaviors. Secure coding duels proposed in this work are coding duels that are carefully designed to train players' secure coding skills, such as sufficient input validation and access control.

This paper addresses Problems 5,1,3.

Xie presented this in April 2015.

[9] W. Yang, X. Xiao, B Andow, S. Li, T. Xie, and W. Enck, "AppContext: Differentiating Malicious and Benign Mobile App Behavior Under Contexts", 37th International Conference on Software Engineering (ICSE 2015), Florence, Italy, May 2015.

Abstract: Mobile malware attempts to evade detection during app analysis by mimicking security-sensitive behaviors of benign apps that provide similar functionality (e.g., sending SMS messages), and suppressing their payload to reduce the chance of being observed (e.g., executing only its payload at night). Since current approaches focus their analyses on the types of security-sensitive resources being accessed (e.g., network), these evasive techniques in malware make differentiating between malicious and benign app behaviors a difficult task during app analysis. We propose that the malicious and benign behaviors within apps can be differentiated based on the contexts that trigger security-sensitive behaviors, i.e., the events and conditions that cause the security-sensitive behaviors to occur. In this work, we introduce AppContext, an approach of static program analysis that extracts the contexts of security-sensitive behaviors to assist app analysis in differentiating between malicious and benign behaviors. We implement a prototype of AppContext and evaluate AppContext on 202 malicious apps from various malware datasets, and 633 benign apps from the Google Play Store. AppContext correctly identifies 192 malicious apps with 87.7% precision and 95% recall. Our evaluation results suggest that the maliciousness of a security-sensitive behavior is more closely related to the intention of the behavior (reflected via contexts) than the type of the security-sensitive resources that the behavior accesses.

This paper addresses Problems 5,1,3.

PhD student Wei Yang presented this in May 2015.

[10] Xusheng Xiao, Nikolai Tillmann, Manuel Fahndrich, Jonathan de Halleux, Michal Moskal, and Tao Xie. "User-Aware Privacy Control via Extended Static-Information-Flow Analysis", Automated Software Engineering Journal, volume 22, issue 3, pages 333-366, 2015.

Abstract: Applications in mobile marketplaces may leak private user information without notification. Existing mobile platforms provide little information on how applications use private user data, making it difficult for experts to validate applications and for users to grant applications access to their private data. We propose a user-aware-privacy-control approach, which reveals how private information is used inside applications. We compute static information flows and classify them as safe/unsafe based on a tamper analysis that tracks whether private data is obscured before escaping through output channels. This flow information enables platforms to provide default settings that expose private data for only safe flows, thereby preserving privacy and minimizing decisions required from users. We build our approach into TouchDevelop, an application-creation environment that allows users to write scripts on mobile devices and install scripts published by other users. We evaluate our approach by studying 546 scripts published by 194 users, and the results show that our approach effectively reduces the need to make access-granting choices to only 10.1 % (54) of all scripts. We also conduct a user survey that involves 50 TouchDevelop users to assess the effectiveness and usability of our approach. The results show that 90 % of the users consider our approach useful in protecting their privacy, and 54 % prefer our approach over other privacy-control approaches.

This paper addresses Problems 5,1,3.

[11] S. Smith, R. Koppel, J. Blythe and V. Kothari, "Mismorphism: a Semiotic Model of Computer Security Circumvention", 9th International Symposium on Human Aspects of Information Security and Assurance, July 2015.

Shorter, revised version of [2] above.

Smith Presented this in July 2015

Addresses 5,1,2,3.

[12] Huoran Li, Xuanzhe Liu, Tao Xie, Kaigui Bian, Xuan Lu, Felix Xiaozhu Lin, Qiaozhu Mei, and Feng Feng, "Characterizing Smartphone Usage Patterns from Millions of Android Users", 2015 Internet Measurement Conference (IMC 2015), Tokyo, Japan, pp. 459-472, October 2015.

Abstract: A number of interesting and important questions in terms of app usage by Android users remain unanswered, such as why a user likes/dislikes an app, how an app becomes popular or eventually perishes, how a user selects apps to install and interacts with them, how frequently an app is used and how much traffic it generates, etc. This paper presents an empirical analysis of app usage behaviors collected from real world Android app users.

This paper addresses Problems 5, 3.

[13] H. Thimbleby and R. Koppel, "The Healthtech Declaration", Security and Privacy, IEEE, Volume 13, Issue 6, pp 82-84, Nov/Dec 2015.

Abstract: Healthcare technology is enmeshed with security and privacy via usability, performance, and cost-effectiveness issues. To address the problems that arise in such a multifaceted field, the Healthtech Declaration was initiated at the 2015 USENIX Summit on Information Technologies for Health.

This paper addresses Problems 5, 2.

[14] B. Korbar, J. Blythe, R. Koppel, V. Kothari, and S.W. Smith. "Validating an Agent-Based Model of Human Password Behavior." The AAAI-16 Workshop on Artificial Intelligence for Cyber Security (AICS). February 2016.

Abstract: Effective reasoning about the impact of security policy decisions requires understanding how human users actually behave, rather than assuming desirable but incorrect behavior. Simulation could help with this reasoning, but it requires building computational models of the relevant human behavior and validating that these models match what humans actually do. In this paper we describe our progress on building agent-based models of human behavior with passwords, and we demonstrate how these models reproduce phenomena shown in the empirical literature.

Kothari presented this paper in February 2016.

This publication addresses Problems 5,1,2,3.

[15] Benjamin Andow, Adwait Nadkarni, Blake Bassett, William Enck, and Tao Xie. A Study of Grayware on Google Play. In Proceedings of Workshop on Mobile Security Technologies (MoST 2016), San Jose, CA, May 2016.

Abstract: While there have been various studies identifying and classifying Android malware, there is limited discussion of the broader class of apps that fall in a gray area. Mobile grayware is distinct from PC grayware due to differences in operating system properties. Due to mobile grayware's subjective nature, it is difficult to identify mobile grayware via program analysis alone. Instead, we hypothesize enhancing analysis with text analytics can effectively reduce human effort when triaging grayware. In this paper, we design and implement heuristics for seven main categories of grayware. We then use these heuristics to simulate grayware triage on a large set of apps from Google Play. We then present the results of our empirical study, demonstrating a clear problem of grayware. In doing so, we show how even relatively simple heuristics can quickly triage apps that take advantage of users in an undesirable way.

This paper addresses Problems 5,1,3.

[16] Tao Xie and William Enck. Text Analytics for Security. In Proceedings of the Symposium and Bootcamp on the Science of Security (HotSoS 2016), Tutorial, Pittsburgh, PA, April 2016.

Abstract: Computing systems that make security decisions often fail to take into account human expectations. This failure occurs because human expectations are typically drawn from in textual sources (e.g., mobile application description and requirements documents) and are hard to extract and codify. Recently, researchers in security and software engineering have begun using text analytics to create initial models of human expectation. In this tutorial, we provide an introduction to popular techniques and tools of natural language processing (NLP) and text mining, and share our experiences in applying text analytics to security problems. We also highlight the current challenges of applying these techniques and tools for addressing security problems. We conclude the tutorial with discussion of future research directions.

This paper addresses Problems 5,1,3.

[17] Sihan Li, Xusheng Xiao, Blake Bassett, Tao Xie and Nikolai Tillmann. Measuring Code Behavioral Similarity for Programming and Software Engineering Education. In Proceedings of the 38th International Conference on Software Engineering (ICSE 2016), SEET, Austin, TX, May 2016.

Abstract: In recent years, online programming and software engineering education via information technology has gained a lot of popularity. Typically, popular courses often have hundreds or thousands of students but only a few course staff members. Tool automation is needed to maintain the quality of education. In this paper, we envision that the capability of quantifying behavioral similarity between programs is helpful for teaching and learning programming and software engineering, and propose three metrics that approximate the computation of behavioral similarity. Specifically, we leverage random testing and dynamic symbolic execution (DSE) to generate test inputs, and run programs on these test inputs to compute metric values of the behavioral similarity. We evaluate our metrics on three real-world data sets from the Pex4Fun platform (which so far has accumulated more than 1.7 million game-play interactions). The results show that our metrics provide highly accurate approximation to the behavioral similarity. We also demonstrate a number of practical applications of our metrics including hint generation, progress indication, and automatic grading.

This paper addresses Problems 5,1,3.

[18] R. Koppel, J. Blythe, V. Kothari, S.W. Smith. "Beliefs About Cybersecurity Rules and Passwords: A Comparison of Two Survey Samples of Cybersecurity Professionals Vs. Regular Users." SOUPS 2016 Security Fatigue Workshop. June 2016.

Abstract: In this paper we explore the differential perceptions of cybersecurity professionals and general users regarding access rules and passwords. We conducted a preliminary survey involving 28 participants: 15 cybersecurity professionals and 13 general users. We present our preliminary findings and explain how such survey data might be used to improve security in practice. We focus on user fatigue with access rules and passwords.

This paper addresses problems 5,2,3.

[19] Blythe, J., Koppel, R., Kothari, V., Smith, S. On Developing a Human Password Behavior Simulation to Improve Password Policy Design: Short Paper. Submitted for publication.

Abstract: We discuss our latest progress and planned efforts toward developing a model for simulating human users' password behaviors. This is a follow-up to our previous work in this space [1-3]. Our end goal is to deploy a tool that security practitioners may use to derive quantitative estimates of the efficacy of security policies and mechanisms, thereby enabling them to make better security decisions. We briefly review previous iterations of the model. We discuss related projects that can be used to inform our model, namely an Amazon Mechanical Turk experiment and surveys. We then detail our plans to extend the password model. These extensions include delving deeper into the aggregate security curve via Mechanical Turk experiment findings, capturing user decision-making processes given limited password security knowledge and an abundance of security advice, and modeling the spread of influence in password perceptions and behaviors.

This addresses Problems 5 and 3.

[20] Blythe, J., Koppel, R., Korbar, B., Kothari, V., Smith, S. Toward Security Assessment Tools: Accounting for the Human with Cognitive Behavioral Agent-Based Models. In preparation

Abstract: The efficacy of organizational security decisions and security policy decisions is often in large part determined by human responses to those decisions. We should therefore design, deploy, and use security assessment tools that incorporate models of human behavior that are sufficiently faithful to predict human responses to security decisions. Security practitioners equipped with such tools would better achieve their security objectives. In this paper, we explore the viability and utility of evaluating different security postures via cognitive behavioral agent-based modeling. We motivate our work, discuss our research agenda, review an example of password simulation and provide results that demonstrate how it might be used in practice. In particular our simulation predicts that in some cases a more stringent security policy will lead to reduced overall security, due to an increase in human coping strategies. We discuss promising directions for future work.

This paper addresses Problems 5, 1, 2, 3.

[21] Xia Zeng, Dengfeng Li, Wujie Zheng, Fan Xia, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. Automated Test Input Generation for Android: Are We Really There Yet in an Industrial Case? In Proceedings of the 24th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE 2016), Industrial Track, Seattle, WA, November 2016.

Abstract: Given the ever increasing number of research tools to automatically generate inputs to test Android applications (or simply apps), researchers recently asked the question "Are we there yet?" (in terms of the practicality of the tools). By conducting an empirical study of the various tools, the researchers found that Monkey (the most widely used tool of this category in industrial settings) outperformed all of the research tools in the study. In this paper, we present two significant extensions of that study. First, we conduct the first industrial case study of applying Monkey against WeChat, a popular messenger app with over 762 million monthly active users, and report the empirical findings on Monkey's limitations in an industrial setting. Second, we develop a new approach to address major limitations of Monkey and accomplish substantial code-coverage improvements over Monkey. We conclude the paper with empirical insights for future enhancements to both Monkey and our approach.

This paper addresses Problems 5,1,3.

[22] Pierre McCauley, Brandon Nsiah-Ababio, Joshua Reed, Faramola Isiaka and Tao Xie. Preliminary Analysis of Code Hunt Data Set from a Contest. In Proceedings of 2nd International Code Hunt Workshop on Educational Software Engineering (CHESE 2016), Seattle, WA, November 2016.

Abstract: Code Hunt (https://www.codehunt.com/) from Microsoft Research is a web-based serious gaming platform being popularly used for various programming contests. In this paper, we demonstrate preliminary statistical analysis of a Code Hunt data set that contains the programs written by students (only) worldwide during a contest over 48 hours. There are 259 users, 24 puzzles (organized into 4 sectors), and about 13,000 programs submitted by these users. Our analysis results can help improve the creation of puzzles in a future contest.

This paper addresses Problems 5,1,3.

Other Presentations

Co-Pi Jim Blythe presented the following three posters at the SoS Quarterly Meeting on July 27:

• Blythe, J., Kothari, V., Koppel, R., Smith, S. Modeling Human Security Behavior: Recent Results on Understanding Compliance (Poster). Presented at SoS Quarterly Meeting on July 27, 2016.

This poster discusses the value of modeling human behavior, our DASH-based approach to modeling human behavior, how we've gone about modeling network effects in password use, how we can use DASH to model resource contention and workarounds, and our current efforts.

This poster addresses Problems 5 and 3.

• Koppel, R., Blythe, J., Kothari, V., Smith, S. (2016). Beliefs about Cybersecurity Rules and Passwords: A Comparison of Two Survey Samples of Cybersecurity Professionals Versus Regular Users (Poster). Presented at SoS Quarterly Meeting on July 27, 2016.

This was a poster adaptation of our recent workshop paper [C]

This poster addresses Problems 5 and 3.

• Smith, S., Koppel, R., Blythe, J., Kothari, V., Reasons for Cybersecurity Circumvention: A Study and a Model (Poster). Presented at SoS Quarterly Meeting on July 27. 2016.

This was a poster adaption of our earlier work on Mismorphisms [2].

This poster addresses Problems 5 and 3.

PI Tao Xie presented the following poster at the SoS Quarterly Meeting on July 27:

• Wing Lam, Dengfeng Li, Wei Yang, Tao Xie. User-Centric Mobile Security Assessment (Poster). Presented at SoS Quarterly Meeting on July 27, 2016.

This poster discusses techniques to understand how users respond to security warnings reported by malware detection tools, and to facilitate user assistance, to understand how users respond to privacy statements accompanying apps, and to facilitate user assistance.

This poster addresses Problems 5 and 3.

ACCOMPLISHMENT HIGHLIGHTS

Via fieldwork in real-world enterprises, we have been identifying and cataloging types and causes of circumvention by well-intentioned users. We are using help desk logs, records security-related computer changes, analysis of user behavior in situ, and surveys--in addition to interviews and observations. We then began to build and validate models of usage and circumvention behavior, for individuals and then for populations within an enterprise--as well as developing some typologies of the deeper patterns and causes.

Via a Mechanical Turk process we are building a password simulation for measuring the security associated with a password composition policy, taking into account human circumventions such as writing down and reusing passwords. We've also taken more steps toward validation.

We have been developing questionnaires for both high-level computer security professionals and general users. The results will enable us to better understand computer security perceptions and behaviors. Moreover, they will allow us to produce more faithful models of human behavior.

We have built a platform on Mechanical Turk for conducting password security experiments. We are beginning to carry out these experiments.

We have begun collaboration with researchers at University of Pennsylvania who specialize in simulating and checking Markov chain models. The goal is to blend these Markov-based models with our DASH model to tackle security problems.

We are now working on implementing a version of DASH in Python. We are also working on implementing a new version of the password simulation built atop this Python version of DASH.