An Assessment of Security Problems in Open Source Software

An Assessment of Security Problems in Open Source Software: Improving software security through changes in software design and development processes appears to be a very hard problem. For example, well documented security issues such as Structured Query Language injection, after more than a decade, still tops most vulnerability lists. Security priority is often subdued due to constraints such as time-to-market and resources. Furthermore, security process outcomes are hard to quantify and even harder to predict or relate to process improvement activities. In part this is because of the nature of the security faults - they are in statistical terms "rare" and often very complex compared to "regular" non-security faults. In part it is the irregular and unpredictable nature of the security threats and attacks that puts the software under attack into states it was not designed for and subjects it to what would be considered "nonoperational" use. In many cases it is the human component of the system that fails - for example, due to phishing or due to incorrect use of a software product. On the other hand, we have decades of experience developing reliable software (admittedly subject to similar resource, cost and time constraints). The central question of interest in this thesis is to what extent can we leverage some of the software reliability engineering (SRE) models, processes, and metrics that work in the "classical" operational space to develop predictive software security engineering assessment and development elements. Specific objectives are a) to investigate use of (possibly modified) SRE practices to characterize security properties of software, and b) assess how software design and development processes could be enhanced to avoid, eliminate and tolerate security problems and attacks.We are particularly interested in open source software security, the conditions under which SRE practices may be useful, and the information that this can provide about the security quality of a software product. We examined public information about security problem reports for open source Fedora and RHEL series of software releases, Chromium project and Android project. The data that we analyzed was primarily about security problems reported from post-release in-the-field use of the products. What can we learn about the non-operational processes (and possible threats) related to security problems? One aspect is classification of security problems based on the traits that contribute to the injection of problems into code, whether due to poor practices or limited knowledge (epistemic errors), or due to random accidental events (aleatoric errors). Knowing the distribution can help understand attack space and help improve development processes and testing of the next version. For example, in the case of Fedora, the distribution of security problems found post-release was consistent across two different releases of the software. The security problem discovery rate appears to be roughly constant but much lower (ten to a hundred times lower) than the initial non-security problem discovery rate. Similarly, in the case of RHEL, the distribution of security problems found post-release was consistent and the number of security problems kept decreasing across six different releases of the software. The security problem discovery rate appears to be roughly constant but again much lower than the initial non-security problem discovery rate. In the case of Chromium, the number of discovered security problems is orders of magnitude higher than for other products, except that does not appear to translate into a higher incidence of field breaches. One reason could be Chromium "bounty" for problem discovery. We find that some classical reliability models can be used as one of tools to estimate the residual number of security problems in both the current release and in the future releases of the software, and through that provide a measure of the security characteristics of the software. For example, to assess whether, under given usage conditions, security problem discovery rate is increasing or decreasing - and what that may mean. Based on our findings, we discuss an agile software testing process that combines operational and non-operational (or attack related) testing with the intent of finding and eliminating more security problems earlier in the software development process. The knowledge of vulnerable components from architectural view and the frequency of vulnerabilities in each of the components helps in prioritizing security test resources.