Biblio

The process of software development and evolution has proven difficult to improve. For example,  well documented security issues such as SQL injection (SQLi), after more than a decade, still top  most vulnerability lists. Quantitative security process and quality metrics are often subdued due to  lack of time and resources. Security problems are hard to quantify and even harder to predict or  relate to any process improvement activity.  The goal of this thesis is to assess usefulness of “classical” software reliability engineering (SRE)  models in the context of open source software security, the conditions under which they may be  useful, and the information that they can provide with respect to the security quality of a software  product.  We start with security problem reports for open source Fedora series of software releases.We  illustrate how one can learn from normal operational profile about the non-operational processes  related to security problems. One aspect is classification of security problems based on the human  traits that contribute to the injection of problems into code, whether due to poor practices or limited  knowledge (epistemic errors), or due to random accidental events (aleatoric errors). Knowing the  distribution aids in development of an attack profile. In the case of Fedora, the distribution of  security problems found post-release was consistent across four different releases of the software.  The security problem discovery rate appears to be roughly constant but much lower than the initial  non-security problem discovery rate. Previous work has shown that non-operational testing can help  accelerate and focus the problem discovery rate and that it can be successfully modeled.We find  that some classical reliability models can be used with success to estimate the residual number of  security problems, and through that provide a measure of the security characteristics of the software.  We propose an agile software testing process that combines operational and non-operational (or  attack related) testing with the intent of finding more security problems faster. 

We have been studying extension of the classical Software Reliability Engineering (SRE) methodology into the security space. We combine “classical” reliability modeling, when applied to reported vulnerabilities found under “normal” operational profile conditions, with safety oriented fault management processes. We illustrate with open source Fedora software.

Our initial results appear to indicate that generation of a repeatable automated test-strategy that would explicitly cover the “top 25” security problems may help considerably – eliminating perhaps as much as 50% of the field observable problems. However, genuine aleatoric and more process oriented incomplete analysis and design flaws remain. While we have made some progress in identifying focus areas, a number of questions remain, and we continue working on them.

Can software reliability models be used to assess software security? One of the issues is that security problems are relatively rare under “normal” operational profiles, while “classical” reliability models may not be suitable for use in attack conditions. We investigated a range of Fedora open source software security problems to see if some of the basic assumptions behind software reliability growth models hold for discovery of security problems in non-attack situations. We find that in some cases, under “normal” operational use, security problem detection process may be described as a Poisson process. In those cases, we can use appropriate classical software reliability growth models to assess “security reliability” of that software in non-attack situations.We analyzed security problem discovery rate for RedHat Fedora. We find that security problems are relatively rare, their rate of discovery appears to be relatively constant under “normal” (non-attack) conditions. Discovery process often appears to satisfy Poisson assumption opening doors to use of classical reliability models. We illustrated using Yamada S-shaped model fit to v15 that in some cases such models may be effective in predicting the number of remaining security problems, and thus may offer a way of assessing security “quality” of the software product (although not necessarily its behavior under an attack).