Biblio

We have been studying extension of the classical Software Reliability Engineering (SRE) methodology into the security space. We combine “classical” reliability modeling, when applied to reported vulnerabilities found under “normal” operational profile conditions, with safety oriented fault management processes. We illustrate with open source Fedora software.

Our initial results appear to indicate that generation of a repeatable automated test-strategy that would explicitly cover the “top 25” security problems may help considerably – eliminating perhaps as much as 50% of the field observable problems. However, genuine aleatoric and more process oriented incomplete analysis and design flaws remain. While we have made some progress in identifying focus areas, a number of questions remain, and we continue working on them.

The process of software development and evolution has proven difficult to improve. For example,  well documented security issues such as SQL injection (SQLi), after more than a decade, still top  most vulnerability lists. Quantitative security process and quality metrics are often subdued due to  lack of time and resources. Security problems are hard to quantify and even harder to predict or  relate to any process improvement activity.  The goal of this thesis is to assess usefulness of “classical” software reliability engineering (SRE)  models in the context of open source software security, the conditions under which they may be  useful, and the information that they can provide with respect to the security quality of a software  product.  We start with security problem reports for open source Fedora series of software releases.We  illustrate how one can learn from normal operational profile about the non-operational processes  related to security problems. One aspect is classification of security problems based on the human  traits that contribute to the injection of problems into code, whether due to poor practices or limited  knowledge (epistemic errors), or due to random accidental events (aleatoric errors). Knowing the  distribution aids in development of an attack profile. In the case of Fedora, the distribution of  security problems found post-release was consistent across four different releases of the software.  The security problem discovery rate appears to be roughly constant but much lower than the initial  non-security problem discovery rate. Previous work has shown that non-operational testing can help  accelerate and focus the problem discovery rate and that it can be successfully modeled.We find  that some classical reliability models can be used with success to estimate the residual number of  security problems, and through that provide a measure of the security characteristics of the software.  We propose an agile software testing process that combines operational and non-operational (or  attack related) testing with the intent of finding more security problems faster. 

In our previous work we showed that for Fedora, under normal operational conditions, security problem discovery appears to be a random process. While in the case of Fedora, and a number of other open source products, classical reliability models can be adapted to estimate the number of residual security problems under “normal” operational usage (not attacks), the predictive ability of the model is lower for security faults due to the rarity of security events and because there appears to be no real security reliability growth. The ratio of security to non-security faults is an indicator that the process needs improving, but it also may be leveraged to assess vulnerability profile of a release and possibly guide testing of its next version. We manually analyzed randomly sampled problems for four different versions of Fedora and classified them into security vulnerability categories. We also analyzed the distribution of these problems over the software’s lifespan and we found that they exhibit a symmetry which can perhaps be used in estimating the number of residual security problems in the software. Based on our work, we believe that an approach to vulnerability elimination based on a combination of “classical” and some non-operational “bounded” high-assurance testing along the lines discussed in may yield good vulnerability elimination results, as well as a way of estimating vulnerability level of a release. Classical SRE methods, metrics and models can be used to track both non-security and security problem detection under normal operational profile. We can then model the reliability growth, if any, and estimate the number of residual faults by estimating the lower and upper bounds on the total number of faults of a certain type. In production, there may be a simpler alternative. Just count the vulnerabilities and project over the next period assuming constant vulnerability discovery rate. In testing phase, to accelerate the process, one might leverage collected vulnerability information to generate non-operational test-cases aimed at vulnerability categories. The observed distributions of security problems reported under normal “operational” usage appear to support the above approach – i.e., what is learned say in the first x weeks can them be leveraged in selecting test cases in the next stage. Similarly, what is learned about a product Y weeks after its release may be very indicative of its vulnerability profile for the rest of its life given the assumption of constant vulnerability discovery rate.