The 2008 Science of Security Workshop (see report here) explored the topic of science from a number of perspectives including the following:
"The methodological sense, of science as a way to conduct research by following the scientific method of forming hypotheses and carrying out experiments. For certain areas of computer security, experiments seem useful and the community will benefit from better experimental infrastructure, data sets, and methods. For other areas, it seems difficult to do meaningful experiments without developing a way to model a sophisticated and creative adversary."
The workshop summary concerning experimentation concluded:
"The main challenge in doing meaningful experiments in computer security is the need to model an adversary. Two main directions for work are suggested: finding ways to improve our adversary models, and finding ways to design more reproducible experiments that are not so dependent on accurate models of adversary behavior.
Improving Adversary Models. One suggested approach to improving adversary models for experiments is to systematize current knowledge about real adversaries and use that as a basis for developing experimental adversary models. Perhaps such an approach can be used to develop a canonical attacker model that could be used in a wide class of experiments, rather than relying on ad hoc models created by individual experimenters. In general, however, it seems extraordinarily difficult to evaluate new security mechanisms using experiments; there is little cause for optimism that a useful model of a creative adversary attacking a new design can be developed.
Design for Reproducibility. One weakness in empirical security research is that few experiments are ever reproduced. Some of this results from the lack of incentives to academic researchers to perform this type of work. One suggestion is to have students reproduce published experimental work in early graduate courses. A more fundamental problem is that many security experiments are not designed in a way that makes them readily reproducible. Better sharing of code and data sets is also important."
This forum topic solicits discussion of the general topic of experimentation as a component of a science of security. A more focused discussion of adversary models can be found
here.
The Role of Experimentation in Security Science
