Gang of Eight: A Defect Taxonomy for Infrastructure as Code Scripts

Defects in infrastructure as code (IaC) scripts can have serious
consequences, for example, creating large-scale system outages. A
taxonomy of IaC defects can be useful for understanding the nature
of defects, and identifying activities needed to fix and prevent
defects in IaC scripts. The goal of this paper is to help practitioners
improve the quality of infrastructure as code (IaC) scripts by developing
a defect taxonomy for IaC scripts through qualitative analysis.
We develop a taxonomy of IaC defects by applying qualitative analysis
on 1,448 defect-related commits collected from open source
software (OSS) repositories of the Openstack organization. We conduct
a survey with 66 practitioners to assess if they agree with the
identified defect categories included in our taxonomy. We quantify
the frequency of identified defect categories by analyzing 80,425
commits collected from 291 OSS repositories spanning across 2005
to 2019.

Our defect taxonomy for IaC consists of eight categories, including
a category specific to IaC called idempotency (i.e., defects that
lead to incorrect system provisioning when the same IaC script is
executed multiple times). We observe the surveyed 66 practitioners
to agree most with idempotency. The most frequent defect category
is configuration data i.e., providing erroneous configuration data
in IaC scripts. Our taxonomy and the quantified frequency of the
defect categories may help in advancing the science of IaC script
quality.