Today, software is making more automated decisions with societal impact. For example, software determines who gets a loan or gets hired, computes risk-assessment scores that help decide who goes to jail and who is set free, and aids in diagnosing and treating medical patients. The increased role of software in such decisions makes software fairness a critical property. As more societal functions operate in cyberspace, the importance of software fairness increases. This project evaluates the feasibility of using software testing technology to identify behaviors whose outputs are more favorable for certain inputs. Using testing in such a manner is aimed at capturing causal relationships between characteristics of the software inputs and the software behavior. The approach is novel compared to typical machine learning classification techniques that analyze data but do not test the behavior of application software. The central idea is to identify causal relationships between software inputs and the way the software behaves, e.g., its outputs. Software testing enables conducting causal experiments consisting of running the software with nearly identical inputs that vary only in a key characteristic under test. Variations in that characteristic that affect behavior provide evidence of a causal relationship. Measuring such causal relationships requires test suites that focus on small variability in a key set of characteristics under test, while existing testing techniques focus on large variability that leads to greater coverage. As a result, existing techniques are ill-suited for measuring causal relationships and new technology is necessary. The result is the ability to test software for new properties for which no testing procedures existed.
EAGER: Exploring the Feasibility of Software Testing Techniques to Evaluate Fairness Algorithms in Software Systems