Securing Safety-Critical Machine Learning Algorithms - October 2019

PI(s), Co-PI(s), Researchers: Lujo Bauer, Matt Fredrikson (CMU), Mike Reiter (UNC)

HARD PROBLEM(S) ADDRESSED

This project addresses the following hard problems: developing security metrics and developing resilient architectures. Both problems are tackled in the context of deep neural networks, which are a particularly popular and performant type of machine learning algorithm. This project develops metrics that characterize the degree to which a neural-network-based classifier can be evaded through practically realizable, inconspicuous attacks. The project also develops architectures for neural networks that would make them robust to adversarial examples.

PUBLICATIONS

N/A this quarter

PUBLIC ACCOMPLISHMENT HIGHLIGHTS

We've made progress on understanding the weaknesses of ML algorithms another practical settings: as used by anti-virus programs for the detection of malware. This setting is interesting -- aside from it's practical relevance -- because previous techniques for creating evasion attacks usually assumed that the input belonged to a continuous domain, whereas malware binaries are drawn from a discrete domain. More specifically, previous evasion attacks, usually on image classifiers, would usually iteratively modify an input until it was misclassified, relying on the property of images that slightly changing any pixel of an image would still result in a valid image. For malware binaries, on the other hand, any small change is likely to result in malformed binary or a binary that no longer has the same functionality as the original one. Hence, previous techniques for evasion attacks don't carry over. We've developed techniques based on binary randomization (as used to harden binaries against return-oriented-programming attacks, for example) that demonstrate how an attacker could alter a malware binary so as to make it avoid detection by malware classifiers.

We have also made progress on leveraging explanations to prune neural network components that contribute to evasion vulnerability. Building on our experiments from the previous quarter, we have developed a model compression technique that results in a smaller model that performs similarly on in-distribution test samples to the original model, and shows promise in correctly classifying adversarial test examples. We are currently evaluating its performance in greater depth on the attack dataset developed by Bauer and Reiter previously.

COMMUNITY ENGAGEMENTS (If applicable)

Bauer and Fredrikson presented work that was part of this project at the annual CyLab partners conference in September, attended by representatives from 25+ companies and from various government agencies. Bauer presented work that was part of this project to Congressional staff on their visit to CMU in September. The work was additionally presented and discussed at a DOT&E workshop in July, and to ARO staff in October.

EDUCATIONAL ADVANCES (If applicable)

N/A this quarter