Toward Trustworthy Deep Learning in Security

In the security area, there has been an increasing tendency to apply deep learning, which is perceived as a black box method because of the lack of understanding of its internal functioning. Can we trust deep learning models when they achieve high test accuracy? Using a visual explanation method, we find that deep learning models used in security tasks can easily focus on semantically non-discriminative parts of input data even though they produce the right answers. Furthermore, when a model is re-trained without any change in the learning procedure (i.e., no change in training/validation data, initialization/optimization methods and hyperparameters), it can focus on significantly different parts of many samples while producing the same answers. For trustworthy deep learning in security, therefore, we argue that it is necessary to verify the classification criteria of deep learning models before deploying them, even though they successfully achieve high test accuracy.