Visible to the public Challenges in Classifying Privacy Policies by Machine Learning with Word-based Features

TitleChallenges in Classifying Privacy Policies by Machine Learning with Word-based Features
Publication TypeConference Paper
Year of Publication2018
AuthorsFukushima, Keishiro, Nakamura, Toru, Ikeda, Daisuke, Kiyomoto, Shinsaku
Conference NameProceedings of the 2Nd International Conference on Cryptography, Security and Privacy
PublisherACM
Conference LocationNew York, NY, USA
ISBN Number978-1-4503-6361-7
KeywordsAnalysis, Bag-of-Words Model, composability, Metrics, Naive Bayes Classifiers, policy-based governance, Privacy Labels, pubcrawl, random forests, Resiliency, Security Policies Analysis, Support vector machines, TF-IDF
Abstract

In this paper, we discuss challenges when we try to automatically classify privacy policies using machine learning with words as the features. Since it is difficult for general public to understand privacy policies, it is necessary to support them to do that. To this end, the authors believe that machine learning is one of the promising ways because users can grasp the meaning of policies through outputs by a machine learning algorithm. Our final goal is to develop a system which automatically translates privacy policies into privacy labels [1]. Toward this goal, we classify sentences in privacy policies with category labels, using popular machine learning algorithms, such as a naive Bayes classifier.We choose these algorithms because we could use trained classifiers to evaluate keywords appropriate for privacy labels. Therefore, we adopt words as the features of those algorithms. Experimental results show about 85% accuracy. We think that much higher accuracy is necessary to achieve our final goal. By changing learning settings, we identified one reason of low accuracies such that privacy policies include many sentences which are not direct description of information about categories. It seems that such sentences are redundant but maybe they are essential in case of legal documents in order to prevent misinterpreting. Thus, it is important for machine learning algorithms to handle these redundant sentences appropriately.

URLhttp://doi.acm.org/10.1145/3199478.3199486
DOI10.1145/3199478.3199486
Citation Keyfukushima_challenges_2018