Visible to the public Hadoop-based Dark Web Threat Intelligence Analysis Framework

TitleHadoop-based Dark Web Threat Intelligence Analysis Framework
Publication TypeConference Paper
Year of Publication2019
AuthorsYang, Ying, Yu, Huanhuan, Yang, Lina, Yang, Ming, Chen, Lijuan, Zhu, Guichun, Wen, Liqiang
Conference Name2019 IEEE 3rd Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC)
Date PublishedOct. 2019
PublisherIEEE
ISBN Number 978-1-7281-0513-0
Keywordsanonymous messaging, anonymous TOR tool, anonymous user communication, crime messages, dark network criminal networks, dark network research, dark web, dark-net threat intelligence, data handling, data privacy, Distributed databases, Hadoop, Hadoop-based dark web threat intelligence analysis framework, HBase-based distributed database, Human Behavior, human factors, Internet, network services, pubcrawl, threat intelligence, threat intelligence information, Tor
Abstract

With the development of network services and people's privacy requirements continue to increase. On the basis of providing anonymous user communication, it is necessary to protect the anonymity of the server. At the same time, there are many threatening crime messages in the dark network. However, many scholars lack the ability or expertise to conduct research on dark-net threat intelligence. Therefore, this paper designs a framework based on Hadoop is hidden threat intelligence. The framework uses HDFS as the underlying storage system to build a HBase-based distributed database to store and manage threat intelligence information. According to the heterogeneous type of the forum, the web crawler is used to collect data through the anonymous TOR tool. The framework is used to identify the characteristics of key dark network criminal networks, which is the basis for the later dark network research.

URLhttps://ieeexplore.ieee.org/document/8984106
DOI10.1109/IMCEC46724.2019.8984106
Citation Keyyang_hadoop-based_2019