| Title | Performance Evaluation of Intrusion Detection Streaming Transactions Using Apache Kafka and Spark Streaming |
| Publication Type | Conference Paper |
| Year of Publication | 2019 |
| Authors | Tun, May Thet, Nyaung, Dim En, Phyu, Myat Pwint |
| Conference Name | 2019 International Conference on Advanced Information Technologies (ICAIT) |
| Keywords | apache kafka, Apache Spark Streaming, Big Data, Big Data analytics, complex attacks, composability, cybersecurity intrusion detection, Data analysis, data mining, decision making, extremely competitive financial market, fault tolerant computing, Government, high-level protection, Internet, Intrusion detection, intrusion detection streaming transactions, intrusion detection system, intrusion tolerance, machine learning, massive Internet-based services, network traffic, parallel processing, performance evaluation, Processing time, pubcrawl, real-time decision making, Real-time Systems, Resiliency, security of data, short period time, Spark Streaming, Sparks, stream data, telecommunication traffic |
| Abstract | In the information era, the size of network traffic is complex because of massive Internet-based services and rapid amounts of data. The more network traffic has enhanced, the more cyberattacks have dramatically increased. Therefore, cybersecurity intrusion detection has been a challenge in the current research area in recent years. The Intrusion detection system requires high-level protection and detects modern and complex attacks with more accuracy. Nowadays, big data analytics is the main key to solve marketing, security and privacy in an extremely competitive financial market and government. If a huge amount of stream data flows within a short period time, it is difficult to analyze real-time decision making. Performance analysis is extremely important for administrators and developers to avoid bottlenecks. The paper aims to reduce time-consuming by using Apache Kafka and Spark Streaming. Experiments on the UNSWNB-15 dataset indicate that the integration of Apache Kafka and Spark Streaming can perform better in terms of processing time and fault-tolerance on the huge amount of data. According to the results, the fault tolerance can be provided by the multiple brokers of Kafka and parallel recovery of Spark Streaming. And then, the multiple partitions of Apache Kafka increase the processing time in the integration of Apache Kafka and Spark Streaming. |
| DOI | 10.1109/AITC.2019.8920960 |
| Citation Key | tun_performance_2019 |
Performance Evaluation of Intrusion Detection Streaming Transactions Using Apache Kafka and Spark Streaming