Biblio

In the information era, the size of network traffic is complex because of massive Internet-based services and rapid amounts of data. The more network traffic has enhanced, the more cyberattacks have dramatically increased. Therefore, cybersecurity intrusion detection has been a challenge in the current research area in recent years. The Intrusion detection system requires high-level protection and detects modern and complex attacks with more accuracy. Nowadays, big data analytics is the main key to solve marketing, security and privacy in an extremely competitive financial market and government. If a huge amount of stream data flows within a short period time, it is difficult to analyze real-time decision making. Performance analysis is extremely important for administrators and developers to avoid bottlenecks. The paper aims to reduce time-consuming by using Apache Kafka and Spark Streaming. Experiments on the UNSWNB-15 dataset indicate that the integration of Apache Kafka and Spark Streaming can perform better in terms of processing time and fault-tolerance on the huge amount of data. According to the results, the fault tolerance can be provided by the multiple brokers of Kafka and parallel recovery of Spark Streaming. And then, the multiple partitions of Apache Kafka increase the processing time in the integration of Apache Kafka and Spark Streaming.

Internet of Things (IoT) Devices are monitoring and controlling systems that interact with the physical world by collecting, processing and transmitting data using the internet. IoT devices include home automation systems, smart grid, transportation systems, medical devices, building controls, manufacturing and industrial control systems. With the increase in deployment of IoT devices, there will be a corresponding increase in the amount of data generated by these devices, therefore, resulting in the need of large scale data processing systems to process and extract information for efficient and impactful decision making that will improve quality of living.