Biblio

Machine Learning has revolutionized many fields of computer science. Reinforcement Learning (RL), in particular, stands out as a solution to sequential decision making problems. With the growing complexity of computer networks in the face of new emerging technologies, such as the Internet of Things and the growing complexity of threat vectors, there is a dire need for autonomous network systems. RL is a viable solution for achieving this autonomy. Software-defined Networking (SDN) provides a global network view and programmability of network behaviour, which can be employed for security management. Previous works in RL-based threat mitigation have mostly focused on very specific problems, mostly non-sequential, with ad-hoc solutions. In this paper, we propose ATMoS, a general framework designed to facilitate the rapid design of RL applications for network security management using SDN. We evaluate our framework for implementing RL applications for threat mitigation, by showcasing the use of ATMoS with a Neural Fitted Q-learning agent to mitigate an Advanced Persistent Threat. We present the RL model's convergence results showing the feasibility of our solution for active threat mitigation.

Software Defined Networking (SDN) is an emerging paradigm that changes the way networks are managed by separating the control plane from data plane and making networks programmable. The separation brings about flexibility, automation, orchestration and offers savings in both capital and operational expenditure. Despite all the advantages offered by SDN it introduces new threats that did not exist before or were harder to exploit in traditional networks, making network penetration potentially easier. One of the key threat to SDN is the authentication and authorisation of network applications that control network behaviour (unlike the traditional network where network devices like routers and switches are autonomous and run proprietary software and protocols to control the network). This paper proposes a mechanism that helps the control layer authenticate network applications and set authorisation permissions that constrict manipulation of network resources.

Distributed attacks originating from botnet-infected machines (bots) such as large-scale malware propagation campaigns orchestrated via spam emails can quickly affect other network infrastructures. As these attacks are made successful only by the fact that hundreds of infected machines engage in them collectively, their damage can be avoided if machines infected with a common botnet can be detected early rather than after an attack is launched. Prior studies have suggested that outgoing bot attacks are often preceded by other ``tell-tale'' malicious behaviour, such as communication with botnet controllers (C&C servers) that command botnets to carry out attacks. We postulate that observing similar behaviour occuring in a synchronised manner across multiple machines is an early indicator of a widespread infection of a single botnet, leading potentially to a large-scale, distributed attack. Intuitively, if we can detect such synchronised behaviour early enough on a few machines in the network, we can quickly contain the threat before an attack does any serious damage. In this work we present a measurement-driven analysis to validate this intuition. We empirically analyse the various stages of malicious behaviour that are observed in real botnet traffic, and carry out the first systematic study of the network behaviour that typically precedes outgoing bot attacks and is synchronised across multiple infected machines. We then implement as a proof-of-concept a set of analysers that monitor synchronisation in botnet communication to generate early infection and attack alerts. We show that with this approach, we can quickly detect nearly 80% of real-world spamming and port scanning attacks, and even demonstrate a novel capability of preventing these attacks altogether by predicting them before they are launched.