The logs in a network are not bound to be perfect perpetually. The behavior of the network traffic is bound to deviate from the expected one sometimes and when that occurs, the traffic is said to be anomalous. Anomalous traffic can be problematic for various reasons, be it external attacks, or transfer of outdated data or even serving customers for networking companies. When the network size is at a large scale, this becomes an even bigger problem to tackle. The anomaly detection systems currently in place are either trained with aging datasets or are not able to handle large loads efficiently. Hence arises the need for a scalable solution which can provide security to a network by detecting anomalies in it and alerting with quick response when an anomaly occurs by learning from its past behavior. The paper offers an end-to-end solution for the introduction of an anomaly detection system using machine learning into an enterprise environment, right from the collection of logs to the generation of alerts, effectively. This is implemented with an infrastructure that includes Elasticsearch, Logstash and Kibana along with the added feature of Machine Learning.

Alerting, anomaly detection, machine learning, networks

[1] S. N. Hussain, N. R. Singha., “A Survey on Cyber Security Threats and their Solutions”, International Journal for Research in Applied Science and Engineering Technology, Vol. 8, Issue. 7, pp. 1141-1146, 2020.
[2] M. Zamani, “Machine Learning Techniques for Intrusion Detection”, arXiv:1312.2177, 2013.
[3] O. Kononenko, O. Baysal, R. Holmes and M. W. Godfrey, “Mining modern repositories with Elasticsearch”, In the Proceedings of the 2014 Conference on Mining Software Repositories, India, pp. 328–331, 2014.
[4] N. Shah, D. Willick and V. Mago, “A framework for social media data analytics using Elasticsearch and Kibana”, Wireless Networks, Vol. 24, Issue.8, pp. 1-9, 2018.
[5] Sharma, Chalsi and Maurya, Satish, “A Review: Importance of Cyber Security and its challenges to various domains”, International Journal of Technical Research & Science Special(Issue.3), pp. 46-54, 2020.
[6] X. Shu, K. Tian, A. Ciambrone and D. Yao, “Breaking the Target: An Analysis of Target Data Breach and Lessons Learned”, arXiv:1701.04940, 2017.
[7] F. Salo, M. Injadat, A. B. Nassif, A. Shami and A. Essex, “Data Mining Techniques in Intrusion Detection Systems: A Systematic Literature Review”, in IEEE Access, Vol. 6, pp. 56046-56058, 2018.
[8] S. D. Bhattacharjee, Y. Junsong, Z. Jiaqi, Y. Tan, “Context-Aware Graph-Based Analysis for Detecting Anomalous Activities”, In the Proceedings of the 2017 IEEE International Conference on Multimedia and Expo (ICME), China, pp. 1021-1026, 2017.
[9] L. Cheng, F. Liu and D.Yao, “Enterprise data breach: causes, challenges, prevention, and future directions”, WIREs: Data Mining and Knowledge Discovery, Vol. 7, Issue.5, 2017.
[10] N. Moustafa and J. Slay, “Creating Novel Features to Anomaly Network Detection Using DARPA-2009 Data set”, In the Proceedings of the 2015 14th European Conference on Cyber Warfare and Security ECCWS-2015, UK, pp. 204-212, 2015.
[11] P. P. Bavaskar, O. Kemker and A. Sinha, “A SURVEY ON: "LOG ANALYSIS WITH ELK STACK TOOL", International Journal of Research and Analytical Reviews (IJRAR), Vol. 6, Issue.4, pp. 965-968, 2019.
[12] S. J. Son and Y. Kwon, "Performance of ELK stack and commercial system in security log analysis", ”, In the Proceedings of the 2017 IEEE 13th Malaysia International Conference on Communications (MICC), Malaysia, pp. 187-190, 2017.