Web server normally produces log files. A weblog is a group of connected web pages that consists of a log or daily record of information, particular fields or views which is altered, every now and then, by owner of site, other websites or by website users. This is used to convert the unstructured data of web server log which will be coming from Apache flume into structured format using Pig. An enterprise weblog analysis system based on Hadoop architecture with Hadoop Distributed File System (HDFS), Hadoop MapReduce Software Framework and Pig Latin Language aids the business decision-making process of the system administrators and helps them to collect and identify the potential value which is hidden within such huge data generated by the websites. Such a weblog analysis includes the analysis of an Internet site’s entry log as well as provides information about the amount of visitors, days of week and rush hours, views, hits, very often accessed pages, application server traffic trends, performance reports at varying intervals and statistical reports which indicate the performance of program. Web log file is a log file created and stored by a web server automatically. Analyzing such web server access logs files will provide us various insights about website usage. Due to high usage of web, the log files are growing at much faster rate with increase in size. Processing this fast growing log files using relational database technology has been a challenging task these days. Hadoop runs the big data where a massive quantity of information is processed via cluster of commodity hardware. In this paper we present the methodology used in pre-processing of high volume web log files, studying the statics of website and learning the user behaviour using the architecture of Hadoop MapReduce framework, Hadoop Distributed File System, and HiveQL query language Pig. 

HDFS, Apache Flume, Pig , Hbase, web log server

[1] Babak Yadranjiaghdam, Nathan Pool, Nasseh Tabrizi, “A Survey on Real-time Big Data Analytics: Applications and Tool”, 2016 International Conference on Computational Science and Computational Intelligence
[2] P. Muthulakshmi1, S. Udhayapriya , “A Survey on Big Data Issues and Challenges”,International Journal of Computer Sciences and Engineering, Vol.-6, Issue-6, Jun 2018 E-ISSN: 2347-2693
[3] SayaleeNarkhede and TriptiBaraskar, “HMR Log Analyzer: Analyze Web Application Logs over HadoopMapReduce,” International Journal of UbiComp (IJU) vol.4, No.3, July 2013.
[4] Mirghani. A. Eltahir ; Anour F. A. Dafa-Alla,” Extracting knowlede from web server logs using web usage minning”, Published in: 2013 International Conference On Computing, Electrical And Electronic Engineering (Icceee)
[5] https://en.wikipedia.org/wiki/Apache_Hadoop
[6] Dr.S.Suguna, M.Vithya,J.I.ChristyEunaicy, “Big Data Analysis in E-commerce System Using HadoopMapReduce”in 2016 IEEE.
[7] G.S.Katkar, A.D.Kasliwal, “Use of Log Data for Predictive Analytics through Data Mining”, Current Trends in Technology and Science, ISSN: 2279-0535. Volume: 3, Issue: 3(Apr-May 2014).
[8] Savitha K, Vijaya M S, “Mining of web server logs in a distributed cluster using big data technologies”, International Journal of Advanced Computer Science and Applications, Vol.5, NO.1, 2014
[9] Mahendra Pratap Yadav ; Pankaj Kumar Keserwani ; Shefalika Ghosh Samaddar, “An Efficient Web Mining Algorithm for Web Log Analysis: E-Web Miner” 2012 1st International Conference on Recent Advances in Information Technology (RAIT)
[10] Xianjun Ni, “Design and Implementation of Web log Minning” International conference of computer engineering and technology 2009
[11] Apache-Hadoop,http://www.hadoop.apache.org