Conventional simulation techniques generate massive amounts of data that are analyzed using various applications. These simulations produce petabytes of data that strains the I/O and storage subsystem. To overcome the high latency in I/O operations, data is analyzed as it is generated, in-situ. This can be successfully achieved by enabling analysis techniques on the same HPC machine that is producing simulation by using the same hardware resources or on a separate analysis machine. In this research paper, we discuss state of the art techniques in this domain and support our conclusion by comparing pros and cons of each approach.

Big Data, Service Oriented Approach, Big Data paradigm

[1] Sergey V. Kovalchuk1, Artem V. Zakharchuk1, Jiaqi Liao1, Sergey V. Ivanov1, Alexander V. Boukhanovsky “A Technology for BigData Analysis Task Description using Domain-Specific Languages “
[2] The SAS versus R Debate, http://insidebigdata.com/2014/03/01/sas-versus-r/, March 1, 2014.
[3] Heba Aly, Mohammed Elmogy and Shereif Barakat , “Big Data on Internet of Things: Applications, Architecture, Technologies, Techniques, and Future Directions”, Nov 2015, Vol 4 No. 06, ISSN : 2319-7323
[4] Understanding BigData Processing and Analytics, http://www.developer.com/db/understanding-big-data-processing-and-analytics.html, September 9, 2013
[5] Scott Klasky at. al., “In situ data processing for extreme scale computing”
[6] Marzia Rivia,*, Luigi Caloria, Giuseppa Muscianisia, Vladimir Slavnicb, “In-situ Visualization: State-of-the-art and Some Use Cases” 1ORNL, 2 U.T. Knoxville, 3LBNL, 4Georgia Tech, 5Sandia Labs, 6 Rutgers, 7NREL, 8Kitware, 9UCSD, 10PPPL, 11UC Irvine, 12U. Utah, 13 Cal. Tech, 14Auburn University, 15NCSU
[7] The Scalable Data Management, Analysis and Visualization (SDAV) Institute, http://sdav-scidac.org/, SciDAC PI meeting 2015
[8] Khanh Nguyen Kai Wang Yingyi Bu Lu Fang Jianfei Hu Guoqing Xu, “FACADE: A Compiler and Runtime for (Almost) Object-Bounded Big Data Applications “ University of California, Irvine
[9] Kwan- Liu Ma, Chaoli Wang, Hongfeng Yu, Anna Tikhonova, “In-Situ Processing and Visualization for Ultrascale Simulations” Department of Computer Science, University of California at Davis, One Shields Avenue, Davis, CA 95616 SciDAC Institute for Ultrascale Visualization (IUSV)
[10] 2015 SAS vs. R Survey Results, http://www.burtchworks.com/2015/05/21/2015-sas-vs-r-survey-results/, May 21, 2015
[11] David Loshin, “ Big Data Analytics “,Morgan Kaufmann Publishers In, ISBN: 9780124186644