Malwares are malicious software, designed to damage computer systems without the knowledge of the owner. Rootkit is also malicious software which hides the existence of certain processes or programs from normal methods of detection and enables continued privileged access to a computer. Now a day the impact of malware and rootkit is getting worst. Their detection is difficult because malicious program may be able to subvert the software that is intended to find it. Detection methods uses an alternative and trusted operating system, signature scanning behavioral-based methods, difference scanning, and memory dump analysis etc. Malware and rootkit detectors are the primary tools in defense against malicious programs. The quality of such a detector is determined by the techniques used by it. There are mainly two types of techniques to detect the malwares, signature based and anomaly based techniques. Signature-based detection is a malware detection approach that identifies a malware instance by the presence of at least one byte code pattern present in a database of signatures from known malicious programs. If a program contains a pattern that already exists within the database, it is deemed. In anomaly based detection malwares are classified according to some heuristics and rules. This survey study about signature based and anomaly based malware detection and list their strengths and limitations. It also compares techniques and helps to choose a desirable technique for secure system.

Anomaly based malware, rootkit, malware detection malcode, malicious code, malicious software, signature-based, behavior based

[1]https://www.cert.gov.uk/wpcontent/uploads/2014/08/An-introduction-to malware.pdf
[2]http://www.ukessays.com/essays/computer-science/the-introduction-to-malicious-software-computer-science-essay.php
[3]http://en.wikipedia.org/wiki/Computer_virus
[4]http://en.wikipedia.org/wiki/Computer_worm
[5]http://en.wikipedia.org/wiki/Trojan_horse_(computing)
[6]” Survey on Malware Detection Methods” Vinod P. Department of Computer Engineering, Malaviya National Institute of Technology, Jaipur, Rajasthan
[7]“A Survey of Malware Detection Techniques”NwokediIdika,AdityaPMathur.Department of Computer Science Purdue University, West Lafayette, IN 47907.
[8]” A Survey on Techniques in Detection and Analyzing Malware Executables” Kirti Mathur M.Tech. Scholar, Department of CSE Rajasthan Technical University, India.
[9]“A Specification-based Intrusion Detection System for AODV” Chin-Yang Tseng, Poornima Balasubramanyam, Calvin Ko,Rattapon Limprasittiporn,Jeff Rowe,Karl Levitt,Computer Security Laboratory University of California, Davis.
[10]http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138.7174
[11] Greoigre Jacob,Herve Debar,Eric Fillol,”Behavioral detection of malware:from a survey towards an established taxonomy”,Springer-Verlag France 2008