Antiforensic techniques deployed by custom developed malware in evading anti-virus detection

Abstract

Both malware and antivirus detection tools advance in their capabilities. Malware aim is to evade the detection while antivirus is to detect the malware. Over time, the detection techniques evolved from simple static signature matching over antiheuristic analysis to machine learning assisted algorithms. This thesis describes several layers of anti-virus evasion deployed by the malware and conducts the analysis of the evasion success rate. The scientific contribution of this research is in the following techniques the malware used -- the new algorithm for identifying the Windows operating system functions, a new custom developed obfuscation and de-obfuscation routine and the usage of USB and sound devices enumeration in the anti-heuristic detection. The new PE mutation engine facilitates the malware static signature variation. In the next stage of the assessment, anti-virus engines then test the malware evasion capabilities. The locally installed antivirus applications and the two multi-scanner online engines inspect the submitted malware samples. The thesis examines the results and discusses the strengths and weaknesses of each evasion technique.