Malware Analysis; Frameworks and Tools for Automated Dynamic Analysis of Malware

Abstract

This thesis describes different automated dynamic malware analysis solutions and their underlying technologies. A good deal of prerequisite concepts has to be covered before the actual solutions are studied. Some of these concepts are virtualization, sandboxing, hooking and the Windows Operating System. It is important for us to understand how malware analysis works before we start to review the different automated solutions. For us to understand how malware analysis is done on a Windows system the Windows architecture has to be studied. The most important parts of the Windows architecture are covered in Chapter 2. Another concept we will stumble upon when dealing with malware analysis is virtualization. Virtualization is often used in the context of automated solutions since it is easy to revert back to saved states of the system. A typical approach is to have a clean state where the system is new; infecting this state with malware will not matter since we can revert back to the clean state at any moment. Closely related to virtualization is sandboxing. Sandboxing utilizes virtualization technology to build a secure environment where malicious code can be executed. When executing the malicious code we have to use some sort of technology to monitor its behavior. When talking about such technology we enter the darker side of malware analysis and find a concept called hooking. Hooking is the number one technique used to monitor the behavior of malware when run live in a virtual environment. It is also used by malicious coders to create rootkits and other nasty malware. After the most important prerequisite topics we will start to look at actual automated solutions that analyze malware dynamically. These solutions will be studied to the degree we can find documentation about them; no reverse engineering will be done. A lot of the tested solutions are commercial but there are also some open source projects out there. After an introduction to these solutions a couple of tests will be conducted. Two live malware samples which utilize new malicious technology will be tested to see if these solutions hold up. At the end of this thesis the automated solutions will be evaluated on the grounds of our test findings and their different underlying technology. We will also discuss how to best utilize one or more of these solutions as well as ideas for possible further work.