Detection of Hidden Software Functionality

Abstract

Downloading software from unknown sources constitutes a great risk. Studies have described file-sharing networks where the probability of downloading infected files is as high as 70% [1] under certain circumstances. This work presents theory on malicious software with emphasize on code turning computers into bots and thereby, possibly botnets. It is observed that malware authors start using more advanced techniques to deceive owners of compromised computers. To evade detection, stealth techniques known from rootkits are more and more commonly adapted. Rootkit technology is therefore studied to be able to determine how bots, and other forms malicious software, can be hidden from both automated anti-virus detection mechanisms and human inspections of computers. The mechanisms used to evade detection by traditional anti-virus tools are in many cases effective. Dynamic behavioural analysis of software during installation is therefore suggested as a strategy to supplement the traditional tools. Several detection strategies are presented, which can be used to determine the behaviour of software during installation. This knowledge is used to design a laboratory environment capable of detecting the mentioned categories of malicious code. An implementation of the laboratory is provided, and experiments are performed to determine the usefulness of the setup. The software used to set up the laboratory environment are all distributed free of license cost. An evaluation is made and improvements to the system are proposed. The value of behavioural analysis has been demonstrated, and the functionality of the laboratory environment has proved to extremely useful. Advanced users will find the functionality of the laboratory setup powerful. However, future work has to be done to automate the behavioural detection processes so the public can benefit from this work.