Abstract

It is discussed that malware authors are moving towards more advanced techniques to bypass detection. Static methods of analysis, while useful, are limited by obfuscated or polymorphic malware. It has thus become necessary for dynamic malware analysis, or the real-time execution of malicious software and its observation of behaviour. There are two main approaches in this paradigm: system call tracing and API monitoring. These techniques are granular in providing visibility of malware behaviour by logging interactions with the operating system and the core services. This review paper gives a comprehensive analysis of these techniques initially by considering their principles, tools, effectiveness, and limitations. Additionally, it examines recent advancements in the hybrid analysis frameworks that combine both techniques to enhance the detection accuracy. This paper seeks to provide a consolidated reference in terms of real-world case studies, comparative tables, statistical graphs, and insights from more than twenty scholarly sources to those researchers and practitioners who seek to enhance their malware detection capabilities. It also defines future directions of integration to make dynamic malware analysis more robust and scalable, as well as using machine learning, cloud-based analysis, and standardized benchmarks.