Detecting Malicious Java Code
Using Virtual Machine Auditing

Sunil Soman, Chandra Krintz, and Giovanni Vigna

Abstract:

The Java language and its execution environment, the Java Virtual Machine (JVM), has evolved from a technology that supports active web pages into an environment for the development and execution of large-scale, network-based applications. Java provides extensive support for authentication and access control but it lacks support for intrusion detection.

Existing operating system auditing facilities and host-based intrusion detection systems operate at the process level, with the assumption that one application is mapped onto one process. However, in many cases, multiple Java-based applications are executed as threads within a single JVM process. This makes it difficult to analyze the behavior of Java applications using the corresponding OS-level audit trail. In addition, the malicious actions of a single Java application may trigger a response that disables an entire execution environment. To overcome this limitation, we developed an auditing facility for the Java Virtual Machine and an intrusion detection tool that uses audit data generated by this facility to detect attacks by malicious Java code. This paper describes the JVM auditing mechanisms, the intrusion detection tool, and the quantitative evaluation of their performance.