Edinburgh Napier University

  Intrusion detection has emerged as an important approach to network, host and application security. Network security includes analysing network packet payload and other inert network packet profiles for intrusive trends; whereas, host security may employ system logs for intrusion detection. In this paper, we contribute to the research community by tackling application security and attempt to detect intrusion via differentiating normal and abnormal application behaviour. A method for anomaly intrusion detection for applications is proposed based on deterministic system call traces derived from a monitored target application's dynamic link libraries (DLLs). We isolate associated DLLs of a monitored target application; log system call traces of the application in real time and use heuristic method to detect intrusion before the application is fully compromised. Our investigative research experiment methodology and set-up are reported, alongside our experimental procedure and results that show our research effort is effective and efficient, and can be used in practice to monitor a target application in real time.