Design of an Anomaly-based Threat Detection & Explication System

Robert Luh, Sebastian Schrittwieser, Stefan Marschalek, Helge Janicke


Current signature-based malware detection systems are heavily reliant on fixed patterns that struggle with unknown or evasive applications, while behavior-based solutions usually leave most of the interpretative work to a human analyst. In this paper, we propose a system able to explain anomalous behavior within a user session by considering anomalies identified through their deviation from a set of baseline process graphs. To minimize computational requirements we adapt star structures, a bipartite representation used to approximate the edit distance between two graphs. Baseline templates are generated automatically and adapt to the nature of the respective process. We prototypically implement smart anomaly explication through a number of competency questions derived and evaluated using the decision tree algorithm. The determined key factors are ultimately mapped to a dedicated APT attack stage ontology that considers actions, actors, as well as target assets.


  1. Anagnostopoulos, T., Anagnostopoulos, C., and Hadjiefthymiades, S. (2005). Enabling attack behavior prediction in ubiquitous environments. In Int. Conference on Pervasive Services, pages 425-428. IEEE.
  2. Dolgikh, A., Nykodym, T., Skormin, V., and Birnbaum, Z. (2012). Using behavioral modeling and customized normalcy profiles as protection against targeted cyberattacks. In Computer Network Security, pages 191- 202. Springer.
  3. Dornhackl, H., Kadletz, K., Luh, R., and Tavolato, P. (2014). Malicious behavior patterns. In 2014 IEEE 8th Intl. Symposium on Service Oriented System Engineering (SOSE), pages 384-389. IEEE.
  4. Hu, X., Chiueh, T.-c., and Shin, K. G. (2009). Large-scale malware indexing using function-call graphs. In 16th conference on Computer and communications security, pages 611-620. ACM.
  5. Hutchins, E. M., Cloppert, M. J., and Amin, R. M. (2011). Intelligence-driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains. Leading Issues in Information Warfare & Security Research, 1:80.
  6. Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97.
  7. Luh, R., Marschalek, S., Kaiser, M., Janicke, H., and Schrittwieser, S. (2016a). Semantics-aware detection of targeted attacks: a survey. Journal of Computer Virology and Hacking Techniques, pages 1-39.
  8. Luh, R., Schrittwieser, S., and Marschalek, S. (2016b). TAON: An ontology-based approach to mitigating targeted attacks. In iiWAS 2016. ACM.
  9. M ünz, G. and Carle, G. (2007). Real-time analysis of flow data for network attack detection. In 10th IFIP/IEEE Int. Symposium on Integrated Network Management, pages 100-108. IEEE.
  10. Noble, C. C. and Cook, D. J. (2003). Graph-based anomaly detection. In 9th Intl. conference on knowledge discovery and data mining, pages 631-636. ACM.
  11. Papagelis, A. and Kalles, D. (2000). GA Tree: genetically evolved decision trees. In 12th Intl. Conference on Tools with Artificial Intelligence , page 203.
  12. Rieck, K., Trinius, P., Willems, C., and Holz, T. (2011). Automatic analysis of malware behavior using machine learning. Journal of Computer Security.
  13. Sheyner, O., Haines, J., Jha, S., Lippmann, R., and Wing, J. M. (2002). Automated generation and analysis of attack graphs. In IEEE Symposium on Security and privacy, pages 273-284. IEEE.
  14. Sperotto, A., Schaffrath, G., Sadre, R., Morariu, C., Pras, A., and Stiller, B. (2010). An Overview of IP FlowBased Intrusion Detection. IEEE Communications Surveys & Tutorials, 12(3):343-356.
  15. Wagner, M., Fischer, F., Luh, R., Haberson, A., Rind, A., Keim, D., Aigner, W., Borgo, R., Ganovelli, F., and Viola, I. (2015). A Survey of Visualization Systems for Malware Analysis. In Eurographics Conference on Visualization, pages 105-125. EuroGraphics.

Paper Citation

in Harvard Style

Luh R., Schrittwieser S., Marschalek S. and Janicke H. (2017). Design of an Anomaly-based Threat Detection & Explication System . In Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, ISBN 978-989-758-209-7, pages 397-402. DOI: 10.5220/0006205203970402

in Bibtex Style

author={Robert Luh and Sebastian Schrittwieser and Stefan Marschalek and Helge Janicke},
title={Design of an Anomaly-based Threat Detection & Explication System},
booktitle={Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP,},

in EndNote Style

JO - Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP,
TI - Design of an Anomaly-based Threat Detection & Explication System
SN - 978-989-758-209-7
AU - Luh R.
AU - Schrittwieser S.
AU - Marschalek S.
AU - Janicke H.
PY - 2017
SP - 397
EP - 402
DO - 10.5220/0006205203970402