Search based GUI Test Generation in Java - Comparing Code-based and EFG-based Optimization Goals

Mathias Menninghaus, Falk Wilke, Jan-Philipp Schleutker, Elke Pulvermüller

Abstract

Modern software systems often communicate with their users by graphical user interfaces (GUI). While the underlying business logic may be fully covered by unit tests, the GUI mostly is not. Despite the widespread use of capture and replay tools, which leave the test generation of GUI tests to the user, recent research also focuses on automated GUI test generation. From the numerous approaches, which include symbolic execution, model-based generation, and random testing, search based test data generation seems to be the most promising. In this paper, we create GUI tests using hill climbing, simulated annealing and several genetic algorithms which deal differently with the sequence length and use multi or single objective algorithms. These different test data generators are compared in terms of runtime and coverage. All approaches are also compared using different optimization goals which are a high coverage of the event flow graph (EFG) of the GUIs and a high coverage of the underlying source code. The evaluation shows that the genetic algorithms outperform hill climbing and simulated annealing in terms of coverage, and that targeting a high EFG coverage causes the best runtime performance.

References

  1. Aburas, A. and Groce, A. (2016). A Method Dependence Relations Guided Genetic Algorithm. In International Symposium on Search Based Software Engineering, pages 267-273. Springer International Publishing.
  2. Arcuri, A. (2012). A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage. IEEE Transactions on Software Engineering, 38(3):497-519.
  3. Baresi, L., Lanzi, P. L., and Miraz, M. (2010). TestFul: An Evolutionary Test Approach for Java. In Third International Conference on Software Testing, Verification and Validation, pages 185-194. IEEE.
  4. Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., and Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5):507-525.
  5. Cadar, C., Dunbar, D., and Engler, D. R. (2008). KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In 8th USENIX conference on Operating systems, pages 209-224.
  6. Carino, S. (2016). Dynamically Testing Graphical User Interfaces. PhD thesis.
  7. Chee, Y. M., Colbourn, C. J., Horsley, D., and Zhou, J. (2013). Sequence Covering Arrays. SIAM Journal on Discrete Mathematics, 27(4):1844-1861.
  8. Chen, M., Qiu, X., Xu, W., Wang, L., Zhao, J., and Li, X. (2009). UML Activity Diagram-Based Automatic Test Case Generation For Java Programs. The Computer Journal, 52(5):545-556.
  9. Corne, D. W., Jerram, N. R., Knowles, J. D., Oates, M. J., and J, M. (2001). PESA-II: Region-based Selection in Evolutionary Multiobjective Optimization. In Proceedings of the genetic and evolutionary computation conference.
  10. Csallner, C. and Smaragdakis, Y. (2004). JCrasher: an automatic robustness tester for Java. Software: Practice and Experience, 34(11):1025-1050.
  11. Dan, H., Harman, M., Krinke, J., Li, L., and Marginean, A. (2014). Pidgin crasher: searching for minimised crashing GUI event sequences. SSBSE 2014 - Symposium on Search -Based Software Engineering.
  12. Deb, K., Agrawal, S., Pratap, A., and Meyarivan, T. (2000). A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimization: NSGA-II. In International Conference on Parallel Problem Solving From Nature, pages 849-858. Springer Berlin Heidelberg.
  13. Fraser, G. and Arcuri, A. (2011). It is Not the Length That Matters, It is How You Control It. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation (ICST) , pages 150-159. IEEE.
  14. Ganov, S. R., Killmar, C., Khurshid, S., and Perry, D. E. (2008). Test generation for graphical user interfaces based on symbolic execution. In Proceedings of the 3rd international workshop on Automation of software test, pages 33-40. ACM.
  15. Godefroid, P., Klarlund, N., and Sen, K. (2005). DART: directed automated random testing. ACM SIGPLAN Notices, 40(6):213-223.
  16. Goldberg, D. E. and Deb, K. (1991). A Comparative Analysis of Selection Schemes Used in Genetic Algorithms. Foundations of Genetic Algorithms, pages 69-93.
  17. Gross, F., Fraser, G., and Zeller, A. (2012). EXSYST: search-based GUI testing. In ICSE Proceedings of th International Conference on Software Engineering, pages 1423-1426. IEEE Press.
  18. Harman, M., Mansouri, S. A., and Zhang, Y. (2012a). Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys (CSUR), 45(1).
  19. Harman, M., McMinn, P., de Souza, J. T., and Yoo, S. (2012b). Search Based Software Engineering: Techniques, Taxonomy, Tutorial. In Empirical Software Engineering and Verification , pages 1-59. Springer Berlin Heidelberg.
  20. Ingber, L. and Rosen, B. (1992). Genetic Algorithms and Very Fast Simulated Reannealing: A comparison. Mathematical and computer modelling, 16(11):87- 100.
  21. Khurshid, S., Visser, W., Pa?sa?reanu, C. S., and Khurshid, S. (2004). Test input generation with java PathFinder. ACM SIGSOFT Software Engineering Notes, 29(4):97-107.
  22. Korel, B. (1990). Automated software test data generation. IEEE Transactions on Software Engineering, 16(8):870-879.
  23. Laumanns, M., Thiele, L., and Zitzler, E. (2001). SPEA2: Improving the strength Pareto evolutionary algorithm. Eurogen, 3242(103):95-100.
  24. McMinn, P. (2011). Search-Based Software Testing: Past, Present and Future. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 153-163. IEEE.
  25. Memon, A. M. (2007). An event-flow model of GUI-based applications for testing. Software Testing, Verification and Reliability, 17:137-157.
  26. Menninghaus, M. and Pulverm üller, E. (2016). Towards Using Code Coverage Metrics for Performance Comparison on the Implementation Level. In the 7th ACM/SPEC, pages 101-104. ACM.
  27. Mitchell, M., Holland, J. H., and Forrest, S. (1993). When will a genetic algorithm outperform hill climbing? Ann Arbor.
  28. Myers, B., Hudson, S. E., and Pausch, R. (2000). Past, Present, and Future of User Interface Software Tools. ACM Transactions on Computer-Human Interaction (TOCHI), 7(1):3-28.
  29. Nguyen, B. N., Robbins, B., Banerjee, I., and Memon, A. (2014). GUITAR: an innovative tool for automated testing of GUI-driven software. Automated Software Engineering, 21(1):65-105.
  30. Pacheco, C., Lahiri, S. K., and Ernst, M. D. (2007). Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering.
  31. Russell, S. J., Norvig, P., Canny, J. F., Malik, J. M., and Edwards, D. D. (2003). Artificial intelligence: a modern approach, volume 2. Prentice hall Upper Saddle River.
  32. Sen, K. and Agha, G. (2006). CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools. In International Conference on Computer Aided Verification , pages 419-423. Springer Berlin Heidelberg.
  33. Soffa, M. L., Pollack, M. E., and Memon, A. M. (2001). Coverage criteria for GUI testing. ACM SIGSOFT Software Engineering Notes, 26(5):256-267.
  34. Vieira, M., Leduc, J., Hasling, B., Subramanyan, R., and Kazmeier, J. (2006). Automation of GUI testing using a model-driven approach. In Proceedings of the 2006 international workshop on Automation of software test, pages 9-14. ACM.
  35. Yuan, X., Cohen, M. B., and Memon, A. M. (2011). GUI interaction testing: Incorporating event context. IEEE Transactions on Software Engineering, 97(4):559- 574.
  36. Yuan, X. and Memon, A. M. (2010). Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback. IEEE Transactions on Software Engineering, 36(1):81-95.
  37. Zitzler, E., Deb, K., and Thiele, L. (2006). Comparison of Multiobjective Evolutionary Algorithms: Empirical Results. Evolutionary computation, 8(2):173-195.
Download


Paper Citation


in Harvard Style

Menninghaus M., Wilke F., Schleutker J. and Pulvermüller E. (2017). Search based GUI Test Generation in Java - Comparing Code-based and EFG-based Optimization Goals . In Proceedings of the 12th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE, ISBN 978-989-758-250-9, pages 179-186. DOI: 10.5220/0006277801790186


in Bibtex Style

@conference{enase17,
author={Mathias Menninghaus and Falk Wilke and Jan-Philipp Schleutker and Elke Pulvermüller},
title={Search based GUI Test Generation in Java - Comparing Code-based and EFG-based Optimization Goals},
booktitle={Proceedings of the 12th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,},
year={2017},
pages={179-186},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006277801790186},
isbn={978-989-758-250-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 12th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,
TI - Search based GUI Test Generation in Java - Comparing Code-based and EFG-based Optimization Goals
SN - 978-989-758-250-9
AU - Menninghaus M.
AU - Wilke F.
AU - Schleutker J.
AU - Pulvermüller E.
PY - 2017
SP - 179
EP - 186
DO - 10.5220/0006277801790186