Where is that Button Again?! – Towards a Universal GUI Search Engine

Sven Hertling, Markus Schröder, Christian Jilek, Andreas Dengel

Abstract

In feature-rich software a wide range of functionality is spread across various menus, dialog windows, toolbars etc. Remembering where to find each feature is usually very hard, especially if it is not regularly used. We therefore provide a GUI search engine which is universally applicable to a large number of applications. Besides giving an overview of related approaches, we describe three major problems we had to solve, which are analyzing the GUI, understanding the users’ query and executing a suitable solution to find a desired UI element. Based on a user study we evaluated our approach and showed that it is particularly useful if a not regularly used feature is searched for. We already identified much potential for further applications based on our approach.

References

  1. Adar, E., Dontcheva, M., and Laput, G. (2014). Commandspace: modeling the relationships between tasks, descriptions and features. In Proceedings of the 27th annual ACM symposium on User interface software and technology, pages 167-176. ACM.
  2. Aho, P., Suarez, M., Kanstren, T., and Memon, A. (2013). Industrial adoption of automatically extracted GUI models for testing. In Proceedings of the 3rd International Workshop on Experiences and Empirical Studies in Software Modelling. Springer Inc.
  3. Aho, P., Suarez, M., Kanstren, T., and Memon, A. (2014). Murphy tools: Utilizing extracted GUI models for industrial software testing. In Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on, pages 343-348.
  4. Buckley, C. (2004). Trec eval ir evaluation package. http://trec.nist.gov/trec eval/. accessed on 2016-10- 25.
  5. Buckley, C. and Voorhees, E. M. (2005). Retrieval system evaluation. TREC: Experiment and evaluation in information retrieval, pages 53-75.
  6. Cockburn, A. and Gutwin, C. (2009). A predictive model of human performance with scrolling and hierarchical lists. Human Computer Interaction, 24(3):273-314.
  7. Davis, M. and Iancu, L. (2016). Unicode text segmentation. http://unicode.org/reports/tr29/. accessed on 2016-10- 25.
  8. Dixon, M. and Fogarty, J. (2010). Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1525-1534. ACM.
  9. Dixon, M., Leventhal, D., and Fogarty, J. (2011). Content and hierarchy in pixel-based methods for reverse engineering interface structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 969-978. ACM.
  10. Dubey, N. and Shiwani, M. S. (2014). Studying and comparing automated testing tools; ranorex and testcomplete. International Journal Of Engineering And Computer Science, 3:5916-5923.
  11. Ekstrand, M., Li, W., Grossman, T., Matejka, J., and Fitzmaurice, G. (2011). Searching for software learning resources using application context. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 195-204. ACM.
  12. Elasticsearch reference (2016). Stop token filter. http://www.elasticsearch.org/guide/en/elasticsearch/ reference/current/analysis-stop-tokenfilter.html. accessed on 2016-10-25.
  13. Fellbaum, C. (1998). WordNet - An Electronic Lexical Database. MIT Press.
  14. Fourney, A., Mann, R., and Terry, M. (2011). Query-feature graphs: bridging user vocabulary and system functionality. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 207-216. ACM.
  15. Gillespie, C. S. et al. (2015). Fitting heavy tailed distributions: The powerlaw package. Journal of Statistical Software, 64(i02).
  16. Google Inc. (2016). Chrome-browser features. https://www.google.com/chrome/browser/features. html. accessed on 2016-10-25.
  17. Grilo, A. M., Paiva, A. C., and Faria, J. P. (2010). Reverse engineering of gui models for testing. In 5th Iberian Conference on Information Systems and Technologies, pages 1-6. IEEE.
  18. Hendy, J., Booth, K. S., and McGrenere, J. (2010). Graphically enhanced keyboard accelerators for guis. In Proceedings of Graphics Interface 2010, pages 3-10. Canadian Information Processing Society.
  19. IMC Information Multimedia Communication AG (2016). Electronic performance support system imc process guide. https://www.im-c.de/en/learningtechnologies/performance-support. accessed on 2016-10-25.
  20. Lafreniere, B., Bunt, A., and Terry, M. (2014). Task-centric interfaces for feature-rich software. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design, pages 49-58. ACM.
  21. Lafreniere, B., Chilana, P. K., Fourney, A., and Terry, M. A. (2015). These aren't the commands you're looking for: Addressing false feedforward in feature-rich software. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, pages 619-628. ACM.
  22. Laput, G. P., Dontcheva, M., Wilensky, G., Chang, W., Agarwala, A., Linder, J., and Adar, E. (2013). Pixeltone: a multimodal interface for image editing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2185-2194. ACM.
  23. Laugwitz, B., Held, T., and Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. Springer.
  24. Memon, A. M. (2007). An event-flow model of GUI-based applications for testing. Software Testing, Verification and Reliability, 17(3):137-157.
  25. Memon, A. M., Banerjee, I., and Nagarajan, A. (2003). Gui ripping: Reverse engineering of graphical user interfaces for testing. In WCRE, volume 3, page 260.
  26. Microsoft Corporation (2000). Microsoft active accessibility: Architecture. http://msdn.microsoft.com/enus/library/ms971310.aspx. accessed on 2016-10-25.
  27. Microsoft Corporation (2011). Office labs: Search commands. http://www.microsoft.com/en-us/download/ details.aspx?id=28559. accessed on 2016-10-25.
  28. Microsoft Corporation (2016a). Do things quickly with tell me. https://support.office.com/en-us/article/Dothings-quickly-with-Tell-Me-f20d2198-17b8-4b09- a3e5-007a337f1e4e. accessed on 2016-10-25.
  29. Microsoft Corporation (2016b). Quick launch. http://msdn.microsoft.com/en-us/library/ hh417697.aspx. accessed on 2016-10-25.
  30. Microsoft Corporation (2016c). What is cortana? https://support.microsoft.com/en-us/help/17214/ windows-10-what-is. accessed on 2016-10-25.
  31. Microsoft Corporation (2016d). Windows automation api. http://msdn.microsoft.com/en-us/library/win dows/desktop/ff486375(v=vs.85).aspx. accessed on 2016-10-25.
  32. Miller, G. A. (1995). WORDNET: a Lexical Database for English. Communications of the ACM, 38(11):39-41.
  33. Miser, B. (2008). Mac OS X Leopard in Depth. Que Publishing.
  34. Naber, D. (2004). Openthesaurus: Building a thesaurus with a web community. Citeseer, 3:2005.
  35. Naber, D. (2005). OpenThesaurus: ein offenes deutsches Wortnetz. Beiträge zur GLDV-Tagung, pages 422- 433.
  36. Naboulsi, Z. (2012). Visual studio 2012 new features: Quick launch. http://blogs.msdn.com/b/ zainnab/archive/2012/06/26/visual-studio-2012-newfeatures-quick-launch.aspx. accessed on 2016-10-25.
  37. Ness, R. (2011). Mac OS X Lion in Depth. Que Publishing.
  38. Pogue, D. (2012). OS X Mountain Lion: The Missing Manual. O'Reilly Media.
  39. Porter, M. F. (1980). An algorithm for suffix stripping. Program, 14(3):130-137.
  40. Ramesh, V., Hsu, C., Agrawala, M., and Hartmann, B. (2011). Showmehow: translating user interface instructions between applications. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 127-134. ACM.
  41. Ranorex GmbH (2016). Ranorex hompage. http://www. ranorex.com. accessed on 2016-10-25.
  42. Savoy, J. (2006). Light stemming approaches for the french, portuguese, german and hungarian languages. In Proceedings of the 2006 ACM symposium on Applied computing, pages 1031-1035. ACM.
  43. Scarr, J., Cockburn, A., Gutwin, C., and Bunt, A. (2012). Improving command selection with commandmaps. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 257-266. ACM.
  44. Scarr, J., Cockburn, A., Gutwin, C., Bunt, A., and Cechanowicz, J. E. (2014). The usability of commandmaps in realistic tasks. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, pages 2241-2250. ACM.
  45. SmartBear Software (2016). TestComplete hompage. https://smartbear.com/product/testcomplete/overview/. accessed on 2016-10-25.
  46. TestStack (2016). White. https://github.com/Test Stack/White. accessed on 2016-10-25.
  47. Yeh, T., Chang, T.-H., and Miller, R. C. (2009). Sikuli: using gui screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology, pages 183- 192. ACM.
Download


Paper Citation


in Harvard Style

Hertling S., Schröder M., Jilek C. and Dengel A. (2017). Where is that Button Again?! – Towards a Universal GUI Search Engine . In Proceedings of the 9th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-220-2, pages 217-227. DOI: 10.5220/0006201402170227


in Bibtex Style

@conference{icaart17,
author={Sven Hertling and Markus Schröder and Christian Jilek and Andreas Dengel},
title={Where is that Button Again?! – Towards a Universal GUI Search Engine},
booktitle={Proceedings of the 9th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2017},
pages={217-227},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006201402170227},
isbn={978-989-758-220-2},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 9th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Where is that Button Again?! – Towards a Universal GUI Search Engine
SN - 978-989-758-220-2
AU - Hertling S.
AU - Schröder M.
AU - Jilek C.
AU - Dengel A.
PY - 2017
SP - 217
EP - 227
DO - 10.5220/0006201402170227