Andreas D. Lattner, Tjorben Bogon, Ingo J. Timm


Simulation is widely used in order to evaluate system changes, to perform parameter optimization of systems, or to compare existing alternatives. Assistance systems for simulation studies can support the user by performing monotonous tasks and keeping track of relevant results. In this paper we present an approach to significance estimation in order to estimate, if – and when – statistically significant results are expected for certain investigations. This can be used for controlling simulation runs or providing information to the user for interaction. We introduce two approaches: one for the classification if significance is expected to occur for given samples and another for the prediction of needed replications until significance migh


  1. Bianchi, L., Dorigo, M., Gambardella, L. M., and Gutjahr, W. J. (2009). A survey on metaheuristics for stochastic combinatorial optimization. Natural Computing: an international journal, 8(2):239-287.
  2. Burl, M. C., DeCoste, D., Enke, B. L., Mazzoni, D., Merline, W. J., and Scharenbroich, L. (2006). Automated knowledge discovery from simulators. In Ghosh, J., Lambert, D., Skillicorn, D. B., and Srivastava, J., editors, Proceedings of the Sixth SIAM International Conference on Data Mining, April 20-22, 2006, Bethesda, MD, USA.
  3. Ekren, B. Y. and Heragu, S. S. (2008). Simulation based optimization of multi-location transshipment problem with capacitated transportation. In WSC 7808: Proceedings of the 40th Conference on Winter Simulation, pages 2632-2638. Winter Simulation Conference.
  4. Hoad, K., Robinson, S., and Davies, R. (2009). Automated selection of the number of replications for a discreteevent simulation. Journal of the Operational Research Society.
  5. Huber, K.-P., Syrjakow, M., and Szczerbicka, H. (1993). Extracting knowledge supports model optimization. In Proceedings of the International Simulation Technology Conference SIMTEC'93, pages 237-242, San Francisco.
  6. James, H. A., Hawick, K. A., and Scogings, C. J. (2007). User-friendly scheduling tools for large-scale simulation experiments. In WSC 7807: Proceedings of the 39th conference on Winter simulation, pages 610-616, Piscataway, NJ, USA. IEEE Press.
  7. King, R. D., Rowland, J., Oliver, S. G., Young, M., Aubrey, W., Byrne, E., Liakata, M., Markham, M., Pir, P., Soldatova, L. N., Sparkes, A., Whelan, K. E., and Clare, A. (2009). The automation of science. Science, 324(5923):85-89.
  8. King, R. D., Whelan, K. E., Jones, F. M., Reiser, P. G. K., Bryant, C. H., Muggleton, S. H., Kell, D. B., and Oliver, S. G. (2004). Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427:247-252.
  9. Klösgen, W. (1994). Exploration of simulation experiments by discovery. In AAAI-94 Workshop on Knowledge Discovery in Databases (KDD'94), Technical Report WS-94-03, pages 251-262, Menlo Park, California. The AAAI Press.
  10. Klösgen, W. (1996). Explora: A multipattern and multistrategy discovery assistant. In Fayyad, U. M., Piatetsky-Shapiro, G., and Uthurusamy, R., editors, Advances in knowledge discovery and data mining, pages 249-271. AAAI Press, Menlo Park.
  11. Laganá, D., Legato, P., Pisacane, O., and Vocaturo, F. (2006). Solving simulation optimization problems on grid computing systems. Parallel Comput., 32(9):688-700.
  12. Law, A. M. (2007). Simulation Modeling & Analysis. McGraw-Hill, 4th, internat. edition.
  13. Park, H. M. (2008). Hypothesis testing and statistical power of a test. Working paper. the university information technology services (UITS), Center for Statistical and Mathematical Computing, Indiana University.
  14. Quinlan, J. R. (1993). C4.5 - Programs for Machine Learning. Morgan Kaufmann Publishers, Inc.
  15. R Development Core Team (2010). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3- 900051-07-0.
  16. Schmidt, M. and Lipson, H. (2007). Comparison of tree and graph encodings as function of problem complexity. In GECCO 7807: Proceedings of the 9th annual conference on Genetic and evolutionary computation, pages 1674-1679, New York, NY, USA. ACM.
  17. Schmidt, M. and Lipson, H. (2009). Distilling freeform natural laws from experimental data. Science, 324(5923):81-85.
  18. Swisher, J. R. and Jacobson, S. H. (2002). Evaluating the design of a family practice healthcare clinic using discrete-event simulation. Health Care Management Science, 5(2):75-88.
  19. Swisher, J. R., Jacobson, S. H., and Yücesan, E. (2003). Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey. ACM Trans. Model. Comput. Simul., 13(2):134- 154.
  20. Witten, I. H. and Frank, E. (2005). Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, San Francisco, 2nd edition.

Paper Citation

in Harvard Style

D. Lattner A., Bogon T. and J. Timm I. (2011). AN APPROACH TO SIGNIFICANCE ESTIMATION FOR SIMULATION STUDIES . In Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART, ISBN 978-989-8425-40-9, pages 177-186. DOI: 10.5220/0003187901770186

in Bibtex Style

author={Andreas D. Lattner and Tjorben Bogon and Ingo J. Timm},
booktitle={Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,},

in EndNote Style

JO - Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,
SN - 978-989-8425-40-9
AU - D. Lattner A.
AU - Bogon T.
AU - J. Timm I.
PY - 2011
SP - 177
EP - 186
DO - 10.5220/0003187901770186