Designing Intelligent Agents to Judge Intrinsic Quality of Human Decisions

Tamal T. Biswas

2015

Abstract

Research on judging decisions made by fallible (human) agents is not as much advanced as research on finding optimal decisions. Human decisions are often influenced by various factors, such as risk, uncertainty, time pressure, and depth of cognitive capability, whereas decisions by an intelligent agent (IA) can be effectively optimal without these limitations. The concept of `depth', a well-defined term in game theory (including chess), does not have a clear formulation in decision theory. To quantify `depth' in decision theory, we can configure an IA of supreme competence to `think' at depths beyond the capability of any human, and in the process collect evaluations of decisions at various depths. One research goal is to create an intrinsic measure of the depth of thinking required to answer certain test questions, toward a reliable means of assessing their difficulty apart from item-response statistics. We relate the depth of cognition by humans to depths of search, and use this information to infer the quality of decisions made, so as to judge the decision-maker from his decisions. We use large data from real chess tournaments and evaluations from chess programs (AI agents) of strength beyond all human players. We then seek to transfer the results to other decision-making fields in which effectively optimal judgments can be obtained from either hindsight, answer banks, powerful AI agents or from answers provided by judges of various competency.

References

  1. Andrich, D. (1978). A rating scale formulation for ordered response categories. Psychometrika, 43:561-573.
  2. Andrich, D. (1988). Rasch Models for Measurement. Sage Publications, Beverly Hills, California.
  3. Baker, F. B. (2001). The Basics of Item Response Theory. ERIC Clearinghouse on Assessment and Evaluation.
  4. Busemeyer, J. R. and Townsend, J. T. (1993). Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment. Psychological review, 100(3):432.
  5. Chinchalkar, S. (1996). An upper bound for the number of reachable positions. ICCA JOURNAL, 19(3):181- 183.
  6. DiFatta, G., Haworth, G., and Regan, K. (2009). Skill rating by Bayesian inference. In Proceedings, 2009 IEEE Symposium on Computational Intelligence and Data Mining (CIDM'09), Nashville, TN, March 30-April 2, 2009, pages 89-94.
  7. Maas, H. v. d. and Wagenmakers, E.-J. (2005). A psychometric analysis of chess expertise. American Journal of Psychology, 118:29-60.
  8. Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47:149-174.
  9. Morris, G. A., Branum-Martin, L., Harshman, N., Baker, S. D., Mazur, E., Dutta, S. N., Mzoughi, T., and McCauley, V. (2005). Testing the test: Item response curves and test quality. Am. J. Phys., 74:449-453.
  10. Muraki, E. (1992). A generalized partial credit model: Application of an em algorithm. Applied psychological measurement, 16(2):159-176.
  11. Ostini, R. and Nering, M. (2006). Polytomous Item Response Theory Models. Sage Publications, Thousand Oaks, California.
  12. Rasch, G. (1960). Probabilistic models for for some intelligence and attainment tests. Danish Institute for Educational Research, Copenhagen.
  13. Thorpe, G. L. and Favia, A. (2012). Data analysis using item response theory methodology: An introduction to selected programs and applications. Psychology Faculty Scholarship, page 20.
  14. Wichmann, F. and Hill, N. J. (2001). The psychometric function: I. Fitting, sampling, and goodness of fit. Perception and Psychophysics, 63:1293-1313.
  15. WikiBooks (2012). Bestiary of behavioral economics/satisficing - Wikibooks, the free textbook project. [Online; accessed 7-August-2014].
Download


Paper Citation


in Harvard Style

T. Biswas T. (2015). Designing Intelligent Agents to Judge Intrinsic Quality of Human Decisions . In Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-074-1, pages 608-613. DOI: 10.5220/0005288406080613


in Bibtex Style

@conference{icaart15,
author={Tamal T. Biswas},
title={Designing Intelligent Agents to Judge Intrinsic Quality of Human Decisions},
booktitle={Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2015},
pages={608-613},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005288406080613},
isbn={978-989-758-074-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Designing Intelligent Agents to Judge Intrinsic Quality of Human Decisions
SN - 978-989-758-074-1
AU - T. Biswas T.
PY - 2015
SP - 608
EP - 613
DO - 10.5220/0005288406080613