Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System

Derek Sleeman, Kiril Kostadinov, Laura Moss, Malcolm Sim

Abstract

Knowledge intensive clinical systems, as well as machine learning algorithms, have become more widely used over the last decade or so. These systems often need access to sizable labelled datasets which could be more useful if their instances are accurately labelled / annotated. A variety of approaches, including statistical ones, have been used to label instances. In this paper, we discuss the use of domain experts, in this case clinicians, to perform this task. Here we recognize that even highly rated domain experts can have differences of opinion on certain instances; we discuss a system inspired by the Delphi approaches which helps experts resolve their differences of opinion on classification tasks. The focus of this paper is the IS-DELPHI tool which we have implemented to address the labelling issue; we report its use in a medical domain in a study involving 12 Intensive Care Unit clinicians. The several pairs of experts initially disagreed on the classification of 11 instances but as a result of using IS-DELPHI all those disagreements were resolved. From participant feedback (questionnaires), we have concluded that the medical experts understood the task and were comfortable with the functionality provided by IS-DELPHI. We plan to further enhance the system’s capabilities and usability, and then use IS-DELPHI, which is a domain independent tool, in a number of further medical domains.

Download


Paper Citation


in Harvard Style

Sleeman D., Kostadinov K., Moss L. and Sim M. (2020). Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System.In Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 5: HEALTHINF, ISBN 978-989-758-398-8, pages 66-76. DOI: 10.5220/0008922000660076


in Bibtex Style

@conference{healthinf20,
author={Derek Sleeman and Kiril Kostadinov and Laura Moss and Malcolm Sim},
title={Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System},
booktitle={Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 5: HEALTHINF,},
year={2020},
pages={66-76},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008922000660076},
isbn={978-989-758-398-8},
}


in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 5: HEALTHINF,
TI - Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System
SN - 978-989-758-398-8
AU - Sleeman D.
AU - Kostadinov K.
AU - Moss L.
AU - Sim M.
PY - 2020
SP - 66
EP - 76
DO - 10.5220/0008922000660076