Purpose: This article reports the user-oriented evaluation of a text- and content-based medical
image retrieval system. User tests with radiologists using a search system for images
in the medical literature are presented. The goal of the tests is to assess the usability of
the system, identify system and interface aspects that need improvement and useful additions.
Another objective is to investigate the system’s added value to radiology information
retrieval. The study provides an insight into required specifications and potential shortcomings
of medical image retrieval systems through a concrete methodology for conducting
user tests.
Methods: User tests with a working image retrieval system of images from the biomedical
literature were performed in an iterative manner, where each iteration had the participants
perform radiology information seeking tasks and then refining the system as well as the
user study design itself. During these tasks the interaction of the users with the system was
monitored, usability aspects were measured, retrieval success rates recorded and feedback
was collected through survey forms.
Results: In total, 16 radiologists participated in the user tests. The success rates in finding
relevant information were on average 87% and 78% for image and case retrieval tasks,
respectively. The average time for a successful search was below 3 min in both cases. Users
felt quickly comfortable with the novel techniques and tools (after 5 to 15min), such as
content-based image retrieval and relevance feedback. User satisfaction measures show a
very positive attitude toward the system’s functionalities while the user feedback helped
identifying the system’s weak points. The participants proposed several potentially useful
newfunctionalities, such as filtering by imaging modality and search for articles using image
examples.
Conclusion: The iterative character of the evaluation helped to obtain diverse and detailed
feedback on all system aspects. Radiologists are quickly familiar with the functionalities but
have several comments on desired functionalities. The analysis of the results can potentially
assist system refinement for future medical information retrieval systems. Moreover, the
methodology presented as well as the discussion on the limitations and challenges of such