Radiologists in hospitals see hundreds of images every day: CT scans, ultrasound images, MRI scans. Complex anatomical structures in which they have to detect the smallest changes. This takes time and requires maximum concentration for hours.
The workload in hospitals is constantly increasing. That is why we will increasingly see AI-based recommendation systems in use for medical diagnoses in the future. They are extremely fast and do not tire.
Prof. Dr. Matthias Uhl To the profile
In the bidt project “Responsibility gaps in human-machine interactions”, Uhl, together with his colleague Dr Sebastian Krügel and an interdisciplinary team, is shedding light on the ambivalence of trust in AI. Because yes, doctors benefit from support systems – but what happens when they trust them too much ?
Research project
Machine learning for tumour classification
AI already provides occasional support in the field of oncological imaging; specifically in the classification of tumours according to internationally defined criteria. How is the severity of the tumour to be assessed? The therapy a patient receives depends on the answer. For tumour classification, the algorithm analyses several thousand images, recognises patterns and applies them to the new case. The more often it does this, the more images it has available, the better it learns.
In future, doctors will be able to include the AI’s recommendations in their diagnoses. However, the machine has no medical experience, no memory of unusual cases and, of course, no relationship with the patient. Are humans and machines working optimally hand in hand here?
How much trust in AI is appropriate?
The research team is convinced: trust in machines is ambivalent. This becomes a problem when AI technologies are used in clinics without defining the framework conditions. Is it legitimate for a doctor to say: “In the end, I only followed the recommendation of the system”? Does she alone bear the responsibility when a patient’s life is at stake? And is the manufacturer of the software used jointly responsible? Questions to which there are currently no clear answers.
In addition: people underestimate their trust in machines. There is a difference between the level of trust we find rationally appropriate and how we actually behave when interacting with machines. That is why the project team is investigating how practical cooperation with AI systems should be designed in everyday clinical practice.
This research could also give a twist to the public debate about AI. Because the researchers see a discursive slant here: “You get the impression that we only need to strengthen trust in digital systems. But rather, we need to practise a proper level of trust,” says Matthias Uhl. “We all know the stories of people who trust their sat nav so much that they even still drive up the one-way street when others are honking wildly.”
AI in the medical context offers great potential. However, increasing trust in AI cannot be the only goal. We also have to deal critically with the technology. We ask: How can we institutionalise this critical approach in practical everyday hospital life?
Dr. Sebastian Krügel To the profile
Designing AI systems in medicine
An essential part of the bidt project is the question of how to design AI-based recommendation systems in clinics. In order to gain insights in the field here, the team led by Professor Dr Marc Aubreville and Jonas Ammeling from the Chair of Image Understanding and Medical Application of AI at TH Ingolstadt is developing adapted AI solutions. In a later project step, the aim is to analyse their use in hospitals.
Particularly “unintended” effects should come to light: Does the software influence medical decisions in a certain direction without the actors being aware of it? If so, caution is called for. “We know these effects from our preliminary work in non-medical contexts,” says Matthias Uhl. “The order of AI recommendation, for example, plays a crucial role:
- If a subject makes his judgement and is only then shown the system recommendation, he is less likely to be influenced. This is because he has set his own anchor in advance and is more critical of the AI.
- If a respondent sees the system’s recommendation first and then makes his or her judgement, the probability is high that he or she will think, “That’s how my decision would have been.”
The project team wants to reduce such influence in medical diagnoses. “We want to derive design paradigms for AI-based recommendation services that allow for a calibration of trust,” says Sebastian Krügel. “We want to understand which levers work and how: for example, individual features that make the doctor’s decision more prominent; or make the doctor pause because the system registers residual doubt.” This is to empower actors in clinics:
- Use AI-based technologies in accordance with their self-image (e.g. as a doctor or clinic management); and
- to be able to explain the use of AI technologies in the respective hospital to patients in concrete terms.
What do patients think about AI?
Computer science, ethics, data science – various disciplines are represented in the bidt-funded consortium project to comprehensively research the field of “AI and medicine”. Professor Dr Alexis Fritz and Angelika Kießig from the Chair of Moral Theology at the Catholic University of Eichstätt-Ingolstadt are analysing human-machine interactions, taking into account philosophical concepts of responsibility and trust.
The interdisciplinary research team also includes the perspective of patients on artificial intelligence, because when algorithms are involved in medical decisions, this must be transparent and comprehensible for those affected. This is stipulated by the World Health Organisation (WHO) in its guidelines on “Artificial Intelligence and Health”. The researchers therefore want to use a representative survey in Germany to find out: How do patients assign responsibility to medical actors, e.g. when an incorrect diagnosis was made with AI support?
“The attitude of patients towards AI systems has an impact on the behaviour of doctors,” explains Matthias Uhl. Do those affected show greater understanding for diagnoses if they correspond to the recommendations of digital assistants? Then it seems reasonable to assume that doctors will tend to adapt their diagnoses to the recommendations and trust their own assessments less. This makes it all the more important to empower doctors to overrule AI recommendations in case of doubt.
Helping to shape human-machine interaction
Although the bidt project is still in its infancy, it is already clear that ethical standards and their anchoring in pragmatic guidelines are needed. In the specific case of AI diagnostic systems, they can help reconcile the requirements of doctors, patients and hospitals. “The current AI guidelines are on too abstract a level. In application, it is extremely difficult to judge: What does this mean for me as a developer of AI systems or as a doctor in a hospital?” emphasises Sebastian Krügel.
It is important to actively shape AI applications in medicine today in order to learn from them for future areas of application.
Prof. Dr. Matthias Uhl To the profile
The use of AI, or more precisely machine learning, still has a lot of potential, especially in the context of medical imaging. This is because the data from which the machine can learn must be digitised. This is already the case on a large scale for CT scans. Soon it will be possible to combine digitally available health data for AI-assisted diagnoses: MRI images, blood analyses or patient records. This makes it all the more important to have a scientifically supported discussion about data security as well as insights into the responsible, human-centred design of AI systems.
