Health advice from a chatbot? Ute Schmid comments on the latest study results

A recent study by a team of researchers at Oxford University, published in the journal “Nature Medicine”, highlights problems with using large language models to answer health questions. Many people use chatbots for initial guidance, but often provide incomplete or vague information. This leads to misunderstandings between users and the model, which reduces the quality of the answers. The results of the study have been widely reported in the media. Professor Ute Schmid, Chair of Cognitive Systems at Otto Friedrich University in Bamberg and bidt Director, was asked by the Science Media Centre (SMC) to assess the study’s findings.

The “Pharmazeutische Zeitung” (PZ) picked up on the results of the study in an article. The article primarily examines the question of why patient communication with LLMs goes wrong. Schmid sees the main problem causing errors in interaction with AI as a lack of experience in formulating queries and a lack of basic medical knowledge. However, she considers the fundamental use of such systems to be sensible.

The “Berliner Zeitung” also discusses in an article how useful language models such as ChatGPT are for assessing health. Here, too, the central problem is highlighted as being the interaction process with AI. Incomplete information, inaccurate questions or selective responses weaken the outcome of the systems. On how the potential of LLMs can be used positively in diagnostics, Schmid says:

Chatbots specialising in medical questions can be very useful in helping people to better assess whether it is advisable to visit a doctor's surgery or accident and emergency department. However, such services must guarantee high-quality responses and be embedded in a meaningful socio-technical context. Quality-assured chatbots could, for example, be offered through statutory health insurance funds and recommended by GPs as a first port of call.

Prof. Dr. Ute Schmid To the profile

The online specialist portal “Medscape” and the “Deutsche Ärzteblatt” also discuss the study results. Both media outlets, which focus on medicine, deal more specifically with the study’s approach and methodology. In the “Deutsches Ärzteblatt,” Schmid emphasises that specialised, quality-tested chatbots could offer useful initial guidance – for example, to better assess whether it is advisable to visit a doctor’s office or an emergency room.

The “Bild” newspaper also refers to Schmid’s assessment. Human expertise is crucial when it comes to targeted searches, but also when it comes to critically reflecting on the results. Specific “operating knowledge” is needed to reduce misunderstandings between humans and AI.

---