Flan-PaLM: would you let an AI diagnose you?

By Will Brown

Artificial intelligence is looking to be the next darling of the tech world in 2023. Whereas cryptocurrencies often struggled to make their case, there are clear practical applications for this kind of technology. OpenAI’s Chat-GPT took the internet by storm in November with its ability to engage in a dialogue with the user and carry out complex tasks, and image generation software such as Dall-E 2 or Midjourney continues to be popular. Microsoft has announced that Chat-GPT is going to be integrated into Bing, and here at SciTech we’ve been using image generation to produce some of our illustrations (such as the one for this article!). But these applications are still fairly minor, and we’ve yet to see major applications outside of the tech world.

Google is looking to change that. A recent publication from their research department shows the results of an intriguing experiment: can you train up artificial intelligence to give medical advice?

After training up Flan-PaLM, their AI, on medical information, the researchers wrote out a set of medical questions that ranged from basic knowledge to making diagnoses based on scenarios. The AI was then tasked with answering these questions, and following evaluation from doctors was able to answer 67.6% of them correctly.

Following this, the researchers carried out instruction prompt tuning – rewriting their prompts individually to give the AI greater assistance in generating useful medical advice. They then gave these questions to both the AI and a group of doctors. After this, the answers were presented to a panel of doctors – but without them knowing whether an AI or a doctor wrote the answer. It was judged that 92.6% of the AI’s answers were right, compared to 92.9% of the doctors’ answers – a mere 0.3% difference. It’s important to note that this impressive result is only achievable after the instruction prompt tuning, which is a laborious process that must be done by hand – but nevertheless, the results are promising.

Now, that’s not to suggest that AI can replace the medical profession entirely. Vivek Natarajan, one of the researchers on this project, explained on Twitter that further research remains necessary before the AI models are viable for real-life clinical use. But it might not be too long before this further research is completed. A similar experiment was carried out in December 2020 with an accuracy rate of 33%, and we’ve therefore seen a massive improvement in accuracy in only two years. With a rate of improvement as significant as that, it might not be long before we have artificial intelligence that’s accurate enough to justify real-life use. But doctors aren’t going anywhere. Perhaps most crucially, the diagnoses were based off of details from an examination that could only be carried out by a trained professional.

However, there’s undoubtedly an application for this kind of technology. Whereas googling your symptoms often encourages you to self-diagnose some fatal disease, a properly trained artificial intelligence might be able to generate an informative answer and advice based off of your personal situation. As we’re seeing more and more interventions to try and alleviate pressure from the NHS, from video appointments to online forms, it’s not unreasonable to suggest that artificial intelligence could be added into this mix. It’s perhaps more likely that we’ll see artificial intelligence introduced as an assistant for doctors rather than a replacement. Earlier this year, a team of researchers from MIT and Massachusetts General Hospital worked to create an AI algorithm that could scan mammograms and predict the risk of developing breast cancer within the next five years. It was found that the models were more accurate than the current tools used for prediction. Analysis that takes radiologists years to perfect could be carried out with greater efficacy and perhaps even greater accuracy by a trained artificial intelligence.

We might be several years off an artificial intelligence that is able to completely replace doctors, but we’re a lot closer than you might think to artificial intelligence that can accurately and safely assist doctors with their work.

Image: Generated by OpenAI’s DALL·E 2