AI-powered transcription tool used in hospitals invents things no one ever said

Expecting intelligence and accuracy to "emerge" from a statistical process is absurb.

In others words, LLMs are only clearly useful if the results don't really matter or they can and will be externally verified.

LLMs negate a fundamental argument for computing --- instead of accurate results at low cost, we now have inaccurate results at high cost.

There is undoubtedly some utility to be had here but it is not at all clear or obvious that this will be widely transformative.

There will always be a need for both human oversight and accountability, and this is a good example. I think the net result will be, eventually, more and better jobs. It's a better job to validate the transcriptions than to actually transcribe.

Another example in medicine, radiologists will start handling orders of magnitude more cases. But the number of scans done might also increase exponentially as costs likewise drop.

  A machine learning engineer said he initially discovered hallucinations in about half of the over 100 hours of Whisper transcriptions he analyzed.

The '100 hours' is almost useless information. 'About half' is meaningless without knowing the sample size. Perhaps he had 5 transcripts averaging 20 hours each, and 2 of the 5 had issues. Or perhaps there were hundreds of short transripts, where the 'almost half' would imply significance.

> In an example they uncovered, a speaker said, “He, the boy, was going to, I’m not sure exactly, take the umbrella.”

> But the transcription software added: “He took a big piece of a cross, a teeny, small piece ... I’m sure he didn’t have a terror knife so he killed a number of people.”

> A speaker in another recording described “two other girls and one lady.” Whisper invented extra commentary on race, adding "two other girls and one lady, um, which were Black.”

> In a third transcription, Whisper invented a non-existent medication called “hyperactivated antibiotics.”

I didn't expect it to be this bad.

This clinical service is not something that you, the patient, should want or allow.

I use a digital recorder app to record audio from my clinical consultations. It's important for me, as a patient, to have a record, because I'm alone in there, and I frequently misremember or misunderstand things that were said.

My current recorder app has a transcription feature. It's fairly good at picking out words. It's supposed to recognize and label speakers as well, but that requires a lot of manual editing after the fact.

Still, it's fantastic having my own durable record of what was said to me, and by me. There are usually a few surprises in there!

Now, I've stopped asking for permission to record, because usually they become hostile to it. Nevertheless, it's legal, and it's my right to have.

I used whisper to create an srt file from some some voice memos I made while I was driving and it 'hallucinated' "subtitles by the amara.org community" at the very end. Re ran as txt and what do you know that line disappeared.

Oh, you don't say.

(Literally)

Well, no shit AI isnt meant for anything as serious as medicinal logging

The whisper of death is a risk.