Is Your AI Scribe Making Things Up? An Accuracy Checklist for Fire and EMS

A vendor shows you an AI tool that writes your crews’ narratives. The demo looks great. A clean chart appears in about ten seconds.

But one question stays in your mind: What if it makes something up?

That is the right question. Every chief and medical director asks us. We would ask it too.

A patient care report is a legal record. Hospitals, quality assurance (QA) teams, attorneys, and sometimes juries may read it. A writing tool carries more risk than an inventory tool. If it adds the wrong fact, someone could get hurt. Your department is responsible for the final chart.

We will not tell you to stop worrying. We will give you a clear way to test the tool.

What “making things up” actually means

The word people use is hallucination. It means the AI adds a detail that was not in the source.

A general-purpose model like ChatGPT is trained to produce text that sounds right. When information is missing, it can fill the gap with something that fits the pattern. Maybe no one said the blood pressure. Maybe the audio dropped for four seconds. In a chart, the result could be a blood pressure no one took, a lung sound no one heard, or a denial of chest pain the patient never gave.

That is the risk. It is real, and it is why this checklist exists. Pasting patient details into a consumer chatbot also creates risks beyond HIPAA. We cover that topic in our guide to writing EMS narratives.

A tool built for EMS can be designed to leave a gap blank. Your job is to make sure it does.

The checklist: six questions to ask any AI documentation vendor

Ask us these questions. Ask every other vendor too. If a vendor will not give you a clear answer, pay attention.

1. Does it ever add facts that were not said? Do not accept “it’s very accurate.” Ask for a written yes-or-no answer: does the system create clinical details that no one said on the call? If a vendor will not put the answer in an email, do not expect the vendor to defend it in sworn testimony.

2. Does a person review and sign the chart before it is filed? No AI should write directly into an electronic patient care report (ePCR) without review. The medic who ran the call must read the draft, fix it, and sign it. Automatic filing may save time, but it also adds risk.

3. Can you see the source? The narrative should link back to a transcript of what was said on scene. Without that source, you cannot check the draft. Ask where the transcript is stored and how long it is kept.

4. What happens offline or when the audio is bad? Think about basements, rural roads, a running engine, a screaming child, or a patient who mumbles. When the system cannot hear a word, does it guess? Or does it leave a gap for the medic? Those are two very different products.

5. Who tests it? A score from the vendor’s lab is not enough. Ask for a pilot where your crews use it on your calls. Have them compare each draft with what was said. A vendor confident in the tool should agree.

6. What record will remain if someone challenges a chart in two years? A complaint or lawsuit may arrive long after everyone forgets the call. Ask whether the record shows when and where the session happened, how long it ran, and who approved the chart. This record is called an audit trail.

How vitalvoice answers each one

Here are our answers in the same order. Hold us to them.

We do not invent facts. If it was not said on scene, it is not in the draft. The product is built around that rule. We will put it in writing for your compliance officer.

A medic reviews, edits, and signs every draft. Ambient Scribe creates a draft. Nothing reaches the ePCR until the person who ran the call reads and signs it. That step is required.

Every narrative has a source transcript. The draft comes from what was recorded on scene. A medic can check the chart against that source. Transcripts stay on the device and never sync to our servers. They are erased at the end of the shift. We sign a Business Associate Agreement (BAA).

It works offline, and gaps stay blank. vitalvoice runs without cell signal. That matters in basements and on county roads. When audio is unclear, the draft leaves a gap instead of making a likely guess. A blank for the medic to fill is safer than a number the software invented.

Your crews test it. Every department starts with a pilot. Your medics use it on real calls and compare the drafts with what they remember saying. Our pilot departments in Colorado did that. CityEMT finished its pilot with a 95% Net Promoter Score (NPS). Crews rate the app 5.0 on the App Store.

Every session has a time and GPS location. If someone reviews a chart two years later, there is a record of when and where the session happened.

The comparison that actually matters

When you judge an AI scribe, you may compare it with a perfect scribe. That person would never mishear, leave something out, or add an opinion. No software can pass that test.

But that is not the real choice. Today, a medic may write from memory at hour 47 of a 48-hour shift. The call may have happened three calls earlier. A family member may translate a stroke check. A phone interpreter may leave no record to review later.

Those methods can also produce mistakes. The errors may be hard to find because there is no source to check. A tool with a transcript, review step, and timestamp makes gaps easier to see. That matters in a legal record. If you are also making your narratives more consistent, read our guide to EMS narrative formats.

Hold AI to a high bar. Hold your current process to the same bar.

---

Quick answers

Can an AI scribe invent vital signs in an EMS narrative? A general-purpose model can add text that seems to fit a gap. A tool built for EMS should not. vitalvoice only writes what was said on scene. A medic reviews and signs the draft before anything reaches the ePCR.

How do I verify an AI-generated narrative is accurate? Check the draft against the source transcript. Make sure a person reviews and signs it. If a vendor cannot show you the transcript, you cannot check where the details came from.

Does an AI scribe work without cell signal? vitalvoice does. It runs offline, and when audio is unclear it leaves a gap instead of guessing.

Run this checklist against us

Chiefs and medical directors: Book a 15-minute demo and bring all six questions. We will answer each one. Then run a pilot so your own crews can compare the drafts with what they said on the call.

Crews: Download vitalvoice from the App Store. Each mode includes free sessions. Use one on your next call, then read the draft and check every detail before you sign.