In short
Decide what the model must never do, send it as little patient data as possible, and keep a clinician's signature on anything clinical.
Clinicians spend a large part of their day on documentation: intake forms, referral letters, discharge summaries. Language models are good at exactly this kind of text work. They are also a new place for sensitive data to leak if the system is designed carelessly.
These are the rules we follow on every healthcare project. None of them replace legal advice for your jurisdiction, but they make that conversation much shorter.
1. Decide what the model is *not* allowed to do
Write the boundaries down before any code. For a patient-facing intake assistant ours usually are:
- It asks questions and summarises answers. It does not diagnose, suggest treatment or comment on medication.
- Emergency symptoms are detected by rules, not the model, and stop the conversation with clear instructions.
- When unsure, it says so and hands over to a person.
These limits shape the prompts, the tests and the user interface, and they are what a clinical lead can actually review and sign off.
2. Send the model as little as possible
The safest data is data the model never sees.
- Separate identity from content. Names, phone numbers and record numbers are replaced with placeholders before text goes to a model, and restored afterwards in your own system.
- Send only the fields the task needs. A referral-letter draft needs the clinical history, not the billing address.
- Choose where the model runs. Depending on the provider and your obligations, that may mean a provider with a data-processing agreement and no training on your data, a regional deployment, or an open model on your own infrastructure.
3. Keep a human signature on anything clinical
A model can draft a discharge summary in seconds. A clinician still reviews and signs it. Design the screen for that review: show the draft next to its source notes, highlight anything the model inferred rather than copied, and make editing faster than rewriting.
4. Log for audit, not for curiosity
You need to know who saw what and when, and which model version produced which draft. You do not need to keep full conversations forever in a debugging tool. Set retention to match your records policy, keep audit logs separate from application logs, and restrict who can read either.
5. Test with clinicians, on realistic cases
Generic benchmarks tell you little about your workflow. Ask clinicians to write test cases from real experience (with identifiers removed): vague answers, contradictions, patients who change their mind, and the rare emergencies. Every release must pass them before it ships, and the set grows each time something surprises you in production.
6. Plan for the model being wrong
It will be, occasionally. The question is whether the system catches it. Structured outputs that are validated, confidence checks that route edge cases to a person, and an easy way for staff to flag a bad output all turn a silent failure into a fixable one.
The payoff
Done this way, AI takes repetitive typing away from clinicians without asking anyone to trust a black box. The team can explain what the system does, what it never does, and what happens when it gets something wrong. In healthcare, that explanation is part of the product.
