LLM fundamentals

In one line

A large language model predicts the next most plausible token, over and over. It is not retrieving facts, reasoning from principles, or consulting a database. Every strength and every failure mode in clinical use falls out of that single sentence.

Why the mechanism matters clinically

Most explanations of LLMs either mystify them or dismiss them. Both are useless at the bedside. The useful framing is narrow:

The model has seen an enormous amount of text and learned the statistical shape of it. Given some input, it produces the continuation that is most plausible given that shape. That is the whole trick — and it is a far more powerful trick than it sounds, because to predict text well across medicine, law and code, you must internalise a great deal of structure.

But note what "plausible" does not mean. It does not mean true, current, safe, or applicable to this patient. Plausibility and truth correlate strongly in the training data — which is exactly what makes the failures hard to spot. A confident wrong answer is not a malfunction. It is the system working as designed, on a question where plausible and true came apart.

That is the sentence to carry into every clinical AI conversation.

What follows directly

Hallucination isn't a bug to be patched. It's the mechanism, viewed from an angle where it fails. A model asked for a citation produces a plausible-looking citation — right journal style, real-sounding authors, plausible year — because that's what it does. Demanding "don't hallucinate" is asking it to be a different kind of system. Grounding it in real retrieved sources (RAG) changes the task from recall to summarise this text, which is one it is genuinely good at.

Fluency is not competence. Human beings use fluency as a proxy for expertise — it's a reasonable heuristic among humans, and it is catastrophically wrong here. The model is maximally fluent at exactly the moment it is inventing.

Training data has a date. The model does not know that guidance changed last month. It will answer anyway, in the same tone.

It has no model of the patient. It has your prompt. If the prompt omits the eGFR, the model does not know to ask.

Where it is genuinely strong

Being clear-eyed cuts both ways. LLMs are excellent at tasks where the truth is in front of them and the work is transformation:

• Summarising a document you supply.
• Restructuring free text into fields.
• Drafting something a human will review.
• Explaining a concept at a chosen reading level.
• Finding the passage in a long text that answers a question.

Notice the pattern: the source of truth is the input, not the model's memory. That is the design rule for clinical LLM systems, and it's why the architecture question ("what did you ground it in?") matters more than the model choice.

The tokens/context bit, briefly

Two mechanics you'll meet constantly:

• Tokens — text is chopped into sub-word pieces. Cost and limits are counted in tokens, not words. This is also why models are oddly bad at counting letters: they don't see letters.
• Context window — how much text it can consider at once. Large, but finite. Feed a model a 200-page chart and something falls out of view; what falls out is not announced.

The governance point

WHO's 2024 guidance on large multi-modal models makes the systemic argument: the risk isn't only a wrong answer, it's automation bias — clinicians deferring to a fluent machine, and the deference growing quietly over time. That's a governance problem, not a prompt problem, and no amount of model improvement solves it.

The practical stance: an LLM is a drafting and transformation tool with a plausible-sounding failure mode. Ground it, show its sources, keep a human accountable, and never let the tone of the output be your evidence for its truth.