Guardrails & safety

In one line

Guardrails are the defence-in-depth around an LLM: input checks, scoped capabilities, output validation and audit — so one clever prompt can't turn your tutor into a prescriber or your agent into a data leak.

Why a model can't guard itself

An LLM is helpful and suggestible — it will follow instructions, including malicious ones buried in a user message or a retrieved document, and it will answer confidently outside its remit. You cannot prompt your way to safety alone. Guardrails are the engineered layers around the model, designed so that no single failure causes harm. The principle is defence-in-depth: assume each layer will sometimes fail, and stack them.

The layers

• Scope rules (system prompt) — what the assistant is and is not allowed to do (e.g. "explain concepts, never give patient-specific advice"). The first fence — but not the last.
• Input screening — detect prompt-injection and jailbreak patterns. The RAG-era twist: injection can hide inside retrieved documents, not just the user's message, so screen context too.
• Capability scoping — for agents/tools: least privilege, schema-validated arguments, and human confirmation for consequential actions.
• Output validation — schema checks, citation-presence checks ("no source, no claim"), PII/PHI filters, and toxicity/safety classifiers before anything reaches the user.
• Audit & feedback loops — log inputs, tool calls and outputs so failures are findable and fixable, and feed them back into evals.

The OWASP LLM Top 10

The community checklist that keeps this honest names the real risks — prompt injection, insecure output handling, sensitive-information disclosure, excessive agency, overreliance, and more. Treat it as a threat model, not a formality.

The medical-advice boundary

The canonical health guardrail: an education assistant explains concepts, refuses patient-specific recommendations, and says so clearly. That boundary is Vaidya's contract in this platform — and it's enforced in layers (scope rule + output check), not left to the model's goodwill. It connects directly to AI ethics & governance: a human owns any decision that affects a patient.

Where it shows up in digital health

Any public-facing clinical assistant needs jailbreak resistance; any grounded one needs grounding enforcement (no citation, no claim); any agentic one needs capability scoping and audit. These are the difference between a knowledge tool and a liability.

Common pitfalls

• Prompt-only safety — a single system-prompt fence is bypassable; layer it.
• Screening the user but not the documents — RAG injection rides in via retrieved text.
• No output gate — the model's first draft shouldn't reach a user unchecked.
• Logging PHI carelessly — audit logs themselves must be governed (HIPAA/DPDP).

Key takeaways

• Guardrails are engineered, layered defences — not a single prompt and not hope.
• Screen inputs and retrieved context; scope tools; validate every output; audit everything.
• The medical-advice boundary is enforced in layers, with a human owning patient decisions.
• Use the OWASP LLM Top 10 as your threat model.