LLMOps & evals

In one line

LLMOps is everything around the model call that makes an AI feature shippable: evaluation suites that catch regressions, versioning of prompts and models, runtime monitoring, and cost/latency budgets.

The problem it solves

LLM features are deceptively easy to demo and genuinely hard to trust. The same prompt can pass ten hand tests and fail the eleventh; a model upgrade can silently change behaviour; a RAG tweak can improve one query and break another. "It seemed fine in the demo" is not a deployment strategy. LLMOps is the discipline that turns a fragile demo into a feature you can ship, monitor and improve.

Evals are the centre of gravity

The core artefact is the eval set: a curated collection of inputs with graded expectations, run automatically whenever the prompt, retrieval, or model version changes. Grading methods, by task:

• Exact / programmatic — where there's a right answer (a code, a JSON field).
• Rubric-by-LLM — an LLM judge scores against criteria where wording varies.
• Human review — for the safety-critical slice that can't be automated.

Without evals you can't tell improvement from regression — you're flying blind on every change.

The operational ring

Around the evals sit the production concerns:

• Tracing — log every request: prompt, retrieved context, output, user feedback.
• Canary rollouts — ship a prompt change to a slice first, watch the metrics.
• Fallback models — degrade gracefully if the primary is slow or down.
• Token & latency budgets — cost and speed are product features, not afterthoughts.
• Drift watch — yesterday's accuracy is not a property of tomorrow's deployment; inputs, models and the world all move.

Where it shows up in digital health

Anywhere an LLM output reaches a clinician or learner. For Vaidya, the eval set is explicit before launch:

• Grounding fidelity — does every claim trace to a cited Kosha entry?
• Refusal correctness — no patient-specific advice (the guardrail boundary).
• Tone-by-fidelity — depth adapts to the learner.
• Cost control — per-tier limits and monitoring live before the feature does, never after the first bill.

Common pitfalls

• No eval set — shipping prompt changes on vibes; regressions ship with them.
• Evals that don't match production — a test set unlike real traffic gives false confidence.
• Monitoring added after launch — instrument before you ship, especially cost.
• Ignoring drift — re-run evals on schedule and on every model/prompt change.

Key takeaways

• LLMOps makes AI features shippable: evals, versioning, monitoring, cost/latency budgets.
• The eval set (graded inputs) is the core — it distinguishes improvement from regression.
• Trace everything; canary changes; budget tokens; watch for drift.
• In health, the eval set encodes safety (grounding, refusal) and cost limits ship first.