EvalPatch

EvalPatch is a web app (with a lightweight CLI) for LLM teams who keep breaking prompts, tools, or RAG pipelines without noticing until users complain. It treats prompts, tool schemas, and retrieval configs like code: every change triggers a targeted evaluation suite, diffs results, and flags regressions with reproducible traces. You define “contracts” (expected JSON shape, refusal rules, citations required, latency/cost ceilings) and EvalPatch runs them across multiple models and temperatures. It stores versioned datasets, golden outputs, and failure clusters so you can see what actually changed and why. The product is intentionally opinionated: fewer dashboards, more actionable gates that plug into GitHub PRs and CI. It also includes red-team templates for common jailbreaks and data-exfil attempts, because most teams don’t have time to build that rigor themselves.