PromptDiff

PromptDiff is a web app for teams shipping LLM features who keep breaking behavior with “small” prompt edits. It provides Git-style diffs for prompts, system messages, tools/function schemas, and key runtime settings (model, temperature, retrieval params). The core value is replay: you can run two versions side-by-side against the same saved test set (real production traces or curated cases) and get a clear, human-readable report of what changed—format drift, missing fields, safety regressions, cost/latency shifts, and tool-call differences. It integrates with common stacks via a lightweight SDK and can ingest traces from OpenTelemetry. This is not a magic evaluator; it’s a practical change-audit layer that makes prompt iteration less guessy and more reviewable, so teams can ship with confidence and avoid silent regressions.