TraceTuner

TraceTuner is a web app (with optional CLI) for LLM ops teams that pinpoints exactly why a prompt or model change caused quality or cost regressions. It ingests production traces (inputs, outputs, tool calls, latency, token usage, user feedback) and automatically clusters failures into a small set of “regression signatures” (e.g., refusal spike, tool-call loop, JSON breakage, hallucinated fields). It then proposes targeted prompt edits and evaluation tests, and can open a PR with a minimal patch plus a reproducible test case. The product is intentionally narrow: it’s not a full observability suite; it’s a regression triage and patch workflow that plugs into existing stacks. Realistically, teams will only pay if it saves hours during incidents and reduces rollbacks. The MVP should focus on one or two providers and a tight integration path, not a sprawling platform.