EvalLedger

EvalLedger is a web app (with optional CLI) for teams who need defensible, repeatable LLM testing and governance without building an internal platform. It turns prompts, datasets, model configs, tool calls, and outputs into immutable, versioned “evaluation runs” tied to code commits and deployments. You define policy checks (PII leakage, disallowed content, citation requirements, jailbreak susceptibility, latency/cost budgets) and the system automatically gates releases with pass/fail thresholds. It also produces audit-ready reports: what changed, why performance shifted, which policies were violated, and who approved the exception. This is an AI app + traditional app combo: AI helps generate adversarial test cases and summarize regressions, while the core value is rigorous tracking, reproducibility, and governance workflows. Realistically, it wins only if it integrates cleanly into CI/CD and doesn’t become another dashboard nobody trusts.