EvalSentry

EvalSentry is a web app (with optional CLI) that continuously tests LLM features in production-like conditions and alerts you when quality drifts. Instead of one-off benchmark runs, it schedules evaluation suites against your real prompts, tools, and retrieval sources, then tracks pass/fail, cost, latency, and safety metrics over time. It supports golden sets, adversarial tests, and “canary” traffic sampling so you can compare new models, prompt changes, or RAG index updates against a baseline. The product is opinionated: it focuses on fast setup, minimal labeling, and actionable alerts rather than endless dashboards. It integrates with CI (GitHub Actions) and observability (Datadog) so failures show up where engineers already work. This is an AI app + traditional app combination: it uses LLMs to generate hard test cases and to summarize failure clusters, but the core value is reliable evaluation automation.