RubricTrain

RubricTrain is a web app (with an optional lightweight desktop reviewer) that helps teams build supervised-learning datasets from subjective, inconsistent human decisions. You define a rubric (rules, edge cases, examples), then the app routes items to multiple reviewers, measures inter-annotator agreement, and pinpoints where the rubric is failing. It uses supervised models to predict likely labels and highlight “high-disagreement” items first, so humans spend time where it matters. The system continuously updates a confusion map by reviewer, category, and data slice, and generates rubric change suggestions backed by evidence (e.g., which rule causes most flips). Export clean training sets, reviewer performance reports, and audit trails for compliance. This is not a magic labeling tool; it’s a process-control layer that makes subjective labeling less chaotic and more defensible, which is the real bottleneck for many supervised ML projects.