Inputs
Prompts, responses, traces, policies, professional standards, known failures, current rubrics, expert requirements, and customer platform rules.
← All data programs
Judgment your model can learn.
Convert professional standards, model failures, and qualified human decisions into demonstrations, preferences, rubrics, and held-out evaluations with a traceable review path.
- artifact
- graded response + rationale
- domain gate
- buyer-approved qualification
- criterion
- factual support / weight 3
- judgment
- partially meets standard
- review
- second expert + adjudication
- provenance
- task, rubric, roles, versions
Buy the artifact, not a vague expert pool.
Each program names the required qualification, work unit, review method, output, and acceptance question before recruitment or production begins.
Managed work
Expert sourcing and verification, demonstrations, reference answers, preferences, rubrics, evaluations, red teaming, and agent workflow grading.
Outputs
Task sets, judgments, rationales, weighted criteria, graded references, held-out evaluations, failure taxonomies, and qualification manifests.
Quality
Production-like qualification, calibration, separated author and reviewer roles, blinded comparison where useful, agreement analysis, and adjudication.
Delivery
Tasks completed in the approved environment or exported in the buyer's schema with rubric versions, role provenance, and a QA report.
One expert system, several artifacts.
The same standard should connect what teaches the model, what grades it, and what exposes its failures.
Teach with strong examples
Author prompts, reference responses, demonstrations, counterexamples, tool-use traces, and structured rationales when the program requires them.
Compare model behavior
Create preference pairs, pointwise judgments, error labels, and reasons against stable criteria rather than reviewer taste alone.
Define what good means
Build weighted rubrics, graded examples, evaluator instructions, acceptance rules, and adjudication guidance from buyer standards.
Measure and challenge
Create held-out tasks, regression sets, policy checks, domain red-team cases, and grading for agent actions or completed work products.
Qualify against the real task.
Credentials can open the gate. Production-like work and calibrated review determine who stays in the program.
- 1.0
Define the judgment
Name the model behavior, artifact, expert requirement, policy boundary, and acceptance method.
Output: task and expert specification - 2.0
Inspect the failures
Review representative prompts, responses, traces, current guidance, and systematic disagreement.
Output: failure and ambiguity map - 3.0
Qualify and calibrate
Verify the agreed signals, run a production-like task, compare decisions, and revise the rubric.
Output: approved reviewer pool and sample - 4.0
Produce with review
Separate authoring, review, and adjudication where the risk requires it, then track agreement and corrections.
Output: reviewed expert artifacts - 5.0
Validate the handoff
Check schema, task coverage, duplicates, leakage risks, rubric versions, and delivery in the approved environment.
Output: artifact set, QA report, and provenance
Make disagreement useful.
Expert disagreement can expose ambiguity in the task, gaps in the rubric, or a real split in professional judgment. It should not be averaged away without inspection.
- Identity, education, licence, work history, portfolio, or task skill checked only as the program requires.
- A qualification task that resembles the production artifact and review burden.
- Shared calibration examples with a versioned rubric and recorded decision changes.
- Blinded or randomized comparisons when presentation order or source identity could bias judgment.
- Checks for ambiguity, leakage, shortcut cues, duplicates, and drift after model or instruction changes.
Representative sample
A judgment tied to its standard.
This example shows the delivery shape. The actual discipline, criteria, evidence, identity policy, and acceptance rule come from the engagement.
criterionfactual supportrubric version 0.3
weight3buyer-defined priority
judgmentpartially meetsevidence span attached
reviewdisagreement adjudicatedreason recorded
decisionaccepted after revisionreview roles attached
Questions before a pilot.
Start with one failure cluster and the qualification needed to judge it correctly.
How are experts qualified?
The program defines acceptable evidence, then uses a production-like qualification task and calibration. A credential alone does not prove task performance.
Will expert identities be shared?
The engagement defines the provenance level. Qualification and review-role evidence can be delivered without exposing personal identity where privacy or safety requires de-identification.
Can experts work in our platform?
Yes, when the customer environment supports the required access, contributor, review, and audit workflow.
How do you handle rubric drift?
Rubrics are versioned. The team records why criteria change, re-calibrates reviewers after material changes, and keeps production artifacts tied to the version used.
Start with one failure cluster.
Receive a sample task set, draft rubric, reviewer plan, and pilot price.