Calibrated judges¶
Review scores are measurements that need calibration, not authority.
Working notes¶
- Check a panel against known outcomes before trusting its ranking.
- Use negative controls that should receive poor scores.
- Separate factual verification from subjective assessment.
- Check whether reviewers discriminate or merely hedge.
- Preserve interesting scientific questions rather than only removing objections.
Source: course evidence synthesis, §2–3, §5. These are selected observations, not general performance guarantees.
Example to prepare¶
- [TODO: hypothetical grant example]
- [TODO: choose a shareable labelled calibration set and report held-out discrimination.]
Before the course¶
- [TODO: select a compact worked example and its verification artifact.]
- [TODO: rehearse the live walkthrough and prepare a fallback.]