Make Every Scholarship Score Mean the Same Thing, No Matter Who Reads the Application

A 30-minute session for scholarship programs on three ways to keep a harsh or easy reviewer from deciding who wins. Every registrant gets a rubric template to fill in with your own criteria.

Reviewers disagree, even experienced ones. A 2018 study of NIH peer review published in PNAS found no agreement between different reviewers scoring the same grant proposals, even though each reviewer was consistent with their own scores.

Scholarship committees run into the same thing. One volunteer rarely gives anything below an 8. Another rarely gives anything above a 6. Both are careful, and both are reading the criteria the way they understand them. Whichever one a student's application lands with can matter as much as what the student wrote.

This session is built around two of the five dimensions of Review Integrity, a framework for judging whether an application and review process is fair, consistent and defensible.

The first is Reviewer Calibration: do reviewers score the same way, and can you see when one scores far harder or easier than the rest? Without a check, a student's rank depends on the draw.

The second is Rubric Clarity: are your criteria weighted, and does each score level have a clear meaning? A rubric that says "strong essay" leaves every reviewer to decide what strong means.

When scores drift, the strongest applicant can finish behind a weaker one who drew an easier reviewer. Decision meetings turn into debates about averages. And when a student, parent, donor or board member asks why one applicant won over another, the scores can't explain it.

The rubric, the reviewer instructions and who reads which application all get set before applications open. Whatever is in place on opening day is what your reviewers score with all season.

In This 30-Minute Session, You'll Learn How To:

• Weight your criteria and describe what a low, middle and high score looks like for each
• Run a short calibration round, with every reviewer scoring the same sample applications first
• Spread applications evenly, so each reviewer gets a fair mix
• See where each reviewer's scores fall against the group, and spot a harsh or easy scorer before results are final
• Settle large score gaps with a clear rule instead of a debate
• Keep a record of how each score and decision was reached

You'll also see how Reviewr's team builds your program from the forms and rubric you already use, in about 30 days.

What You'll Get

The Scholarship Rubric Template, with weighted criteria, score levels and a calibration checklist you can fill in for your own program.

Who Should Attend

Scholarship program administrators at foundations, nonprofits, associations, credit unions, companies and education foundations, plus the committee chairs and volunteer reviewers who score applications.

FAQ

Does adjusting for harsh and easy reviewers override what reviewers decided?No. Reviewers still score every application, and your committee still makes the final decisions. Checking score spread shows you where one reviewer's scale differs from the group, so you can account for it before ranking.

Does Reviewr send scholarship payments?No. Reviewr runs the application, review and follow-up, such as acceptance and proof of enrollment. It doesn't send payments or track fund balances. You can export a recipient report for your finance team.

‍