Automated assignment checking: where rubric-based grading breaks down
An open-ended answer — an extended rationale, a code snippet, a problem solution — is not reducible to comparison with a reference answer. Checking follows a rubric: a set of criteria, levels, and weights.
The difficulty arises right here — the rubric describes quality in words, and the decision on the score is made by a human, anew each time.
Manual review of submissions scales exactly linearly: the more submissions in the flow, the more person-hours.
With thousands of answers per week, checking becomes a bottleneck — feedback turnaround grows, some submissions get a cursory score based on the first paragraph, and students' weak spots surface only after the next module, when it is too late to reteach.
Then come discrepancies. Two experts using the same rubric assign different scores for the same answer, and for one reviewer the criteria "drift" by the middle of the shift.
Without calibration, uniform grading criteria exist only in the document: the score ceases to be reproducible, the student does not understand why points were deducted, and appeals consume the time of those same experts.
Automated rubric-based assignment checking does not solve the task of "removing the teacher" but the task of removing the mechanical part: break down the answer by criteria, show the rationale with a reference to a fragment of the submission, and deliberately hand controversial and borderline cases to a human.
Below we examine how this is addressed — from rubric calibration to the threshold at which the system passes a submission to an expert.
Implementation effect: speed, score convergence, error analysis
On a flow of several thousand submissions per day, an expert spent 10–15 minutes on an extended answer: checking was limited not by the quality of the rubric but by physical time.
The system analyzes one submission in seconds and processes the nightly export in full, leaving humans only what truly requires a decision. What exactly changes — by metrics.
We calculate score convergence not by the final score but by each rubric criterion separately: this shows where the system confuses, for example, "argumentation" and "structure." After calibrating weights on a labeled sample, the share of discrepancies with the expert falls from a third of submissions to 8–12%, and they remain mainly at level boundaries.
Score transparency rests on references: each score is accompanied by the fragment of the answer the system relied on and the rubric rule that triggered.
Submissions with low confidence, a close neighboring level, or an unrecognized fragment go to manual review — its share drops from exhaustive to 15–20%, and this is already sampling-based control rather than review of the entire flow.
The system's decision history stores the rubric version, model instructions, and input data; we keep tracing in LangSmith, so any score can be reproduced even months later.
Error-cause analysis becomes a procedure: every two weeks we review where the system under- or overestimates, add controversial examples to the calibration set, and update criterion wording. A side measurable result is the share of appeals that must be revised in the student's favor.
Comparison of grading modes: automated, expert, hybrid
The first two modes differ not in model accuracy but in the type of answer and the nature of the rubric.
Full automation confidently handles short answers with formal criteria — a test, a calculation, finding an error in code — and on a flow of thousands of submissions produces a consistent score.
An extended argument or a rubric phrased as "the argumentation is convincing" runs into the limitations of automation: a score is assigned, but the system cannot explain it to the student.
Options and boundaries of applicability
| Mode | When it fits | Limitations |
|---|---|---|
| Full automation | Short answers with formal criteria: test, calculation, code, error finding. Flow from hundreds of submissions per day | Does not handle free-form text or controversial rubrics with a qualitative criterion. The score is difficult to dispute and explain |
| Sampling-based expert review | Final and graded assignments, appeals, a pilot batch for calibration | The expert reviews everything in a row — review time and score divergence among specialists grow |
| Hybrid grading mode | Mixed flow: the formal part goes to the machine, the extended part to a human | An explicit rule is needed for what goes to manual review and at what threshold; otherwise the queue grows uncontrolled |
Hybrid grading mode proves workable in most scenarios: automation handles the bulk of submissions, and the expert receives only controversial rubrics and low-confidence answers.
The threshold after which a submission goes to a human is set for your rubric, and then it is adjusted as verified labels accumulate.
How we implement the system: from rubric analysis to handover to operation
A rubric is not the assignment text but a formalized scale: criteria, weights, levels. While it lives in the teacher's head, automated assignment checking diverges from manual grading, and there is no trust in the system.
Therefore we begin not with the model but with rubric analysis: we bring criterion descriptions into the form accepted in Rubric-Based Grading — with observable indicators instead of evaluative wording.
- Rubric and disputed-case analysis. We pull up submissions from previous periods and mark places where teachers themselves disagree in grading. Such points are either clarified or removed from automation to manual review.
- Reference labeling. Two teachers independently label a sample of submissions, and a third resolves discrepancies. The output is an agreed reference against which we compare the system's score.
- Criterion-based scoring setup. We tie every decision to a specific criterion and fragment of the answer so the student sees the rationale and the teacher can check the logic.
- Threshold calibration. We select the boundaries of "pass / revision / manual review" so that the agreement rate with the teacher is high and disputed cases go to a human rather than an automatic verdict.
- Acceptance on a holdout sample. We test on submissions that did not participate in setup: agreement rate, stability across repeated runs, and the share sent to manual review.
- Handover to operation. We deliver the teacher interface, decision log, rubric version management, and instructions. After launch we monitor the share of manual reviews — its growth shows that the rubric is due for an update.
Handover package: rubrics, grading schemes, review procedures
Automated checking should not remain a black box: rubrics, thresholds, and revision rules live in documents that the client reads and edits independently, without contacting us.
The handover package is assembled during implementation, not after — each item is tested on real submissions and refined until the system reaches agreed accuracy.
- Rubric descriptions — for each criterion, the wording, weight in the final score, and examples of submissions at the upper and lower boundary: it is clear what exactly the teacher distinguishes when assigning a score.
- Grading scheme — how criteria add up to the total, which of them are blocking, and at what confidence automation stops and sends a submission to manual review.
- Review procedure — the sequence of actions for a disputed score, response times to the student, who makes the final decision, and how it is recorded.
- Discrepancy log — all cases where the system's score and the teacher's score diverged, indicating the rubric, configuration version, and decision made. It shows which criteria should be reworded.
- Access policy — roles, who sees submissions and personal data, who has the right to change rubrics, where the change history is stored, and how to roll back to a previous version.
- System documentation — input formats of assignments and answers, integration with the learning platform, and behavior with incomplete or unreadable data.
- Reference submission set — a labeled archive used to run any rubric change: it shows whether scores on past assignments have shifted.
Such a package allows you to run the system with your own team or hand it over to another contractor: grading rules are described, decision history is preserved, and changes are verifiable.
Case study: grading a flow of submissions without manual review of each
The flow of similar submissions — essays, open-ended problems, lab reports — grew faster than the reviewing staff.
Teachers reviewed each submission in full, one took an average of nine minutes, and the score discrepancy between two reviewers reached one and a half points on a ten-point scale. Feedback returned after four to five days, when the student had already moved on to the next topic.
What was changed
The rubric was broken down into criteria with formal indicators: what exactly in the text counts as met and what does not.
A model based on OpenAI assigns a score for each criterion separately and attaches the fragment of the submission it relied on — the reviewer sees the rationale.
Submissions with low confidence on a key criterion and all cases of disagreement between criteria go to manual review: automation does not take on the disputed.
Metrics before and after
- Share of manual review: 100% → 14% of submissions.
- Review time per submission: 9 minutes → 40 seconds to check the ready score.
- Feedback turnaround: 4-5 days → a few hours.
- Share of disputed scores: 8% → 2.3%, and the dispute is reviewed based on the attached rationale.
The class of errors that disappeared is "a score was assigned, but it is unclear why." Previously, during an appeal, the reviewer reconstructed the line of reasoning from memory and often revised the score; now the decision is tied to a text fragment and a specific criterion, so appeal review takes minutes rather than a full re-check.
How does automated checking differ from an answer-key test?
An answer-key test compares the answer with a reference: exact match or a list of acceptable variants. As long as the answer is discrete — a choice, a number, a short string — the key works.
As soon as the answer is open-ended, the key breaks: a correct idea in different wording, a reordering of points, or a synonym of a term produces a false "incorrect."
A rubric describes not a string but the features of an answer: which elements are covered, whether connections between them are shown, and whether there is a conclusion and rationale.
Grading of open-ended answers follows criteria with levels, and for each one a verdict is returned with a reference to a fragment of the submission.
| Option | When it fits | Limitations |
|---|---|---|
| Key, exact match | numbers, short facts, closed questions | any paraphrase is considered an error |
| Key with a list of formulations | a predictable set of short answers | the list is replenished manually, and it is still incomplete |
| Rubric with criteria | open-ended answers, problems, reasoning | agreed criteria and calibration are needed |
| Rubric with formulation analysis | the solution path and rationale for the conclusion matter | more computationally expensive; reference analyses are needed |
The answer format determines the choice. The closed part of the assignment stays on the key — these are cheap and predictable checks — open-ended tasks go to the rubric, and the score is assembled on one scale so that a mixed assignment is graded as a whole.
The rubric provides a reproducible score and analysis: it is clear which criterion points were deducted for and which formulation was not credited.
A disputed score is rechecked against the quote, and formulation analysis separately catches an answer with a correct conclusion and incomplete rationale.
Technical concerns: data, quality, responsibility
Concerns during a pilot are almost always the same: how to measure grading quality, how much labeling is needed, what happens when the rubric is changed, and who is responsible for a disputed score. Below are engineering answers for each point.
How do you measure grading quality?
We calculate agreement between the system's and teacher's scores on a reference sample — separately for each rubric criterion, not by the average score per submission.
Discrepancies greater than one scale step go into a report by error type: where there is underestimation, where level confusion occurs, and where length instead of content triggered a score.
We place ambiguous submissions in a separate set — for them the system marks "requires review" rather than guessing a score.
What are the labeling requirements?
A corpus is needed where each criterion and each scale level has examples: several hundred submissions per subject usually cover the "criterion × level" matrix.
We label not only the final score but also the rationale — it shows the decision logic and makes clear exactly where interpretation of the rubric diverges.
What happens when the rubric changes, and where is the boundary of responsibility?
The rubric remains the source of rules: if you change the wording or weights, we run submissions through the control set and see where the score shifted. The final word belongs to the teacher: low confidence and appeals go to manual review with a ready rationale.
Let's discuss your rubrics and submission flow volume
To come back with an estimate and a plan rather than generalities, we need a minimum: your rubrics and flow statistics.
We do not require a full brief — what you already have in your workflow is enough: teaching materials, examples of checked assignments, and an export from the journal for a period.
From them it is clear where a criterion can be formalized unambiguously and where expert judgment is needed, and the effort to start the project can be calculated.
In the conversation we clarify:
- current rubrics and criteria: what counts as a complete answer and what as a partial one;
- sample submissions with expert labels, including disputed cases;
- flow volume: how many assignments arrive per day and what peak load looks like;
- answer formats: text, formulas, code, files, images;
- critical errors that must not be missed, and what a false positive entails;
- where to return the result: learning platform, journal, export;
- roles: who approves rubrics, who reviews appeals, and who edits the reference.
After discussing the task, we return with an estimate and work plan: stages, timelines, acceptance metrics — agreement rate with the expert's score, the procedure for reviewing disputed cases, and the rubric retraining procedure.
We run the pilot on part of the flow so that quality is confirmed on your data before fully switching over checking.
