Student performance forecasting: where the model breaks
Student dropout risk surfaces at the moment when the decision has already been made.
The signal is collected based on the results of the examination period: grade sheets are closed, the order is signed, and the model neatly shows those who were at risk — there is no one left to influence.
Early dropout warning requires different data: weekly attendance, timestamps of assignment submissions, results of intermediate assessments, rather than final grades.
Learning analytics breaks even earlier — at the stage of assembling performance data. Grades live in the learning platform, attendance — in the advisor's log, retakes — in the dean's office record system, and some departments keep their own spreadsheets.
The student identifier differs across systems, exports are done manually and at different intervals, so cohorts, study modes, and retroactively corrected grades get mixed in a single row.
The quality of performance data drops not because of the algorithm: the share of empty fields, duplicates, and discrepancies between systems simply was never measured.
The second gap is between signal and action.
Even when student performance forecasting honestly shows risk in the middle of the semester, the score does not turn into action: the advisor has no regulation, the instructor has no access to the forecast, and the dean's office has no clear threshold after which a tutor gets involved.
We break down how this is closed: from the data assembly scheme to the moment when the signal becomes action before the examination period, not after it.
What an early warning system delivers: the effect in numbers
The system recalculates features for each student once a week, so the effect does not appear as a one-off but accumulates over the semester.
The first signals go to advisors in the second or third week; by mid-semester, the risk group list is stable, and by the examination period there is already a history of work for each student on it.
Shifting the signal to the start of the semester
The first retake is the point after which there is almost nothing left to intervene with.
The system raises students into the risk group before it: 7-8 out of 10 people who ultimately leave due to poor academic performance get onto the list already at early checkpoint assessments — with a buffer of 6-8 weeks to work.
Previously, the signal came after debts had already accumulated, when only one chance for a retake remained.
Reducing advisor workload
The advisor used to assemble the picture manually: attendance log, grade sheets, conversations with instructors — days went into a cohort. Now they receive a list of 10-15 names with the reason: attendance, activity in Moodle, missed checkpoint assessments.
The response time to a warning is reduced from 2-3 weeks to 2-3 days, and the freed-up time goes into a conversation with the student, not into data collection.
Over the semester horizon, this yields a 1.5-2x reduction in dropout due to poor academic performance: some students clear their debts before the examination period, and some are moved to an individual plan.
In the second semester, the system works more accurately — a history appears for each student, and the forecast relies on it.
Approaches to building a forecast: comparing options
Dropout risk forecasting is built on different amounts of signal: from an export of grades to the digital footprint in the learning environment. The more sources, the more accurate the forecast — and the harder it is to explain the decision to an advisor.
| Option | When it fits | Limitations |
|---|---|---|
| Threshold rules based on grades | Only grade sheets are available, no history of graduating cohorts | Triggers late — after debts have already accumulated |
| Statistical model on historical data | Cohorts with known outcomes have been accumulated | Retraining is needed when the curriculum changes |
| Model with attendance and activity | A digital footprint exists: logins, assignment submissions | Missing data produces false positives |
| Hybrid of rules and model | An early signal and understandable justification are needed | Harder to tune thresholds |
Rules and thresholds as a baseline
Threshold rules are immediately readable: two debts in core disciplines, an average score below the passing grade, an unclosed examination period.
They do not require historical data on graduating cohorts, but they capture the problem when it is already difficult to intervene — this is the lower bound of quality.
Training on historical data
A model on historical data about graduating cohorts catches combinations that a rule cannot describe: a drop in one discipline despite strong attendance, a gap between grades over the semester.
Here interpretability is decisive — feature importance, probability calibration, and an explanation understandable to an advisor.
We check quality on a holdout sample: the share of missed risks and false positives at the chosen threshold. The hybrid connects the levels — the model provides an early signal, and thresholds keep the forecast within bounds.
Dropout risk forecasting: implementation stages
Student performance forecasting relies not on a single export but on a repeatable process. Below are six stages — from data audit to the moment when the dropout risk signal reaches advisors on schedule, not as a one-off.
Source audit and feature dictionary
- Data source audit. We verify where grades, attendance, checkpoint assessment submissions, and debts come from: how often the export is updated and whether there is data for previous semesters.
- Feature schema. We fix the dictionary: average score and its dynamics, absence rate, assignment submission timeliness, number of retakes. Each feature has a source and a recalculation rule.
- Assembly and training. We configure the model on history and check how features behave across different faculties and study modes: a single threshold for everyone creates bias.
Validation on past cohorts
- Retrospective run. We train on some cohorts and validate on later ones the model has not seen: this shows real quality, not overfitting to history.
- False positive traps. We examine erroneous labels: a one-off low grade, academic leave, a change of study mode. We tune the threshold so that the advisor receives a short list, not a hundred "alerts."
- Handover to operation. A data update regulation, an advisor dashboard, staff training, and feedback: a confirmed or rejected risk returns to the model.
What remains with the academic office: artifacts and regulations
A model without regulations lives only until the first change of advisor or curriculum edit.
That is why the handover includes not only services but also documents the academic office then works with on its own: without them, student performance forecasting remains an opinion, not a basis for a decision.
- Model passport: input features, forecast horizon, applicability boundaries, and known weaknesses.
- Data flow diagram between the dean's office, the learning platform, and the storage — with points where records are de-identified.
- Signal processing regulation: who receives the risk notification, when, and in what form.
- Access policies for students' personal data: roles, scope of visible fields, request log.
- Instructions for advisors: how to read the risk score and which decisions it does not replace.
- Retraining procedure: triggers, owner, quality validation on past cohorts.
- Change log: what changed and why, and how to roll back to the previous version.
Regulation for working with signals
A signal is a reason for contact, not a verdict. The document fixes the trigger threshold, response time, message templates for the student, and a ban on automatic conclusions about dropout: the decision is made by a human.
It separately describes what to do with a false positive — it returns to the labeling and is taken into account during the next retraining.
Access policies for student data
Access is granted by role: an advisor sees their list and the risk score without sensitive fields, the academic office sees a cohort summary, and a researcher sees a de-identified slice.
Every access to personal data is logged, exports are limited in volume and retention period, and access is revoked together with a role change.
Case study: how the model identified the risk group before the first retakes
The task came from an educational institution: a cohort of several thousand students, and the support service learns about problems only after a retake.
The model calculates risk from current grades, attendance, submitted assignments, and activity in the learning system; recalculation runs once a day, and the result goes to advisors as a ready list.
What the data showed before implementation
The only signal was the first retake — before it, the student remained out of sight. About one third of the cohort got into advisors' work: those who came forward themselves and those noticed by an instructor.
Forecast accuracy was not measured at all, the risk score was assigned retroactively, and the false positive rate reached four cases out of ten.
Up to three weeks passed from the first sign of a problem to contact with the student — during that time, part of the group managed to withdraw their documents.
How the metrics changed afterward
The Академический радар system produces a risk list every morning, and the advisor sees not only the name but also the features that contributed to the score.
The share of students identified before the first retake grew from one third to seven out of ten, and cohort coverage — to nine out of ten. False positives fell from four cases out of ten to one: some candidates are removed by the rule about debts closed in the last week.
The time from signal to contact dropped to two business days — the advisor writes to the student and schedules a meeting.
What to do if student data is fragmented and incomplete
The cause is usually not that data is absent, but that it is scattered across different record systems — the dean's office, the course portal, the attendance control system, scholarship accounting.
Each stores records with its own key and its own granularity: somewhere a row per student, somewhere per group, somewhere per class attendance event.
The first step is not model training but assembling a unified student profile: matching by a persistent identifier (student record book number, full name as an auxiliary key in case of discrepancies) and bringing all events to a single timeline, usually the academic week.
The second step is to reduce features to a minimum.
A stable forecast is provided by ten to fifteen: average score for the previous period and its dynamics, absence rate in mandatory classes, assignment submission timeliness, number of retakes, change of major or academic leave, semester workload.
Beyond that, quality hardly grows, while the risk of overfitting and the number of "why did the model decide this" questions increase.
Missing data is of two different natures, and they must not be confused. The absence of an event is a signal: the student did not log into the portal, so the probability of dropout is higher. A failed export is noise, which is not a signal.
That is why missing values are not filled with a single column average: for counters, zero is set plus a separate "no record" flag; for continuous values, the median by cohort and course; and features with a missing rate above the threshold we simply do not take into account.
The profile is updated on schedule, and each version is recorded so that the forecast can be reproduced and explained to an advisor.
If a source goes down, the model does not fail — it lowers confidence and marks the score as incomplete, and a notification is sent only for a stable signal over several weeks in a row.
Technical concerns: data, privacy, model accuracy
A forecast model looks convincing right up to the moment when a decision about a student is made based on it. Below are five questions asked before a pilot, and substantive answers, without generalities.
How does the model behave if a student has only one semester of history?
The forecast is built on a minimal window — one semester: current grades, attendance, submission deadlines.
While there is little history, the model does not produce a confident number but marks the student as "insufficient data" and recalculates the score weekly — the feature set expands as results accumulate.
The institution uses different grading scales. Will the forecast be comparable?
Grades are normalized within the course: the student's position relative to the cohort is taken into account, not the absolute score.
The five-point, hundred-point, and pass/fail systems are brought to a single scale, so a change in regulations or transfer of a discipline does not break the model retroactively.
What should be done with classes where no grades are assigned?
Attendance, submission timeliness, repeated attempts, and seminar activity are also taken into account, but with less weight than grades.
If the log is maintained uniformly, forecast accuracy is preserved; with missing data, the section is marked as unreliable rather than filled with a guess.
How is students' personal data protected?
De-identification happens at the input: the model works with an identifier, while the link to the name is stored in the customer's environment. Access is role-based — an advisor sees their own students, an analyst sees only aggregate slices, and exports are aggregated.
Does the system replace the advisor?
No. The model ranks the group and shows which disciplines and events contributed to the risk — this is a guide for where to look first.
The conversation with the student and the decision remain with the advisor, and we examine false positives and take them into account when tuning thresholds.
Let's discuss the task: what the first month of work looks like
To come back with a realistic estimate, at the start we need to understand three things: what student data is already being accumulated, where it resides, and who in the organization makes the decision on the risk group.
Usually an export of the grade log, attendance, and checkpoint assessment results for several semesters is enough — this is sufficient to check whether the data contains a stable signal for forecasting rather than noise.
Then we come with a specific package:
- Source schema: a list of tables and fields, update frequency, missing data, and how it is processed before model training.
- Stage plan: from labeling historical semesters to putting the forecast into the production loop, with points where your team joins acceptance.
- Pilot on one cohort: the model trains on previous intakes and is validated on the current one, while the academic process remains unchanged.
- Timeline estimate: for each stage, with assumptions — what must be ready on your side and by when.
- Composition of handover materials: code, feature description, retraining regulation, and instructions for advisors who work with the forecast.
- Acceptance metrics: how we measure the quality of student performance forecasting and what share of the risk group must be found before the checkpoint.
Describe the task: what data is available and which process you want to support — early warning, tutor allocation, or reporting. We will come back with a timeline estimate and a stage plan.
