Developing an AI Tutor for Personalized Learning on LLMs

The AI tutor doesn't scale: where personalization breaks

The AI tutor does not work as a chat with ready-made answers, but as a system that assembles context at every step: the student's profile, attempt history, completed topics, and the rubric of the current assignment.

The more accurately this context is assembled, the higher the quality of the tutor's answers — the explanation lands in a real gap rather than a generic textbook section. This is exactly what personalized learning rests on, and this is exactly where the limitations begin.

Everything breaks on long courses. The profile grows, attempt history runs into hundreds of events, and only part of it makes it into the request — the rest is discarded based on formal criteria.

Answers converge on the same explanation for different students: personalization degenerates into a retelling of the module, and the student stops getting a response to their specific mistake. Externally the system works; substantively, it no longer does.

The second bottleneck is the load on instructional designers. Student error analysis cannot yet be fully delegated to the model: rubrics, reference solutions, and the boundaries of acceptable wording have to be maintained manually.

Instructional designers revise the same unsuccessful answers, update reference solutions after every program change, and do not have time to check whether the assessment has drifted from the course's intent.

As the flow grows, both bottlenecks compound: the more students there are, the more edits are needed and the less time remains for spot checks.

Weak points are discovered only after the module has been completed — through complaints and final grades, when it is too late to retrain and trust in the tutor is already lost.

Next, we break down how this is addressed: what context to assemble, how to check answers automatically, and where to deliberately keep an instructional designer in the loop.

Implementation effect: response speed, accuracy, load on instructional designers

Implementing an AI tutor is measured not by the number of dialogues but by four shifts: the share of correct analyses, lesson preparation time, the volume of manual edits, and system behavior under peak load. Below is what exactly we check at acceptance and which values we consider a working result.

Accuracy of analysis on course materials

The tutor answers not from a general encyclopedia but from the program of a specific course: methodological materials, solution examples, and assessment criteria are loaded into the index, and every answer comes with a source link.

We measure analysis accuracy on a held-out sample of real student work — those already checked manually by instructional designers and labeled as reference. Before implementation, such verification is done selectively and rarely rises above two-thirds.

After configuring analysis scenarios, dividing material into semantic blocks, and separately testing on edge-case tasks, the share of correct analyses reaches 85–90%.

The remaining discrepancies relate to disputed wording in the assignments themselves — that is already grounds to fix the conditions, not the model.

How much manual work is removed from instructional designers

Automated checking takes over the routine: assignments with a formalizable answer — tests, calculations, problems with solution runs — are checked immediately, leaving the instructional designer only with works where the tutor has flagged ambiguity.

The share of manual edits in checked works drops from the majority to isolated cases, and this is visible in the log: every intervention is recorded with a reason.

Lesson preparation time is reduced from several hours to a short proofread: the tutor assembles a draft analysis, a selection of examples, and a list of the group's typical mistakes; the instructional designer edits the wording and adds what only they know.

Under peak load — before a deadline or exam — the system maintains a request queue and distributes it across service replicas, so answers arrive without quality degradation and without manual firefighting by instructors.

AI tutor formats: when each mode fits

The tutor mode is chosen based on where the knowledge resides and how often it changes, not on model size.

For general topics, an off-the-shelf model is enough; for answers based on internal documents, knowledge base search is needed; for skill practice, a scenario with step assessment. The difference is visible in data requirements and in what the customer receives.

Working with an off-the-shelf model vs. with your own materials

An off-the-shelf model starts without data preparation: the tutor maintains a dialogue and answers general questions but does not know your terminology.

Fine-tuning on your own materials removes this — the tutor reproduces the accepted analysis style and operates with your definitions.

The price is a corpus of approved question–reference answer pairs; updating knowledge requires retraining, so this mode suits a stable methodology rather than changing regulations.

Boundaries of the knowledge base search mode

Knowledge base search answers with facts from specific documents and leaves a source link — knowledge is not hardwired into the model but inserted into the request.

Hybrid mode combines both approaches: the error analysis style comes from fine-tuning, and facts come from the base.

The boundaries of the mode lie in the quality of document preparation: if they are not split into semantic fragments and labeled with metadata (role, department, version), the system retrieves irrelevant chunks and builds an answer from them.

Scenario-based simulators stand apart: branching logic is defined manually, and the tutor assesses step completion.

Implementation stages: from material audit to handover

Implementing a tutor begins not with code but with an inventory of what it should teach. Each stage below is closed with an artifact that is checked before moving to the next — this way quality discrepancies are caught at the input rather than after launch.

Material audit and labeling

1. Material audit. We collect notes, lesson recordings, an assignment bank, and internal regulations; we mark contradictions, duplicates, and topics without sources. The result is a coverage map: which topics the tutor can answer based on the material, and where authored text is needed.
2. Data labeling. We bring everything to a unified form and label it: topic, difficulty level, assignment type, typical mistakes. Clean labeling is what personalization is built on: the tutor selects the next assignment based on a specific gap, not on an average score.
3. Interaction schema. We fix dialogue scenarios: how the tutor clarifies the goal, how it guides through a chain of topics, what it does when uncertain. We separately describe an honest refusal — if the material is absent, the tutor says so rather than constructing an answer.

Testing on real scenarios

• Configuring answer logic. Together with instructional designers, we assemble reference questions and reference answers: wording, acceptable variations, and pass criteria. This is both configuration and a future measurement baseline.
• Acceptance testing. We run the reference set and look at the share of correct answers, the share of justified refusals, and the correctness of difficulty selection. We analyze and fix discrepancies until the metrics match the agreed ones.
• Pilot on a group. A limited flow of students works with the tutor in normal mode; dialogue logs show where it loses context or goes into unnecessary explanations.
• Team training and handover. For instructional designers — how to add materials and update reference answers; for support — how to read logs and quality metrics. We hand over the configuration, interaction schema, set of checks, and documentation.

Handover package: configurations, schemas, regulations, labeling

Delivering an AI tutor is not a demonstration of a working chat but a package that the customer's team maintains further on its own.

Everything configured for the subject domain is recorded in artifacts: the tutor configuration, data flow schema, reference question set, material update regulation, and access rules.

Below is what exactly is included in the handover and by what criterion it can be checked at acceptance.

Documentation and update procedure

• Data flow schema: the path from a student's question through knowledge base search and answer generation to log recording, with notes on where dialogue history is stored and how it is de-identified before export.
• Tutor configuration: system instructions, answer templates, the confidence threshold below which the tutor says "I don't know" instead of guessing, and source citation rules.
• Reference questions: a set of question–expected answer pairs linked to course materials. Running them is a regression before every update: you immediately see what has degraded.
• Update regulation: who adds material to the knowledge base, in what order quality is checked after upload, and how an unsuccessful update is rolled back without affecting the rest.
• Support documentation: where configurations are stored, how to bring up the environment on a staging server, what goes into logs, and which metrics alerts are tied to.

Access rights and topic boundaries

• Roles: administrator, material editor, instructor, observer — each has its own set of operations; the editor cannot change system instructions.
• Topic boundaries: a whitelist of subject domains, prohibited topics, and a response scenario when going beyond them — instead of free generation.
• Data access: separation of learning and personal data, dialogue retention period, and deletion procedure on request.

The package is considered accepted when the customer manually performs one test material update, runs the reference questions, and rolls back the change.

Case study: increase in completed assignments after scenario rework

An online school with several dozen courses launched an AI tutor as the first line of support: it answered questions about the material and checked homework.

The tutor worked on top of the school's learning platform (Moodle), into whose environment the current branch model was integrated.

How it was before: template answers and drop-off in the middle of the course

The answer was assembled by keyword search: the student received a paragraph from the study guide even when asking about a specific mistake in their solution.

On composite assignments in the third and fourth modules, completion dropped from 71% to 39%, and instructors manually rewrote 6 out of 10 tutor answers — the model saw neither the student's previous attempts nor the checking criteria for the specific assignment.

What changed: metrics before and after

The scenarios were reworked: the module was split into short steps with a comprehension check after each, attempt history and assessment criteria were added to the answer context, and error analysis was built from the student's solution rather than from the study guide text.

When confidence is low, the dialogue immediately goes to the instructor — instead of a template brush-off.

After three months, the share of independently closed assignments more than doubled, and instructor workload shifted from editing answers to analyzing complex cases. Personalization delivered results not through model size but through what goes into the answer context.

Off-the-shelf model or fine-tuning: what to choose for the task?

Short answer: an off-the-shelf model covers an AI tutor that explains a standard course, checks answers against a reference, and maintains a dialogue.

Fine-tuning is needed where the value lies in your methodology: your own terminology, your own error analyses, your own order of explaining a topic.

Analysis quality diverges specifically on narrow topics. An off-the-shelf model confidently retells general material but conflates similar concepts and inserts plausible yet incorrect source references.

Fine-tuning on verified question–reference analysis pairs reinforces your terminology: the model less often drifts into general reasoning and more accurately shows where the mistake is in the student's solution.

Material preparation is the main dividing line. For an off-the-shelf model, it is enough to describe scenarios, assemble a course knowledge base, and configure checks: launch timelines are measured in weeks.

Fine-tuning starts with a corpus — several thousand verified dialogues, unified labeling, and rejection of contradictory examples; collection and cleaning usually take more time than the training run itself.

Solution maintenance also differs. An off-the-shelf model lives on the vendor side: updates change the tone and completeness of answers, so regular control question runs are needed.

A fine-tuned version is your artifact: behavior is reproducible, but new subjects and methodology changes require retraining.

A hybrid more often wins: the off-the-shelf model maintains the dialogue and general language, while fine-tuning is responsible for the narrow methodology layer and analysis of typical mistakes.

According to the findings of the Review of language model adaptation practices and our experience, this option delivers quality close to full fine-tuning with a noticeably smaller amount of material preparation and a shorter launch time.

Technical concerns: data, answer quality, privacy

Implementing an AI tutor rarely comes down to the model itself — questions arise about data, boundaries, and maintenance. Below are five technical concerns that we resolve before development starts.

What should be done if the tutor confidently answers incorrectly?

Generation is limited to the knowledge base: the answer is assembled from retrieved fragments with a source link. If there are no relevant fragments, the tutor reports that this is not in the materials instead of guessing.

The share of answers with a confirmed source and the number of escalations to an instructor are visible in the log.

Can the topics the tutor answers be limited?

Yes, allowed topics are set as a list at the configuration level: a request outside the list receives a polite refusal and a suggestion to contact the instructor. For sensitive courses, we add wording rules and answer validation before output.

Will the solution withstand growth in student flow?

Request processing scales horizontally, and the knowledge base is cached, so growth in concurrent sessions does not require redesign. A load test on a peak-week profile will reveal bottlenecks in advance — before opening the flow.

How can knowledge be updated without stopping the service?

The new version of the knowledge base is indexed in parallel and connected by a switch; the previous one remains available for quick rollback. Changes are tested on a control question set: if quality drops, the release does not go into production.

Let's discuss the task: we'll select a mode and return with a plan

A domain-specific conversation begins not with a model demonstration but with your materials.

The more specific the input data, the fewer assumptions we build into the project estimate and the more precise the implementation plan — without the phrase "we'll clarify as we go."

• 2–3 learning scenarios with real student questions: error analysis, exam preparation, practicing a topic from scratch. They show where the tutor must ask a clarifying question and where it should explain right away.
• 10–20 reference questions with expected answers. From them we assemble acceptance checks and determine how strictly the answer must rely on your sources.
• Data constraints: which materials may be used, whether personal data is present, what cannot be transferred to external services, and whether isolation by student groups is needed.
• Metric expectations: the share of correct answers on the reference set, time to first useful answer, and the share of "I don't know" refusals instead of an invented fact.
• Current infrastructure: learning platform, material formats (notes, PDF, video), student authorization method, and who maintains and updates the content.
• Roles and accesses: who sees dialogues, who edits the knowledge base, whether answer moderation and a tutor action log are needed.

In response, we provide a project estimate broken down by stages, an implementation plan with checkpoints, and the boundaries of the first version: which scenarios we cover immediately and which in the second iteration.

If some materials are missing, we will suggest what to replace them with during the pilot so as not to shift the start.