Development of an AI system for automatic generation of tests and exams

Automatic test generation: where the manual process breaks

While a manual test bank contains two or three hundred items, assembly rests on a single expert: they remember what has already been asked and keep the wording in their head.

The bank outgrows that size — and gaps appear in it: topics are missing, although they are present in the curriculum.

Quality control is the first to fall apart. Duplicates in items are almost invisible to the eye: the same fact is asked in three successive versions, and the learner gets a point for memory rather than understanding.

Alongside it is inconsistency in wording: somewhere "choose one option," somewhere "select all that apply"; weights and scales differ among authors, and the same level of knowledge is assessed differently.

A coverage matrix is usually not maintained at all. To understand which sections of the curriculum are covered by items and which are empty, one can only manually go through them — that is days of work.

As a result, complex topics end up overrepresented and rare ones uncovered, and assessment stops proving anything: it measures not competence but a random set of encountered questions.

The labor intensity of review grows not with the number of learners but with the volume of cross-checking.

The expert spends most of their time finding duplicates, proofreading wording, and sorting out disputed answer keys — that is, putting the bank in order instead of assessing knowledge.

When this work runs in parallel with training, missing assessment deadlines becomes a matter of time, not a risk.

We examine how this is addressed: with a formalized competency model — Bloom levels, wording rules, and automated checking of the bank for duplicates and gaps before items are issued to the learner.

What automation delivers: measurable effects of implementation

The list below is what the customer sees in the system after launch. Each item is checked at acceptance, and if there is no measurable effect, the item is not closed — we return to the generation settings.

• Test variants are assembled many times faster. The generator outputs a ready set for a specified number of groups and difficulty levels, so the instructor spends time proofreading rather than manually assembling and rearranging items.
• The share of duplicates falls to single-digit percentages. Overlaps are checked by normalized wording and by meaning, rather than by exact string matching, so items that are essentially the same do not end up spread across different variants.
• A unified quality standard. The question structure, number of answer options, and levels according to Bloom's taxonomy are set by generation rules rather than by individual authors' habits — sets remain even in difficulty.
• The item bank scales without proportional team growth. A new subject is connected with a glossary of terms and a regulation: the bank volume grows many times over, while manual work increases only for proofreading.
• Review becomes transparent. Each question stores its source, version, and rationale for the correct answer — during an appeal, it is clear where the item came from and by which rule the variant was assembled.
• An entire class of errors disappears. Typos, missing answer options, and discrepancies between the key and the wording are caught by the validator before publication, not during the exam itself.

Item generation modes: comparison of approaches

The generation mode is chosen not by the "power" of the model but by the cost of an error in a specific item type and by the availability of a structured source.

A numerical example with parameter substitution is safely assembled by a template, while a question on understanding a topic requires a source text — otherwise it tests memory, not understanding.

At the output, any mode produces an item bank in a unified schema: wording, options, correct answer, topic and difficulty tags, and a link to the source fragment.

This removes the compatibility problem — export to QTI or flat JSON connects to your learning system without manual reformatting.

The hybrid mode usually becomes the working one: an item goes to a template or to generation based on its type, and the expert reviews not everything in a row but a sample and everything that ended up in the assessment block.

This increases the share of items that pass validation on the first attempt, and the expert is occupied with disputed wording rather than proofreading similar examples.

The main limitation of any mode is the source. On unstructured materials, the share of usable items drops, so we preserve each item's link to a text fragment: a batch can be regenerated after a study guide edit without rebuilding the bank from scratch.

How implementation proceeds: stages of engineering work

We structure the order of work so that each stage relies on the previous one already verified. Below is the sequence we go through on automatic test generation tasks: from material audit to handover to the team.

1. Audit of learning materials. We review lectures, study guides, and previous item banks, and mark topics with sufficient material and gaps. We close gaps before the start — otherwise generation produces plausible but incorrect questions.

2. Generation schema. We describe the input (topic, material fragment, target difficulty) and the item type at the output. The schema is stored as a configuration edited by the instructional designer, not the developer.

3. Setting up review rules. For closed items — reference answers and tolerance for wording; for open items — rubrics with required points and the condition for sending to manual review.

4. Test sample. We run the schema on materials it has not seen, calculate the share of items accepted by the instructional designer and the share of factual errors. Based on the results, we adjust the rules and repeat the run.

5. Variant assembly. Sets are assembled from the bank by topics and difficulty, so that variants in a group do not repeat, and any issued set can be reproduced and reviewed.

6. Handover and training. We provide a description of the schema, rubrics, and bank structure, and show instructional designers how to add topics and handle disputed reviews.

After launch, we establish a regulation for updating the bank and re-running validation after schema edits — this keeps item quality aligned with the course curriculum.

What remains in the project: items, schemas, regulations

We hand over the project as a set of artifacts that the customer's team can work with further without our involvement, rather than as a demonstration of a working service. Below is what exactly remains after launch and why each element is needed.

• Documentation set — generator architecture, item bank data model, integration points with the learning platform, list of external services and their role.
• Item flow diagram — the path from loading learning material and tagging topics to generation, expert review, publication, and delivery to the student, with checkpoint markers.
• Update regulation — how to add item types and topics, change model request templates, release versions, and roll back if quality has dropped.
• Access policies — roles of instructional designer, expert, administrator, and student: who edits the item bank, who sees logs, who exports data.
• Instructions for experts — item acceptance criteria, typical defects in wording and incorrect answer options, procedure for rejection and rework.
• Quality metrics — share of items accepted on the first attempt, topic coverage, share of duplicates; the summary is collected automatically from the review log.
• Operations regulation — bank backup, environment update procedure, and reconciliation with the curriculum after every change.

We show each artifact at acceptance and explain how to use it: documentation and diagrams must be functional, not decorative.

Case study: how manual work on the item bank was reduced

An educational center with a bank of 12,000 items updates its curriculum every semester, and variant assembly rested on four subject-matter experts.

One variant with 30 questions was assembled manually in 6–8 hours: items were selected by topic, difficulty level was cross-checked, and overlaps between variants were cleaned.

Reviewing one draft took an expert 12–15 minutes, so the bank almost did not grow — there was no one to keep it up to date.

Topic coverage in ready variants stayed at 60–70%: rare sections remained without questions because there were not enough prepared items for them. The share of repeated wording between variants reached 18% — students exchanged answers, and no one tracked overlaps systematically.

What was restructured

Generation was divided into three steps: drafts by topic map, automatic validation for compliance with the section and difficulty level, and manual proofreading.

OpenAI models paired with a separate layer of checks for duplicates and going beyond the topic became the working option.

What was achieved

Variant assembly was reduced to 20 minutes including final proofreading, and topic coverage in each variant was 100% of the curriculum.

The share of repeats fell below 2%, and expert review of one item takes 3–4 minutes: formal cross-checks moved to automation, and the human makes the substantive decision.

The class of errors "item does not match the declared topic or level" is closed at the input — such drafts are filtered out before reaching the expert.

Subject-matter experts spend the freed-up time expanding the bank and resolving disputed wording, rather than rechecking the obvious.

What to do if the item bank has already been assembled manually?

There is no one-size-fits-all answer: a bank of several thousand items is almost always cheaper to migrate than to rebuild from scratch — but only if it has formal tagging.

If items are stored in scattered files with notes inside the text, migration turns into repeated manual work and loses its point.

Migration begins with an inventory: materials are exported into a single structured form, and duplicates and defective items (incorrect key, missing answer option, outdated wording) are filtered out immediately.

Then each item is tagged by curriculum topic and subtopic, difficulty level, and type of skill being tested, after which it is matched against the curriculum.

The result is a coverage map: it shows where sections are covered in excess and where there are gaps that automatic generation will fill.

After migration, the bank lives as a single source: new items are generated into the same tagging and undergo the same checks, so curriculum coverage does not drift. This removes the main risk — a set of manual files that can neither be checked nor expanded.

Common doubts about item quality and review

Doubts about quality do not arise out of nowhere: if generation relies only on the model's free text, items drift away from the curriculum, allow double interpretation, and rely on sources that have already become outdated.

That is why our generation works not over "raw" text but over a formalized course structure — a list of topics, mastery criteria, and an approved bank of examples.

Every generated item carries a link to a specific curriculum point, not to the material "as a whole."

Then two filters come into play: automatic analysis for ambiguity and discrepancies in answer options, followed by review by a subject-matter expert who confirms correctness and resolves disputed wording.

This order provides traceability: at any stage it is clear why an item appeared and which requirement it is tied to.

What to do with ambiguous wording?

Unambiguity rules are built into the check: one correct option, no double negation, no hints in distractors, a unified style of conditions. Disputed cases are not silently discarded — they go into a report with reasoning so the expert can decide on correction or rejection.

How are outdated materials updated?

Sources are versioned: when the curriculum or regulatory framework changes, affected points are marked, and related items are reassembled. Old versions remain available for comparison, so changes in tested requirements are not lost between releases.

Who is responsible for quality after handover?

We hand over the generation rules, validation criteria, and expert acceptance regulation — they can be changed by your team without us.

Responsibility for the meaning of items on the customer's side rests with the subject-matter expert; we are responsible for ensuring that generation follows the specified rules. After handover, support and analysis of quality incidents are available.

Let's discuss the task: where implementation begins

The first conversation is needed not for a presentation but to outline the boundaries of the task: what subject, what audience, what question formats are already accepted in the organization.

From these inputs, we estimate the scope of work and assemble an implementation plan — from connecting sources to delivering ready variants. For the estimate to be substantive rather than "plus or minus," we need a short set of data from the customer:

• types of assessments and their purpose — entrance assessment, interim check, final exam, certification;
• volume and state of materials: educational texts, study guides, accumulated item bank, and the form in which they are stored;
• roles of reviewers and approvers — who approves wording, who is responsible for subject-matter correctness;
• requirements for item types: one correct option, multiple correct, open response, linkage to difficulty levels;
• internal control rules — allowable share of repeats, sensitivity to wording, reproducibility requirements;
• acceptance criteria: what we consider the result — the share of valid questions after expert review, time to prepare a variant, volume of manual correction.

Then we propose a pilot run on one topic: we run the full cycle on limited material, compare the result with your criteria, and only after that scale to the entire course. Describe the task — we will come back with a scope estimate and implementation plan.