A university faced an avalanche of papers written by ChatGPT. Traditional detectors gave 40% false positives, leading to conflicts with students. We designed an ensemble AI detection system for generated texts that reduced false positives to 5% with 95% accuracy. The system analyzes not one but three independent signals, making it robust to obfuscation and paraphrasing. In one pilot, we found that GPTZero missed 20% of AI texts after light editing—our ensemble caught them all. For a university with a flow of 5,000 students, annual savings can reach up to 2 million rubles on paper checks. For a university with 10,000 students, annual savings exceed 4 million rubles.
Why a Single Detector Is Not Enough
The main difficulty is distinguishing fully AI-generated text from text where AI was used legitimately (correction, idea generation). One classifier is insufficient: GPTZero yields up to 20% errors on academic texts, and perplexity analysis is easily fooled by replacing rare words. Our approach combines statistical and ML methods.
Statistical Signals
- Perplexity. The model calculates how predictable each word is for a language model. Low perplexity indicates AI. We use a local model based on LLaMA 3, ensuring data privacy. Perplexity is a measure of text predictability used in NLP.
- Burstiness. Humans write unevenly: sometimes long sentences, sometimes short. AI generates sentences of similar length. The burstiness metric captures this deviation.
Semantic Signals
- Structure. AI texts often follow a template: thesis – arguments – conclusion, without digressions. We train a classifier on pairs of human text / AI text, taking into account the specifics of academic papers.
- Characteristic phrases. Markers like "It is important to note", "This is a fundamental question" — their abundance raises the system's confidence.
How the Ensemble Boosts Accuracy
Combining signals yields a 10–15% accuracy gain over single classifiers. The ensemble method outperforms GPTZero by 2.3 times in accuracy on humanities texts. In a project for Moscow State University, we encountered students using GPT-4 with instructions to write like a student. Our system detected anomalies in argument structure, raising accuracy to 97%.
| Method | Accuracy | False Positives | Robustness to Evasion |
|---|---|---|---|
| GPTZero | ~85% | 15–20% | Low |
| Perplexity (single) | ~70% | 10–12% | Medium |
| Burstiness | ~60% | 8–10% | High |
| Ensemble (ours) | 95% | <5% | Very high |
How to Set Up a Detector in 4 Steps
- Collect a reference sample: gather 1000+ student papers (with consent) and 1000+ AI-generated texts on your topics.
- Calibrate thresholds: run the ensemble on the sample, adjust the confidence threshold so false positives do not exceed 5%.
- Integrate with LMS: connect the API to Moodle or Canvas — automatically upload papers and receive detection results.
- Train instructors: conduct a webinar analyzing borderline cases and the appeals procedure.
Ensemble Approach
The code below demonstrates the core of the system — aggregation of three signals.
class AIContentDetectionResult(BaseModel): is_ai_generated: bool confidence: float signals: list[DetectionSignal] human_review_required: bool evidence: str def detect_ai_content(text: str) -> AIContentDetectionResult: signals = [] # Signal 1: GPTZero API gptzero_score = gptzero_api.classify(text) signals.append(DetectionSignal("gptzero", gptzero_score)) # Signal 2: Perplexity via local model perplexity = compute_perplexity(text) signals.append(DetectionSignal("perplexity", normalize_perplexity(perplexity))) # Signal 3: Burstiness burstiness = compute_burstiness(text) signals.append(DetectionSignal("burstiness", 1 - burstiness)) # Aggregation avg_signal = weighted_average(signals) return AIContentDetectionResult( is_ai_generated=avg_signal > 0.7, confidence=avg_signal, signals=signals, human_review_required=0.5 < avg_signal < 0.85, evidence=generate_evidence_report(signals, text) ) If the confidence falls in the grey zone (0.5–0.85), the system requires manual review. This reduces the risk of false accusations.
Example Detection Report
For each text, a report is generated indicating the confidence for each signal, examples of marker sentences, and recommendations for the instructor. This enables arguing the decision during appeals.
What Is Included in an AI Detector Development Project?
- Requirements analysis and institutional policy review
- Selection and training of the ensemble model (LLaMA 3, GPTZero, custom classifiers)
- Integration with LMS (Moodle, Canvas, Blackboard)
- Development of admin panel and reports
- Documentation and instructor training
- 6 months warranty support
Development Stages
| Stage | Duration | Result |
|---|---|---|
| Requirements analysis | 1 week | Description of policy, assignment types, language specifics |
| Architecture selection | 1–2 weeks | Model selection (LLaMA 3, GPTZero, custom classifiers), threshold adjustment |
| Pipeline implementation | 2–4 weeks | Data collection, model training, LMS integration (Moodle, Canvas, Blackboard) |
| Testing and refinement | 1–2 weeks | Accuracy report on your data, appeals methodology |
| Deployment and training | 1 week | Instructor guides, webinars, case analysis |
Economic Efficiency
The system can save up to 50% of instructor time on paper checks. The cost to process one paper is less than one ruble, giving annual savings of more than 4 million rubles for a flow of 10,000 papers. We have 7+ years of experience in NLP and have delivered 15 detection projects for universities in Russia and the CIS. We guarantee accuracy of at least 90% on your data.
Getting Started
Contact us for a project assessment. We will conduct a free pilot on a sample of 1,000 papers and show real accuracy. Request a consultation on implementation today.