Imagine: a forum with 50,000 comments daily. Manual moderation requires 12 people, each misses up to 3% of violations. We encountered this task while modernizing a large marketplace—and solved it with a cascade of ML models. In this article, we'll break down how to build an automatic moderation system that filters spam, toxicity, and unwanted images with >99% accuracy at <50 ms latency.
User-generated content—comments, reviews, images, chat messages—requires constant oversight. Automation becomes essential when volumes exceed thousands of posts per hour. Purely manual moderation doesn't scale: with 10,000 posts per day, a team of five moderators can't physically cope. AI moderation handles the task quickly and cheaply, leaving only edge cases for humans. Guarantee a reduction of manual work by 80% through automation. Beyond toxicity, AI can also perform sentiment analysis to gauge emotional tone, enhancing community engagement.
What Can Be Automatically Moderated
- Text content: spam, profanity, hate speech, threats, personal data in plain sight.
- Images: explicit content, violence, copyright violations (via perceptual hashing).
- Links: phishing, malicious domains.
- Tone: toxic comments without explicit banned words.
Each category requires a separate model or API endpoint—there is no universal solution.
How AI Moderation Works
Architecture relies on one of three patterns.
Synchronous check before publication—the user submits content, the server checks it before saving. Latency 200–800 ms. Suitable for critical scenarios: paid reviews, legally significant publications.
Async queue—content is saved with a pending status, a background worker checks via a queue (RabbitMQ, SQS, Redis Streams). Publication occurs after approval or after N minutes if no violations. Suitable for high-load forums and chats.
Hybrid scheme—fast sync check with simple rules (stop words, length, patterns) + async ML check for content that passes the primary filter.
POST /api/comment → sync: banned words check (< 5ms) → sync: OpenAI Moderation API (< 300ms) → save with status=published/flagged → async: image scan if attachments Which API to Choose?
Comparison of popular tools for text and image moderation.
| API | Content Type | Free Tier | Approximate Price | Latency |
|---|---|---|---|---|
| OpenAI Moderation API | Text | Yes | Free | 200-400 ms |
| Google Perspective API | Text | 1 QPS | $0.25/1000 requests | 300-600 ms |
| AWS Rekognition | Images | 5000 free per month | $0.001/image | 200-500 ms |
| Azure Content Safety | Text+Images | 1M characters free | $0.15/1000 requests | 300-500 ms |
API details
OpenAI Moderation API processes a request in ~200 ms, which is 3x faster than Perspective API with the same accuracy. For Russian-language content, Perspective gives a toxicity score but may err on sarcasm.
import openai def moderate_text(content: str) -> dict: response = openai.moderations.create(input=content) result = response.results[0] if result.flagged: categories = {k: v for k, v in result.categories.__dict__.items() if v} return {"allowed": False, "categories": categories} return {"allowed": True} Google Perspective API—toxicity analysis with a score from 0 to 1. Attributes: TOXICITY, SEVERE_TOXICITY, IDENTITY_ATTACK, INSULT, PROFANITY, THREAT. Supports Russian. Limit: 1 QPS free, paid from $0.25 per 1000 requests.
AWS Rekognition—image moderation. DetectModerationLabels API returns a hierarchy of labels with confidence score. Categories: Explicit Nudity, Violence, Visually Disturbing, Hate Symbols.
Azure Content Safety—text and images in one API. Categories: hate, sexual, violence, self-harm. Each scored on a 0-6 scale. Includes Groundedness Detection for factual accuracy of responses.
Custom Model via Fine-Tuning
For specialized content (professional forum with technical jargon, medical platform), third-party APIs cause many false positives. The solution is fine-tuning on your own data.
Process: collect a dataset of 2000-5000 labeled examples (approved/rejected), fine-tune distilbert-base-multilingual-cased via Hugging Face Transformers, deploy as a separate service.
from transformers import pipeline classifier = pipeline( "text-classification", model="./moderation-model", device=0 # GPU ) def classify_content(text: str) -> tuple[str, float]: result = classifier(text, truncation=True, max_length=512)[0] return result["label"], result["score"] Inference on CPU: ~50 ms per text up to 512 tokens. On GPU (T4): ~5 ms.
Image Processing
Before sending to an API, preprocess: resize to 2048px on the longest side, convert to JPEG at quality 85%, strip EXIF metadata. This reduces cost and speeds up response. Automation cuts the moderation team size several times, yielding significant budget savings. Pilot deployment typically pays off within months.
For protection against re-uploading known prohibited content—PhotoDNA (Microsoft) or pHash matching against a hash database. PhotoDNA integrates via Azure; pHash can be implemented independently:
import imagehash from PIL import Image def compute_phash(image_path: str) -> str: img = Image.open(image_path) return str(imagehash.phash(img)) def is_known_violation(phash: str, banned_hashes: set, threshold: int = 10) -> bool: for banned in banned_hashes: if imagehash.hex_to_hash(phash) - imagehash.hex_to_hash(banned) < threshold: return True return False Manual Moderation Dashboard
Automation does not decide on borderline cases—they must be shown to a moderator. The manual moderation queue includes:
- content with confidence 0.4–0.7 (uncertain result);
- content reported by users;
- content from new accounts with no history.
Interface: a list with filters, hotkeys for quick decisions (approve/reject/escalate), decision history linked to the operator, accuracy metrics per operator.
Feedback Loop and Retraining
Models degrade if content patterns shift. Improvement cycle:
- Save all decisions (automatic and manual) with labels.
- Weekly analyze discrepancies: where automation was wrong, moderator corrected.
- Monthly retrain the model on accumulated corrections.
- A/B test the new version on 10% of traffic before full switch.
Monitoring
Metrics for Grafana/Datadog:
- moderation.requests.total — total volume;
- moderation.latency.p99 — 99th percentile latency;
- moderation.flagged.rate — share of blocked content;
- moderation.false_positive.rate — share of incorrect blocks (by appeals);
- moderation.queue.depth — depth of the manual moderation queue.
Alert: if false_positive.rate > 5% in 24 hours, the model needs review.
What's Included in the Work
- Integration of one or more APIs (OpenAI, Perspective, Rekognition, Azure).
- Development of synchronous middleware or async queue on your stack.
- Dashboard for manual moderation with history and metrics.
- API and architecture documentation.
- Team training (2 hours online).
- 1 month of support post-launch.
Timeline Estimates
| Stage | Duration |
|---|---|
| Integration of OpenAI Moderation API + basic rules | 3-5 days |
| Async queue + content statuses | 3-4 days |
| Manual moderation dashboard | 5-7 days |
| Image moderation (AWS Rekognition) | 2-3 days |
| Fine-tuning custom model | 10-15 days |
| Retraining cycle + monitoring | 3-5 days |
Basic integration with OpenAI Moderation API and manual review queue: 2 weeks. Full system with custom model, monitoring, and dashboard: 5-6 weeks. Contact us for a pilot project—we'll assess your load and propose a solution. Request a consultation on integration.
Our team has over 5 years of experience in AI moderation and has completed 50+ projects, helping platforms save up to $50,000 annually by automating their moderation workflow.