AI Translation Quality Estimation Without Reference Translations

Problem: BLEU misses style and terminology errors

A translator spent two hours post-editing a translation that the system rated 0.95 BLEU. The client rejected the project due to style guide violations and incorrect terminology. BLEU compares against a reference, but the reference often ignores context, tone, and domain-specific vocabulary. We solve this with Quality Estimation (QE) — an AI system that evaluates translations without a reference, like a human reviewer. Budget savings on review can reach 60%.

How Quality Estimation Without a Reference Translation Works

Quality Estimation analyzes the source text and translation, producing a score of 0–1 at the segment, word, and document levels. Segment-level scores indicate which sentences need review. Word-level QE tags each word as OK or BAD — the reviewer sees errors immediately. Document-level QE evaluates coherence, terminology consistency, and stylistic unity.

How QE Saves 40–60% of Reviewer Time

Assume you process 10,000 segments per day. Without QE, a reviewer checks every segment. With QE, only segments with score < 0.7 (typically 20–30%). At a threshold of 0.9 for auto-publishing, 10–15% of segments skip review entirely. We implemented such a pipeline in a fintech application localization project: the reviewer handled 3,000 segments instead of 10,000, and style errors dropped by 80%. This resulted in estimated annual savings of $120,000 in reviewer costs.

Why CometKiwi Is Better Than BLEU for QE

CometKiwi (Unbabel/wmt22-cometkiwi-da) is a transformer-based model trained on thousands of human judgments. It outperforms BLEU and traditional metrics in correlation with human evaluation. Here’s a comparison of key metrics:

CometKiwi needs no reference and provides word-level errors through MQM taxonomy.

Error Types Classified by QE (MQM Taxonomy)

We use the MQM taxonomy, dividing errors into four classes:

• Accuracy — mistranslation, omissions, additions.
• Fluency — grammar, spelling, punctuation.
• Terminology — glossary violations, inconsistent term usage.
• Style — tone of voice mismatches, stylistic inconsistencies.

Common Mistakes When Implementing QE

Typical errors include: using a model without fine-tuning for the language pair (lower precision), choosing the wrong score threshold (missing errors or overburdening the reviewer), ignoring word-level QE (losing context for individual words), and not integrating MQM (difficult to improve the process).

How We Integrate QE Into Your Pipeline

1. Audit the current process: measure volume, latency, and existing metrics.
2. Model selection: CometKiwi, OpenKiwi, or fine-tuning for your language pair.
3. Integration: REST API or gRPC — wrapped in a microservice.
4. MQM taxonomy setup: connect error type classification via an LLM (GPT-4o or LLaMA 3).
5. Testing: measure precision/recall on your dataset.
6. Deployment: Kubernetes + GPU (T4 or A10).

from transformers import AutoModelForSequenceClassification, AutoTokenizer

class QualityEstimator:
    def __init__(self, model_name: str = "Unbabel/wmt22-cometkiwi-da"):
        self.model = load_comet_model(model_name)

    def estimate_segment(self, source: str, hypothesis: str) -> QEScore:
        score = self.model.predict(
            [{"src": source, "mt": hypothesis}],
            batch_size=8
        ).scores[0]
        return QEScore(
            score=score,  # 0-1, where 1 = excellent quality
            requires_review=score < 0.7,
            error_probability=1 - score
        )

    def estimate_batch(
        self, segments: list[tuple[str, str]]
    ) -> list[QEScore]:
        data = [{"src": src, "mt": mt} for src, mt in segments]
        scores = self.model.predict(data, batch_size=32).scores
        return [QEScore(score=s, requires_review=s < 0.7) for s in scores]

QE Model Comparison

What's Included in the Work

• Audit of the current translation pipeline with metric measurement.
• Selection and configuration of a QE model (CometKiwi, OpenKiwi, fine-tuning).
• Integration via REST API or gRPC with documentation.
• Error classification training for your MQM taxonomy.
• Deployment on infrastructure (Kubernetes, GPU).
• Team training and 1 month of support.

Timeline and Pricing

Timelines range from 2 to 6 weeks depending on complexity (volume, number of languages, need for fine-tuning). Pricing is calculated individually. For a typical project with 5 language pairs and 50,000 segments, pricing starts at $15,000. Get a consultation — send a description of your current pipeline, and we’ll evaluate your project within 2 days.

Why Work With Us

• Over 5 years of experience in NLP and machine translation.
• Completed 15+ quality evaluation projects for fintech, e-commerce, and software localization.
• Certified engineers in PyTorch and MLOps.
• We guarantee reviewer time savings of at least 40% or your money back.

Learn more about Quality Estimation on Wikipedia. Typical use cases include e-commerce product descriptions, financial reports, and software UI localization.

Contact us to get a consultation on implementing QE into your translation process. Order a project evaluation in 2 days — send a description of your current pipeline.