Text Sentiment Analysis Implementation

Text sentiment analysis: where accuracy is lost

Feedback comes from support, marketplace reviews, surveys, and correspondence with managers, and each channel has its own evaluation format. Manual review labeling cannot keep up with the flow: while annotators process a weekly export, a new one arrives.

Some messages remain in the backlog without a label, and the overall sentiment picture is assembled from fragments.

Accuracy is lost at the very input — when there is no single rating scale. One annotator considers "neutral" only a statement of fact, another includes a restrained complaint, and a third includes a question without emotion.

Classification errors of requests accumulate not in the model but in the labels: messages identical in meaning receive different signs, and training proceeds on contradictory examples.

Then the effect compounds: feedback analysis shows growth in negativity where there is none, and disputed examples go back to the manual stage.

Open corpora like RuSentiment help only partially — their labeling works as a reference, but not as a ready-made scale for a specific product, its vocabulary, and customers' slang.

This is closed by labeling discipline and regular agreement checks, not by changing the model. We break down how this works and exactly where accuracy is lost in text sentiment analysis.

What well-established feedback analysis provides

A well-established feedback analysis changes not the tool but the mode of working with it: instead of selective reading, the entire flow is processed.

The share of processed requests rises to full coverage, and the reaction speed to a surge of dissatisfaction is measured in minutes — from the appearance of the first negative messages to a signal to the responsible person.

The share of manual processing drops predictably: the annotator gets involved only in disputed cases — sarcasm, mixed sentiment in one review, industry slang. The rest is parsed automatically, and the manual review queue shrinks by multiples, not by percentages.

Monitoring customer dissatisfaction is built on the threshold of the negative share by channel and on the rate of growth of that share.

If within a short window negativity exceeds the usual background, an alert goes to the business owner — before the topic spreads to public platforms.

A consolidated score by channel provides a comparable scale: reviews, support correspondence, and comments are brought to a single form. Then it is clear where the problem is local and where it is systemic.

A separate class of missed complaints also disappears — those that did not make it into the sample and remained unanswered.

All these metrics are fixed before work starts: the share of automatic processing, time to signal, share of disputed cases in manual review. The result is also accepted by them — without a subjective "it became more convenient."

Comparison of approaches to review analysis

Review analysis begins with choosing a method, and there is no universal winner: accuracy depends on text volume, domain, and the cost of an error.

The dictionary approach delivers results on day one, but breaks on irony, slang, and constructions like "not bad, but I won't order again." Training on labeled data removes some of these problems — and shifts the costs to corpus labeling.

Fine-tuning a language model handles context, abbreviations, and typos better, but requires examples specifically from your domain and predictable infrastructure for regular retraining.

A hybrid classification scheme covers edge cases: rules handle critical topics and service formulations, while the model handles the rest of the flow.

We choose the option based on your data: we look at whether there is a labeled corpus, how quickly the vocabulary changes, and what happens when an error occurs — the customer leaves or an extra support request appears.

If a review analysis system is already working, a hybrid scheme most often wins: rules are added to the current model, and there is no need to rebuild everything from scratch.

Stages of implementing feedback analysis

We start not with model selection but with an audit of what already exists: request exports, reviews, and manual labeling history. Here it also becomes clear whether there is enough data for the required number of classes and where annotators disagreed with each other.

1. Labeling audit. We look at data volume and distribution, the share of duplicates, and agreement of ratings between annotators. From the discrepancies, it is clear which wording of the instructions needs to be rewritten before training.
2. Sentiment class scheme. We fix the set of classes: usually positive, negative, neutral plus domain categories such as delivery complaints. We do not take an overly granular scheme — accuracy drops on rare classes.
3. Processing setup. We combine text preparation, training, and model application into one reproducible scenario: cleaning, normalization, slang, typos, short replies, mixed languages.
4. Validation on held-out examples. We calculate metrics for each class separately, not only on average. We run texts from a period the model has not seen and analyze the confusion matrix: where neighboring classes get confused.
5. Error analysis and fine-tuning. Frequent mistakes are returned to labeling, and the cycle repeats until the gain on the holdout set becomes insignificant. We monitor drift: an increase in the share of neutral responses or new vocabulary signals the need for fine-tuning.
6. Handover of the solution to the team. We provide labeling instructions, the class scheme, training and application code, a dashboard with quality metrics, and a retraining procedure. We train employees and stay in touch during the period when the system runs independently.

Documentation package and labeling rules

A model without documentation is a black box: if the contractor leaves, the customer can neither label a new corpus nor explain why a specific review was classified as negative.

Therefore, labeling rules, the class scheme, and procedures are handed over as part of the solution, not as an appendix to it. Below is the package that remains within your team and allows you to develop sentiment analysis on your own.

• Labeling rules — instructions with edge cases (sarcasm, double negation, mixed sentiment), label priority in case of conflict, and requirements for agreement between annotators.
• Class scheme — label hierarchy, confidence thresholds, and the "class → action" link: where we respond automatically, where we bring in an operator, and what we consider noise.
• Configured processing routes — which flows go in real time, which are batched, where low-confidence responses go, and how they enter fine-tuning.
• Quality control procedure — the share of documents for manual review, the procedure for analyzing discrepancies between reviewers, metrics for each class, and deadlines for revising labeling.
• Model update procedure — retraining triggers (vocabulary change, accuracy drop on fresh data), the composition of the training set, and who approves the rollout.
• Integration map — request and response formats, behavior when the service is unavailable, and rollback to the previous version.
• Decision log — which hypotheses were tested, why the current confidence threshold was chosen, and what was rejected.

The documents are written for your domain and your data flow, not copied from a general instruction — that is why a new annotator reaches consistent labeling within days.

Case study: how accuracy grew on a flow of requests

Starting point: a flow of requests from several channels was processed manually, while automation worked on dictionaries and rules.

The rules confidently caught explicit complaints but broke on negations, irony, and short replies — the share of correctly processed requests stayed around 61%, and the review queue grew faster than the team could close it.

The process was rebuilt: a sample was labeled by channels and request types, disputed examples were aligned between annotators, a model was trained on pretrained language models from Hugging Face, and confidence thresholds were calibrated separately.

The "neutral" class stopped pulling in borderline cases, and low-confidence responses go to manual review instead of the general report.

What the metrics showed before and after

• Share of correctly processed requests: 61% → 92%, on irony and negations — from 48% to 86%.
• Incident response time: from 40 minutes to 6 — a surge of negativity is visible in the first minutes, not the following week.
• Manual checks: from ~1800 to 260 per week, and the freed-up time went to analyzing causes, not consequences.

Along the way, bottlenecks emerged that simply were not visible with dictionaries. Transliteration and language mixing within one request, composite messages with two topics at once, and drift in wording after product changes.

For these, text normalization, splitting composite requests into separate records, and drift monitoring with scheduled retraining were added.

Every model update goes through a shadow run on historical data and can be rolled back to the previous version — accuracy does not drop unnoticed. Sentiment analysis stopped being a routine that had to be kept up with and became a source of data about the causes of requests.

How is sentiment different from the topic of a request?

Sentiment and topic are two independent layers of analyzing a single request, and they are constantly confused. Assessing customer attitude answers the question "how are they saying it": positive, negative, neutral, and for complaints — how harshly.

Topic classification answers "what are they writing about": payment, delivery, product quality, support performance.

The data for setting up these tasks is different.

For attitude assessment, you need labeling of emotional tone with attention to negations, irony, slang, and mixed formulations — "thanks for making me wait a third time" is formally gratitude but essentially sharp negativity.

For topic classification, you need a taxonomy: a hierarchy from large groups to subtopics, rules for requests about several things at once, and a separate "topic not determined" class, otherwise the model will stretch any text onto the nearest rubric.

The validation methods also differ. Sentiment is evaluated on the rare class: negativity in the flow is usually a smaller share, so we look at accuracy specifically on it, not the overall share of correct answers.

Topic is checked by taxonomy coverage — how many requests are confidently assigned and how often the model confuses neighboring topics, for example "return" and "warranty."

The practical cost of confusion is a model trained on topics starts outputting "negative" for any mention of a problem. As a result, a factual message without emotion goes into the red zone, while a polite complaint with a threat to leave does not.

In working analysis, the topic determines the request route, while sentiment determines priority and a sign of churn risk.

We set up these tasks separately, on their own samples and their own metrics, and connect them only at the output. Then a failure in one part does not drag down the second, and each can be fine-tuned without rebuilding the entire analysis.

Disputed cases and limitations of the method

Sentiment analysis works confidently where the assessment lies on the surface of the text and noticeably underperforms on real reviews.

Irony, slang, fragments of phrases, and typos shift the prediction: the reply "well thanks, you really helped" is formally positive, but in meaning it is a complaint.

Below are disputed cases that most often come up during acceptance and how they are handled at the level of labeling, dictionaries, and post-processing rules.

What should be done with mixed and short replies?

We split a mixed review into segments and evaluate it step by step: "delivery is fast, but the packaging is crushed" yields two classes and a neutral overall result tied to the aspect — delivery or quality.

Short replies like "ok" and "fine" are not pulled into the overall conclusion but marked as low-information, otherwise they blur the distribution of ratings.

How robust is the system to typos and changes in wording?

We suppress typos at the normalization stage: an abbreviation dictionary, rules for compound spelling, and fuzzy matching of word forms.

Wording drift is tracked using a control sample — if new phrases fall into the wrong class, they are added to labeling and the model is retrained.

Let's discuss the task and come back with an estimate and plan

So that the estimate of the scope of work is based on your data, not average market figures, half an hour of conversation and a couple of sample exports are enough for us.

We break down the task into parts: where the text appears, in what form it reaches the model, and what is considered a correct answer.

• Sources and languages: where texts come from, whether there are mixed languages, slang, transliteration, typos.
• Data requirements: the volume of available history, availability of reference labeling, and who confirmed it.
• Classes and their balance: which ratings are needed, what to do with neutral and ambiguous formulations.
• Load profile: average and peak flow, acceptable response latency, whether batch processing is needed.
• Integration points: interface, message queue, reporting, and already operating internal systems.
• Data constraints: where it can be stored, what to anonymize, and how long to keep it for fine-tuning.
• Acceptance criteria: the share of correct answers on the holdout set and model behavior on rare classes.

In response, you receive an implementation plan, not a general proposal: stages with the result of each, scope of work and areas of responsibility, a data flow diagram, and a list of what will be required from your side.

If there is no labeling, we record its preparation and the quality control method as a separate item.

Describe the task in two or three paragraphs: domain, text examples, desired estimate. We will come back with a plan and clarifying questions.