Why manual feedback analysis cannot keep up with the flow
Post-event surveys arrive in a single stream: closed scales and open-ended responses, where people write what did not fit into the ratings.
A human hand processes such an array selectively — in a day, one can realistically read a few dozen surveys, and the rest is skimmed diagonally.
The bottleneck in feedback processing here is not reading speed, but the fact that labeling criteria live in a particular person's head and change from one evening to the next.
Open-ended responses fit poorly into manual labeling: the same meaning arrives in different words, and irony and caveats are easy to read backwards.
Analysis of open-ended responses turns into a set of vague topics — “overall liked it,” “organization was fine” — on which no decision can be made.
Worse still: critical signals are lost quietly, because a rare comment about entrance, registration, or cold in the hall drowns among hundreds of neutral reviews and does not have a high average score that would make it noticeable.
After several cycles of this work, trust in the data itself is lost: the team is sure that “everything is fine,” while recurring complaints surface after the fact, already at the next event.
Manual feedback labeling does not scale — the number of surveys grows, the number of conclusions does not. We explain how this is addressed, so that the flow of surveys turns into a prioritized list of decisions, rather than a check-the-box report.
Effect of implementation: speed, completeness, and repeatability
Effect of implementation is visible in three shifts: how much time the summary takes, what share of feedback actually gets analyzed, and whether labeling is consistent from run to run. Below is what we record at acceptance, not general words about “it has become more convenient.”
Summary turnaround: hours instead of days
Previously, an analyst exported responses from the form, consolidated scores into a table, and read comments by eye — the first conclusions appeared several days later, when the flow had already cooled and decisions for the next event had been made.
The system processes the corpus in batches immediately after the surveys close: by the morning of the next day, you have a topic summary with source quotes. For a series of events, this means analyzing each one, not only the flagship.
Analysis completeness: the whole corpus, not a sample
Previously, long comments and scores below the threshold usually made it to analysis — short “liked everything, but…” responses settled unanalyzed.
The classifier goes through all responses, including open-ended fields without a score, and sorts them by topic: program, logistics, catering, speakers, organization. The share of reviews that received at least one label reaches full coverage instead of the usual 10–15%.
Labeling repeatability and quality metrics
The same review receives the same topic on a repeated run — consistency between runs is higher than between two people reading the text one after another.
At acceptance, we look at precision and recall for each topic, the share of responses that went into “outside the taxonomy,” and the discrepancy when reprocessing the same set.
Labeling by Net Promoter Score and open-ended responses are calculated in one scheme, so the detractor segment is visible immediately, without a separate reconciliation.
Feedback processing methods: batch analysis, streaming, and manual sampling
The feedback processing mode is chosen not by “speed in general,” but by the acceptable delay between feedback arrival and the appearance of a conclusion.
Batch analysis runs on a schedule over an accumulated array: one pass over the entire sample, a full recalculation of metrics, and the ability to rerun the model on the entire history if the labeling rules have changed.
Streaming mode calculates as data arrives, maintains aggregation windows, and delivers a signal on alarming scores almost immediately, but recalculating the past in it is more expensive.
The hybrid combines both: final analytics are calculated in batch, while the hot slice goes through the stream. Manual sampling picks up what the model does not confidently handle — irony, mixed scores, rare wording.
| Option | When it fits | Limitations |
|---|---|---|
| Batch feedback processing | Arrays for a period, final reports, wave comparison, training rubrics on history | Conclusions delayed until the next run; frequent recalculations load the storage |
| Streaming data processing | A signal is needed during the event: a spike in negativity, complaints about one program block | No full picture until the window closes; retrospective recalculation is a separate pass |
| Hybrid processing scheme | A constant feedback stream with regular reporting and review of results | Harder to maintain: two pipelines, unified labeling rules, and result reconciliation |
| Sample for manual analysis | Model quality control, disputed cases, rare topics and segments | Does not scale: the volume of analysis is limited by analysts' hours |
How we choose the mode
We look at three things: whether a conclusion is needed during the event, whether recalculation over the entire history is required after changing rubrics, and how many raw records we are obliged to store.
If the answer is “no, yes, a lot” — batch works; if a signal is needed in the moment — we add streaming and move manual sampling into quality control.
How implementation proceeds: from data audit to handover
Implementation starts not with the model, but with the data: until it is clear what exactly is being collected at the input, any classifier produces pretty numbers in a demo and useless ones in production. That is why the first two stages take as much time as all the others combined.
-
Event data audit. We review all feedback sources: surveys, free-form comments, support correspondence, reviews on platforms. We record formats, languages, the share of empty and duplicate responses, and find fields where participants write text instead of selecting an option.
-
Data schema and normalization rules. We agree on a single record representation: what counts as one review, how we link it to a track, speaker, and session. Without this, the same comment behaves differently across different exports.
-
Setting up the topic classifier. We assemble a list of topics for your program and terminology, label a reference set manually, and configure the model so that it relies on these examples rather than general notions of a “good event.”
-
Labeling quality check. We calculate the share of matches with manual labeling and review all discrepancies by category.
We look separately at rare but important topics — complaints about organization, accessibility, safety: missing them is more costly than an error in a high-volume category.
-
Report assembly and integration into processes. We determine who receives the summary and when: after each session, daily, or at the end of the event. We configure export in a convenient form so that analysis does not require manual assembly.
-
Team training and system acceptance. We show how to add examples, change topics, and check quality after edits. Acceptance is based on pre-agreed indicators on your data, not on a test set.
At the output, you receive a working system, documentation on topics and labeling rules, access, and a clear support procedure — after that, the team runs it independently, and we get involved for changes.
What you keep after the project: documents, schemas, regulations
A working service is not the project's final point. In a month, new wording will appear in the surveys, some categories will stop hitting the target, and the rules will have to be adjusted by your team.
That is why, along with the code, we hand over a set of artifacts that let changes be made without relearning how the system is built.
- Feedback labeling regulation — category taxonomy, rules for assigning disputed cases, the model's confidence threshold, and what to do with reviews below it.
- Data flow schema — from survey exports and participant lists to the aggregate mart: where we anonymize, where we deduplicate, where we validate.
- Configured processing rules — classification by topic and sentiment, exception dictionaries, grouping of similar reviews, separation of questions from complaints.
- Access policies — roles, who can access raw texts with personal data and who only sees aggregates, retention periods, and the anonymization procedure.
- System documentation — component composition, external dependencies, failure points, instructions for updating rules and troubleshooting errors.
- Support regulation — which quality metrics and share of unlabeled items we monitor, when an alert triggers, and how to roll back to the previous rule set.
- Handover-to-support plan — training your team on real data, review of complex cases, and the escalation procedure to our engineers.
- Reference review set with manual labeling — used to check quality after every change without waiting for the next event.
Case study: summarizing the feedback stream instead of manual analysis
How it was set up before
An organizer of a series of industry conferences collected surveys after each event: an exit form, follow-up emails, and some responses by voice.
The analyst managed to read about a tenth of the array; the rest sat in the export — yet conclusions for the next event's program were made precisely from this sample.
Analyzing one wave took almost a week: export and cleaning, manual topic labeling, consolidation into a summary. By the time it was ready, the next event had already passed — too late to make edits to logistics and the schedule.
What was changed and what the result was
We assembled the stream into a single loop: surveys and free-form comments are collected automatically, voice responses are transcribed via Whisper, and text is labeled by rubric, sentiment, and topic. Clustering is reproducible — the same topic gets the same name regardless of who runs the analysis.
- Completeness: from about 10% of surveys to 100% in each wave.
- Summary speed: from 5 business days to 12 hours — before the next event begins.
- Labeling consistency between analysts on the same rubric: 0.6 → 0.9, with half as many manual edits.
- Topic stability: 14 permanent categories instead of a “floating” set from run to run.
- Reaction: a navigation problem surfaced on the first day of a five-event series, edits were made between events — the share of negative scores on that item fell by a factor of three.
What to do if feedback comes through different channels
Post-session surveys, chat correspondence, emails to organizers, and open forms are collected by different people at different times, so each export arrives with its own structure: somewhere five questions, somewhere free text in one field, somewhere a score on a ten-point scale.
While there are two sources, they can still be consolidated manually; by the fifth, lost records and double-counting of the same reviews appear.
The first step is a unified data format. Each message becomes a record with the same set of fields: source channel, timestamp normalized to one zone, anonymized participant ID, text, numeric score, link to a talk or session.
Scores on different scales are normalized to one, free text is separated from structured responses — after that, analysis works with a homogeneous array and does not depend on where the row came from.
The second step is response deduplication.
A participant often writes in chat during a session and repeats the same thought in the survey in the evening: exact duplicates are filtered out by comparing normalized text, and semantically close ones by the semantic similarity of wording.
The similarity threshold is configured on a labeled sample: too low will merge different opinions, too high will leave duplicates.
The third step is consolidating sources into a common analysis. Records from all channels enter one stream, where topic labeling, sentiment scoring, and speaker attribution are applied to them.
If a source did not provide data on time, the analysis does not stop: it is marked as incomplete, and the share of each channel is visible in the report.
At the output is a single cross-section across all channels: you can see which topics are raised only in chats and which only in surveys, and where scores diverge. Separating sources by fields and keys is cheaper than consolidating them manually at every event.
Technical concerns: labeling, languages, data privacy
The concerns are usually the same: will the model understand sarcasm, what to do with feedback in multiple languages, and will an array of texts with names and contacts turn into a leak. Below is how these three points are addressed in the feedback analysis systems we build.
Will the model understand sarcasm and “polite negativity”?
We assemble labeling on your event's reviews: several hundred examples where negativity is expressed indirectly — “thanks for the forty minutes in line.” We calculate agreement between annotators and only after the threshold send the set for training. If the metric is lower, we change the labeling rules rather than “tweaking” the model by eye.
Feedback comes in different languages — will the meaning be lost?
Each review is analyzed in its original language; translation is connected only for the report and stored next to the source: in a disputed assessment, the original is visible.
Mixed phrases, where an insertion in another language appears inside Russian, go through in one pass — half the meaning is not lost at the language boundary.
How do reviews with names and contacts not go outside?
Names, phone numbers, email, and booking numbers are replaced with labels before the text enters the model: anonymization at the input, not after the fact.
Access, roles, and retention periods are described according to ISO 27001 requirements; the anonymized array remains suitable for analytics by day, venue, and session format.
Let's discuss the task: what data and access are needed for assessment
For the project assessment to rely on facts rather than assumptions, three things are needed: examples of real feedback, access to data, and an understanding of what decision you want to make based on the analysis.
The more specific the inputs, the fewer assumptions in the timeline and the more accurate the work plan — without a “we'll clarify as we go” stage.
- Examples of feedback from the last period — an export in its original form: free text, scores, intake channels (surveys, chats, email, external platforms). They show the audience's language, message length, and share of irrelevant noise.
- Storage schema — where feedback currently resides, how often it is updated, whether there are manual exports from questionnaires and summary tables that someone maintains in parallel.
- Data access — test access to the storage or an anonymized export. We separately discuss which fields are personal and how to mask them before transfer to processing.
- What has already been done with this data — manual labeling, reports, attempts to calculate sentiment. This shows which categories are truly important to you and what did not stick.
- Environment constraints — where the system should operate, who owns the data, who approves access from security and legal requirements.
- Expected result — what decisions you plan to make: adjust the next event's program, change the venue, respond to negativity within the first day.
With this set, we come back with a project assessment: scope of work, composition of stages, and what will be required from your side at each of them.
Describe the task in a convenient format — a short call or an email with the inputs — and we will propose a work plan with responsibility boundaries.
