Why a Haystack pipeline breaks on real data
A demonstration of a Haystack pipeline almost always looks convincing: a dozen clean texts as input, precise questions, relevant answers.
A real stream of requests is scans with poor recognition, emails with quotes and signatures, attachments in various formats, and correspondence in two languages within one thread.
As soon as the share of "dirty" documents grows, data extraction quality drops unevenly: some requests are processed perfectly, some end up as garbage.
The reason is usually not the model but the pipeline setup — chunking was done for demonstration: text is cut to a fixed length, tables and lists are split in the middle, and headings are separated from their content.
The second layer is retrieval and attribute extraction. Without a reranking model, the top places are taken by fragments that are similar in wording, not in meaning.
And during field extraction, data is spread across the document, some fields are missing or contradict each other: without grounding in the source and an explicit "this is not in the document" answer, the pipeline inserts plausible values.
Hence the third cause of degradation — there is nothing to measure. While quality is assessed by reviewing five examples, no change is tracked: model swap, new chunking method, node rearrangement.
A holdout set of labeled requests and a regression run after every change provide numbers instead of feelings — and turn a turnkey NLP pipeline into a manageable system. Below we break down how this is addressed during Haystack integration.
What a well-built text processing pipeline changes
The first thing that changes is the share of manual processing. A pipeline on Haystack determines the document type itself, extracts fields, and arranges them into a structure, leaving only disputed cases for the operator.
On a stream of requests, manual work is reduced from most incoming items to single-digit percentages.
Document processing speed grows not by several times but by an order of magnitude: what used to sit in the queue for days is processed by the pipeline in minutes — in parallel and in batches. Throughput stops depending on shift size, and a growing stream does not require new people.
Search quality changes along with the way data is accessed. Instead of exact word matches, the system searches by meaning: it finds a fragment using a rephrased question, synonym, or typo and returns it with a link to the source. The answer is found in one pass, not after three attempts to guess the wording.
Entire classes of failures disappear. Non-standard layout no longer breaks parsing, duplicates are merged by key rather than by operator eyes, and documents in another language go through the same chain.
Automation of request processing removes the monotonous part from people and leaves them decisions on substance.
The effect is verifiable: before starting, we record metrics on a historical sample; afterward, we compare on the same set. This shows where manual processing still holds and which failures remain — the next step is chosen based on data, not feelings.
Operating modes and deployment options: what to choose
The deployment mode is chosen not by model but by load profile and data perimeter requirements.
If documents and correspondence cannot leave the customer's infrastructure, a local environment is the only workable option: the processing pipeline, embedding store, and model live inside a closed network, and external exchange goes only through an approved gateway. If data is insensitive and the load is peaky, a managed environment scales faster.
Within any environment, the solution architecture splits into two task intake modes — streaming and batch. Below are the applicability conditions and what limits each mode.
| Option | When it fits | Limitations |
|---|---|---|
| Local environment | Personal data, regulated perimeter, isolation requirement | Requires own capacity for peaks, model updates are coordinated internally |
| Managed environment | No restrictions on data leaving, fluctuating load | Dependence on an external service, limits on request size |
| Streaming processing | Dialogue with a knowledge base, search, answer in seconds | Resources are reserved constantly and idle part of the time |
| Batch processing | Archive parsing, indexing, reports on text arrays | Result is not instant, a scheduler and repeat runs are needed |
Compatibility of options is ensured by a unified contract: the same pipeline input and output on Haystack, while storage, model, and mode parameters are moved into configuration.
Then migration from a managed environment to a local one or switching from batch mode to streaming is a settings change, not a rewrite of logic.
How we build a Haystack pipeline: work stages
Work proceeds in iterations: first we fix what the pipeline should do and on what data to verify it, then we build and hand it over to operation. This order eliminates the situation where the solution is ready but no one has prepared the source data for it. Below are the stages of NLP pipeline implementation that every project goes through.
- Data audit. We review sources, labeling, duplicates, and gaps, and assess corpus suitability. The output is a report: how many documents are ready for work and what needs additional labeling.
- Quality criteria. Before development, we fix what we consider a correct answer: accuracy on a holdout set, failure rate, response time. Otherwise there is nothing to accept the result against.
- Pipeline design. We design the chain: document parsing, chunking, indexing, retrieval, answer generation. We agree on the design in advance so we do not have to redo the code.
- Build. We implement components on Haystack, select models for the task, and configure retrieval parameters. We monitor progress on labeled examples.
- Evaluation on held-out data. We run the pipeline on a sample that was not used for tuning and compare against the criteria. We analyze discrepancies and return to the needed step.
- Solution acceptance. We hand over documentation, configurations, and the restart procedure, and train the customer's team. We measure the result on their data.
- Support. We update the index when the knowledge base changes and monitor quality according to an agreed procedure.
What we hand over with the finished solution
Handover is not a demo show but the transfer of an environment that your engineers can operate and develop without us.
Therefore, together with the working system, a set of artifacts is delivered: some describe how everything is arranged, others describe how to change, control, and safely break it.
- Haystack pipeline configurations — node composition, model binding, retrieval and generation parameters, environment settings. Edited in one place, without digging into code.
- Architecture and data flow diagrams — the request path from intake to answer, where indexes and vector representations are stored, which external services are involved, and what happens if they fail.
- Access policies and roles — who may change configuration, reindex the base, read logs, and see model answers; where keys are stored and how they are rotated.
- Model update procedure — frequency, criteria for rolling out a new version, the process for reverting to the previous one, and conditions under which an update is blocked.
- Quality monitoring — a set of metrics and a monitoring dashboard with thresholds and alerts, plus a reference query set for regression runs after every change.
- Operation instructions — start, stop, scaling under load, typical failures, and actions for the on-call engineer.
- Handover protocol — training your team according to the scheme "we do it together — you do it under our supervision — you do it yourselves", with incident review on your data.
The result is not a black box but a system with a clear entry point for any new engineer. Describe the task: we will come back with an estimate and plan, including the composition of the handover set.
Case study: how support stopped processing emails manually
Support for a service with a stream of several thousand requests per month processed emails manually: an operator opened a request, read the correspondence, determined the topic, and moved it to the right queue.
Sorting and the first response took an average of seven hours, and up to 40% of a shift went not to solving the problem but to processing the incoming stream.
What was changed
We built a processing pipeline on Haystack: an intent classifier on vector representations of text determines the request topic, a reranker selects relevant knowledge base articles, and routing rules place the request into a queue and priority.
Typical scenarios — order status, return, personal account access — are closed with an automatic answer. Everything below the confidence threshold goes to an operator along with a draft response and a pre-assigned topic.
What was achieved
The share of requests handled without a human grew from zero to 63%, automatic routing covers 94% of the stream, and first response time on typical scenarios dropped from seven hours to 25 minutes. The load on operators fell almost by half: the shift goes to complex cases, not sorting.
We keep quality under constant measurement — once a week we run a labeled sample and check accuracy by topic and the share of erroneous auto-replies.
If confidence drops below the threshold, the rule is disabled and the request is taken by an operator; logs and rule versions are retained, so we analyze any error step by step.
How does Haystack differ from a custom pipeline?
A custom pipeline wins exactly once — on the first prototype: a few scripts, one document store, one model, and answers are already being served.
The problem starts at step two, when there are three sources, part of the documents needs reindexing, and the answer requires a link to the source fragment. Each such requirement in custom code becomes a new layer of branching that no one but the author reads.
The Haystack framework describes processing as a graph of nodes with fixed input and output. Replacing a model, adding reranking, or adding a second store is an edit of configuration and one node, not a rewrite of the output logic.
Interfaces between steps are explicit, so nodes are tested separately and replaced with stubs during debugging.
| Criterion | Custom pipeline | Ready-made framework |
|---|---|---|
| First working version | Fast for one scenario | Need to break the task into nodes |
| Second data source | Changes in core logic | New node and connection parameters |
| Model change | Rewriting calls throughout the code | Replacing one node |
| Error analysis | Logs by script | State of each step and its arguments |
| Handover to team | The design lives in the author's head | The pipeline reads as configuration |
A custom solution remains justified when processing requirements are atypical: custom storage format, strict order of data access, embedding into an existing service. Then the framework adds an abstraction layer that must be bypassed.
In practice, such steps are written natively, while ready components are left for extraction, connecting steps, and observability.
We measure maintainability not by feelings: how long it takes to connect a new source, what share of scenarios is covered by checks, how quickly a model change is rolled back.
During pipeline support, these measurements show exactly where the custom part is slowing down solution development, and it is moved into the framework step by step, without stopping service operation.
Common technical concerns before starting
Concerns before starting are usually the same: will the solution integrate with what is already running, is there enough data, and what will remain after handover. We answer briefly and to the point.
We already have internal services and search running — will everything have to be rebuilt?
The text processing layer connects on top of existing sources: documents, knowledge base, and metadata are read from where they currently reside, and application services access it through familiar interfaces. Only search and answer generation change — the rest of the system remains untouched.
There is no ready labeled data — who creates it and how is quality checked?
Labeling is part of the work, not the customer's task: we describe the rules, label the sample, and calculate agreement between annotators. If agreement is low, we revise the instructions rather than increase volume.
An acceptance set is assembled from labeled examples — it shows whether the solution improved after each edit.
What happens after handover — who is responsible for quality?
We hand over configurations, labeling rules, the acceptance set, and operation documentation. Then two modes are possible: support with model updates and metric monitoring on our side, or autonomous operation by your team.
We fix the boundary in advance: which metrics we are responsible for at acceptance and what we consider degradation.
How safe is it to send data for processing?
Processing is deployed in your environment; external services are connected only where that is a deliberate decision. Access is role-based, and logs retain requests without document contents. If data cannot leave the perimeter, we build a local configuration on Haystack and open models.
Let's discuss your task and come back with a plan
To come back with a scope estimate rather than general wording, we need a minimum of input data. We select the stack for the task: for text processing chains, this is most often Haystack on models for your language and domain. What is worth preparing for the conversation:
- Problem statement in terms of outcome: what queries come in, what we consider a correct answer, what an error.
- Live examples: 100–300 real queries with expected answers and source links — we measure quality on them before starting.
- Environment: where data resides and the service runs — cloud, closed perimeter, or own servers; which systems become knowledge sources.
- Access restrictions: what cannot be sent to external services, how to store dialogue history and logs.
- Load and updates: how many requests are expected, what response time is acceptable, how often the knowledge base changes.
- Acceptance: who accepts the result, who checks intermediate versions, and how quickly they provide feedback.
Based on this data, we assemble a scope estimate, implementation plan, and acceptance criteria. We launch the pilot stage on your data in your environment: it shows real quality before moving to production operation. After that, the project starts according to agreed stages — with status and metrics at every step.
