Filtering, cleaning, and vectorization of email correspondence for RAG

Why email indexing breaks down at large volumes of correspondence

At a volume of tens of thousands of emails, email indexing breaks not on the model but on data preparation: the share of noise in the index grows faster than the volume of useful text. Semantic search begins to return formally similar fragments that do not answer the question. We examine exactly where the preparation of correspondence falls apart and what is done about it.

The first source is duplicate emails. In a thread, each message pulls the history of previous ones, replies go out with full quoting, forwards duplicate attachments.

The same paragraph ends up in the index five to ten times, and noise in search results grows not because the model is weak, but because one meaning is represented by a dozen nearly identical fragments.

The second is service traffic: notifications, auto-replies, calendar invitations, disclaimers, and signatures.

In the volume of correspondence they occupy a noticeable share of characters but carry no meaning; during vectorization they pull in queries similar in form and blur the quality of search across correspondence.

The third is attachments and structure. Scans of invoices, contracts, tables outside the text layer fall out of the index, while email headers — participants, dates, folders, subject — do not get into filters.

Without them, it is impossible to limit a slice by contractor or period, and search results mix work discussions with mailings.

Chunking also suffers: a fixed window cuts meaning at quote boundaries, and overlap has to be selected for a specific correspondence. All of this together produces blurred semantic search results — next we examine how the pipeline is cleaned at each step.

What changes after cleaning and vectorizing correspondence

While an entire email gets into the index — with signature, service confidentiality notice, reply history, and nested quotes — search finds a lot, but not the right thing.

The share of relevant fragments at the top of results stays around random, and response time grows in proportion to the number of processed characters rather than the number of meanings.

Vectorization by semantic blocks breaks this dependency: only meaningful paragraphs go into the index.

After cleaning correspondence, one question corresponds to one or two precise fragments instead of a dozen nearly identical ones.

The answer is assembled from the primary source — with the author, date, and subject of the email — so any statement by the system can be verified in the original. Recall does not suffer: noise is cut off, not meaning, and rare wordings do not fall out of the results.

Reduction in storage is a consequence of the same cleaning. Repeated threads are collapsed to unique replies, headers and signatures are stored once, attachments are indexed separately from text.

A vector index on Elasticsearch in this form is rebuilt incrementally: a new email becomes available to search within minutes, not through an overnight run over the entire database.

Along with volume, a whole class of errors goes away: the system no longer answers with a quote from a signature, does not merge correspondence from different people with the same email subject, and does not return an outdated revision of a document as the current one.

The output is a predictable share of hits in the top and a response time that does not depend on how long ago the mailbox was created.

Correspondence processing modes: streaming, batch, and hybrid scheme

Index update mode is a fork that is rarely chosen once and for all: the schemes have different requirements for answer freshness, load on the mail server, and storage volume. Below are three working alternatives among which people most often choose.

In all three variants, the email is first brought to a clean form: the body is separated from service headers, signatures and quotes of previous replies are cut out, attachments are described by separate text.

The quality of this step determines whether meaning or duplicates of the same email thread get into the index.

Comparison of modes

Boundaries between schemes are fluid: streaming mode is often supplemented with nightly reconciliation, and batch reindexing with a queue for fresh emails.

The choice depends on how quickly the answer must take yesterday's correspondence into account and whether you are ready to give load to the mailbox on a schedule.

Implementation stages: from mailbox audit to handover of policies

Work proceeds in iterations: each step ends with an artifact and a criterion by which the customer accepts the result. Below are six stages that we go through when indexing email correspondence, indicating exactly what is checked at the output.

1. Mailbox audit. Inventory of sources: how many mailboxes and folders, what attachment formats, what encodings and languages, the share of service mailings and automatic notifications. The output is a list of what goes into processing and an explicit list of exclusions.
2. Pipeline design. We agree on the sequence of operations: extraction of the email body, cutting of quotes and signatures, merging duplicates from threads, normalization of encodings, splitting into semantic fragments, and building the vector index. The design is fixed before writing code — edits at this stage are cheaper than rework after full processing.
3. Pilot on a control sample. We run several hundred emails manually labeled as "meaningful" and "noise." The acceptance criterion is the share of correctly extracted fragments and the absence of service data in the index.
4. Full processing in batches. We go through mailboxes with error logging; emails on which parsing failed automatically go into a retry queue rather than being lost.
5. Search quality check. A set of questions with reference answers: we look at whether the needed email is found in the first results and whether the system points to the original source.
6. Acceptance of the result and handover of policies. Demonstration on your queries, instructions for users, access rules, and the procedure for updating the index when new correspondence arrives.

Key control points are audit, pilot, and acceptance: it is there that you can see that quality matches the agreements, rather than "generally works."

What the team hands over: designs, dictionaries, and policies

We hand over not only working search across email correspondence but also the entire set of solutions around it — so that the team can maintain the system without us.

For each artifact, it is clear what decision was made, why exactly that one, and how to change it without losing search quality.

• Filtering rules. What does not get into the index: service mailings, auto-replies, duplicates, emails without a text body. With thresholds by number of recipients, signs of mass sending, and a list of exclusions.
• Cleaning dictionaries. How quotes of previous emails, signatures, disclaimers, and service headers are removed — with an exceptions table so as not to cut out useful context.
• Splitting and vectorization design. Fragment size, overlap, binding to the email thread — the search returning the email itself or a quote from it depends on this.
• Access policies. Roles and visibility scopes, what is masked before indexing, how an email is deleted from the index on request, and how this is confirmed by the log.
• Pipeline documentation. A map of stages from mail collection to model response, logging points, metrics, and behavior during failures and reprocessing.
• Update policy. How often dictionaries and the index are rebuilt, who accepts changes, and how to return to the previous configuration.
• Set of control checks. Questions with expected sources, a run after each edit — degradation is visible before rollout.

All of this is handed over to the customer's repository with change history: an edit to filtering rules or a dictionary can be traced and rolled back without involving us again.

Case study: how a sales department's correspondence became a knowledge source

The customer is a distributor with a sales department of several dozen managers: correspondence with clients had been conducted for years, but new employees again and again asked questions whose answers were already in the mail.

We connected to Microsoft 365 mailboxes and assembled the correspondence into a single knowledge base where one can search with an ordinary question in Russian.

The main work was not in indexing itself but in cleaning: quotes of previous replies, signatures, legal addenda, and automated mailings were removed from messages, and the thread was merged into one document — otherwise the answer lost the context of the question and search returned fragments. Service copies and attachments did not get into the base: they added volume but not knowledge.

Metrics before and after

Measurements were taken on the same 200 manager questions: before launch and one month after working with the base.

What changed in the work

Reusing emails became a common practice: a manager finds a ready-made wording of a proposal or a rebuttal of an objection in a minute instead of writing it anew. Interns stopped occupying mentors with the same questions, and the head sees which topics clients raise most often.

Department knowledge no longer leaves with a departing employee: new emails get into the base automatically, and old ones remain linked by correspondence topics.

How does email indexing differ from simple mailbox search?

Client mailbox search is full-text search: the query must lexically match the word forms in the email fields. The local index in Exchange and compatible storages returns an answer instantly, as long as you remember the exact wording.

Then the gap begins: correspondence in two languages, abbreviations, long reply chains where the same text repeats in ten copies, signatures, and legal addenda.

Email indexing is not search but data preparation. The email is parsed into structure, new text is separated from quoted text, the correspondence thread is merged into one document, duplicates are removed, addresses, dates, and participant names are normalized.

Then the text is cut into fragments along semantic boundaries and only after that gets into the index.

The full-text layer is not discarded — it remains for exact matches: contract number, address, SKU, last name. Semantic search works on top of the cleaned corpus and finds emails where the needed words are not present at all.

The effect is visible in engineering quantities: the share of service and quoted text in the index falls from two thirds to a few percent, so there are fewer false positives in results, and several years of history fit into a reasonable volume.

The boundary is simple: the client copes while exact matches within one mailbox are needed; indexing is needed when searching by meaning, across all correspondence, and taking access rights into account.

Technical doubts and limitations worth knowing in advance

Processing limitations almost always surface during a trial run: the archive is read halfway, attachments are lost, an employee after transfer to another department continues to see others' emails.

None of these issues blocks the project, but each changes the scope of work and the quality of answers — therefore we begin email correspondence indexing with an explicit fixing of limitations, not with discovering them after launch.

How are attachments and nested quotes processed?

Attachments are a separate layer: tables and scans are recognized, archives and executable files are cut off. Reply chains are collapsed into one semantic block, otherwise the same phrase gets into the index five or six times and breaks ranking.

Will search become a leak channel?

The index is built with inheritance of source permissions: only emails available to the employee in the mail system get into results. When a role changes or an employee leaves, permissions are recalculated — by a separate job, not at the next rebuild.

How often is the index updated?

New emails are loaded overnight, changed threads are rebuilt once a week. Full reindexing is needed only when changing the vectorization model or cleaning rules: it consumes resources, and during that time search answers from the old snapshot — we warn in advance.

How to start: we'll discuss the task and come back with a plan

Starting a project does not require a framework agreement and a separate preparation stage: we analyze the task and immediately determine what email indexing should cover.

The more precisely the source and real search scenarios are described, the fewer assumptions go into the estimate.

What is needed from the customer:

• export source: mbox archive, set of EML files, export from a mail service, or export in JSON;
• volume and structure: number of mailboxes, order of magnitude of emails, share of attachments, and number of participants in one thread;
• what to cut off: mailings, service notifications, automatic replies, duplicates from correspondence history;
• privacy requirements: which fields to mask, where the index is stored, who has access to email text;
• search scenarios: for example, correspondence under a specific contract or the history of solving a task with attachments;
• environment for work: a test mailbox or bench and restrictions on data export.

Then the order of steps is as follows: we look at a sample export of several hundred emails, show how cleaning, thread assembly, and vectorization proceed, and come back with a work plan.

There are no obligations at this stage — you receive a clear indexing design and an estimate of the scope.