Development of a Parser for Contact Data from Open Sources

Why a contact data parser breaks on live sources

A live public source is not a static page but a system that changes its markup without warning: a block was renamed, a wrapper was added, email was moved into a dynamically loaded fragment.

The contact data parser does not crash — it returns empty cards, and the crawl formally completes successfully. The error is noticed later, during calling: half the addresses are missing from the registry, while the logs look clean.

Then field normalization begins. The same number arrives as "+7 (900) 000-00-00", "8 900 000 00 00" and "9000000000", email — with a space or capital letters, job title merges with department name into one line.

Without explicit value normalization rules, deduplication by string matching does not work, and one person ends up in the database two or three times.

Blocks and limits add a second layer of failures. Anti-crawling protection cuts frequent requests, returns a stub or a challenge instead of the page, and changes the response shape at the same address.

A parser that does not distinguish "there are really no contacts" from "we were cut off temporarily" keeps writing empty records into the registry — the output is volume with no data behind it.

The cost of such failures is counted not in percentages but in manual hours: an operator rechecks records, cleans out duplicates, and calls through non-working channels.

Stable contact collection from public sources rests not on one lucky request but on monitoring markup changes, explicit normalization rules, and handling access failures. Below we break down how this is addressed.

What careful collection gives: speed, completeness, field accuracy

Manual crawling of sources hits arithmetic: a card takes one to two minutes, several hundred records accumulate per shift, and each is transferred into a table by hand.

From there come typos in numbers, different formats for writing the same phone, and duplicates that surface during calling.

The list of sources and crawling rules are set by configuration, and pages are parsed by document structure, not by the "approximate" location of blocks.

If a page has schema.org markup in structured form, contacts are taken from it directly — without layout heuristics that break at the first redesign.

Collected data is brought to a unified schema: phones are normalized to one format and checked by checksum, email addresses by syntax, duplicates are collapsed by the composite key "organization + contact".

The output is not a raw set of strings but a registry where each record is valid by construction. A repeated run updates only changed cards, so up-to-dateness is maintained without a full rebuild.

According to our measurements, the gain in collection speed is eight- to tenfold: the volume that takes weeks with manual crawling is completed in a shift.

More than ninety percent of cards where contacts are published at all reach the registry, and manual edits after import are a few percent — input errors no longer occur at entry.

Collection modes: markup parsing, API, export

The three data acquisition modes do not replace each other: one works for one set of sources, another for a different one.

The choice depends on whether the source provides data machine-readably, whether it has a ready-made export, and how stable its page markup is — for each set of sources a specific mode is selected.

Below are the three modes and where each hits its ceiling.

In practice, the set of sources is almost always mixed, so we combine modes: where there is an API, we go through it; we cover the rest with markup parsing; and rare platforms with ready-made exports.

We keep request limits and crawling rules for each source in a separate configuration, not in code: a fix when layout or quota changes does not require rebuilding, and statistics by mode show where the share of collected contacts has dropped.

When a source changes the rules of the game, its mode is switched — the rest continue to work unchanged.

How we handle implementation: from source audit to handover

Implementation goes in stages, and each ends with an artifact that can be verified. The project does not turn into a black box: you can see where data comes from, by what rules it is collected, and how quality is controlled.

1. Source audit. We compile platforms into a table: output volume, terms of use, availability of a public API, layout stability. We filter out sources without collection rights immediately — otherwise the crawl will have to be rewritten.
2. Crawling scheme. A request queue with rate limiting, pagination crawling, retries on failures. For pages with dynamic rendering — rendering; for catalogs — crawling through lists of sections and cards.
3. Field extraction. We extract contacts by block context, not from the entire page text: a value from the card header and from an ad block are different entities.
4. Normalization. Phones — to a unified format with country code, emails — to lowercase, addresses — to the region/city/street structure. We merge duplicates by a composite key.
5. Validation on a reference sample. We label the control set manually and compare: share of correct values, share of misses, false positives. If below threshold, we fix the rules, not pad the volume.
6. Handover to operation. We hand over the code, source configurations, run schedule, and quality report. We show your engineers how to connect a new source and troubleshoot a failure.

Then — support: regular runs, alerts when the share of valid records drops, and updating rules for layout changes.

What remains with you: configurations, schemas, regulations, access

We hand over not a "working script" but a set of artifacts by which the contact data parser can be run, checked, and developed without us. Each item undergoes acceptance on your side: we show where it is, how to use it, and what happens when the source markup changes.

• Working configurations — parsing rules for each source, schedule, delays between requests, concurrency limit, retries on failures, and a list of addresses from which crawling is performed.
• Source schema — a map where for each source the type, access method, update frequency, set of extracted fields, and typical markup changes that break parsing are specified.
• Access rights and legal framework — which sources are open for collection, rate limits, separate accounts for the service, keys in secure storage, not in code and not in logs.
• Data model and normalization — bringing phones, emails, and addresses to a unified form, record deduplication rule, field priority when data differs between sources.
• Launch procedure — scheduled or on-demand operation, procedure for reviewing the run report, actions when a source is blocked, and escalation points within your team.
• Documentation — description of fields and export formats, request examples, instructions for connecting a new source by editing the configuration, without changing code.
• Checks and monitoring — a reference sample with expected results, an alert when the share of successful parses drops, a configuration change log with rollback capability.

Then the service lives according to your regulations: updating rules when markup changes, adding sources, rotating access. If help is needed, we will connect to an already documented system, not to someone else's code without description.

Case study: registry from a supplier catalog, metrics before and after

What it was and what we changed

The client has a catalog of industrial equipment suppliers with several thousand entries, and contacts of procurement departments were collected manually: two employees crawled websites and transferred data into a table.

Almost a full workday was spent on a hundred cards; on repeat crawling, up to a third of records turned out to be outdated, and some companies did not make it into the registry at all. No one measured completeness — lost entries were simply not noticed.

We built a contact data parser from public sources: crawling the catalog by sections and extracting fields from page markup via Schema.org where it exists, and from contact blocks where it does not.

Each record stores the source address and collection date, duplicates are merged by domain and company name, and disputed matches go into a queue for manual review. Crawling runs on a schedule, so the registry is updated rather than rebuilt from scratch each time.

Metrics before and after

A full run through the catalog takes about two hours versus weeks of manual sampling; employees only need to check the report. Registry completeness grew from 71% to 96% — the difference was shown by a verification run on a control sample of companies.

The share of duplicates fell from 18% to 2%, 100% of records have a confirmed source, and the reduction in manual labor is visible in the number of edits after each crawl.

Where does the parser lose to an API and why?

The API wins where it exists and covers the needed fields. The source provides data in a consistent form: the response structure is described, fields are labeled, request limits are known from the documentation. As long as this is the case, markup parsing is an extra layer that only adds points of failure.

Markup parsing is more reliable in the opposite situation: there is no API at all, it is closed behind a paid plan, limited by number of requests, or provides a reduced set of fields.

Then public pages are the only available channel, and data is taken from there and mapped to our own schema.

The price of this solution is dependence on the source layout. A template change or redesign breaks parsing rules, so regular checks, monitoring of empty samples, and an alert when the share of found contacts drops are needed. The API breaks predictably — with an error code in the response, not a silent loss of data.

Source limitations influence the choice no less. CAPTCHA, blocking by address and behavior, and authorization requirements force parsing to be slow, with pauses; otherwise the source starts returning stubs. For an API, the same constraints are expressed explicitly — by a request limit per unit of time.

Therefore we choose the approach based on the task. First we check whether an API exists and whether it covers the needed fields; if yes, we build on it.

If not, or it is narrow, we use markup parsing, but with a normalization layer so that a source change does not break the entire pipeline. Most often it turns out to be a hybrid: the API provides the main volume, and parsing picks up the rest.

Technical concerns: field quality, blocks, retries

Concerns about data quality arise before development begins, and that is normal: both the storage schema and the amount of support depend on the answers. Below are three questions that are asked most often, and how they are addressed in code, not in words.

What happens when the source site's markup changes?

Field extraction rules are moved to separate configurations, so a fix takes hours rather than rewriting the parser.

Each run calculates the share of recognized fields — a drop below the threshold raises an alert, and we learn about the change before the client opens an export with empty columns. Pages that are rendered by scripts are processed via Playwright.

How to limit scope and confirm accuracy?

Scope is set by a list of sources, regions, and topics — collection does not run across the entire internet.

Accuracy is confirmed by a control sample: some records are checked manually, and the report includes the share of confirmed phones and addresses, not the total number of lines.

Let's discuss the task: we'll build a collection scheme for your sources

To come back with an estimate of scope rather than general words, we need input on your task.

Below is a list of what removes the main uncertainty: with it, we can immediately estimate labor costs, choose a collection method, and build a work plan for specific sources and the registry composition.

• Sources: catalogs, classifieds platforms, industry directories, registries — and whether you have active access or subscriptions to them.
• Registry composition: which fields are mandatory (name, contact person, phone, email, address, field of activity), which are optional.
• Duplicate merging rules: by which attribute we consider two records to be the same company — phone, address, or name match.
• Volume and frequency: whether you need a one-time snapshot or regular updates, how many records you expect in the output.
• What is already in place: previous exports, tables, databases — so as not to rebuild what works.
• Boundaries of what is acceptable: which sources and fields you do not take according to your internal rules.
• What happens next with the data: export format, where we store it, who searches and by which fields.

Send us what you have, even in draft form — we will come back with a scope estimate, timeline by stages, and a clear work plan. If some points are unclear, we will go over them in discussion and record the agreements in writing.