Parsing Bitrix News: Where Auto-Population Breaks
Auto-populating an infoblock breaks not at the stage of "grabbing the feed and parsing it," but during operation: the source changes format, Bitrix agents accumulate stuck processes, and the database grows from duplicates.
After a month or two, such a pipeline starts requiring manual intervention more often than it saves the editorial team time.
The most common cause is duplicate items. The feed serves the same text under different URLs; on re-import, a new record is created instead of updating the existing one.
Without matching by a stable attribute — a text hash, the source's external identifier — the infoblock becomes overgrown with copies, and site search starts returning three identical news items.
The second point of failure is the source encoding. Some feeds arrive in a single-byte encoding, others in UTF-8; when mixed, you get garbled characters in headings and broken symbols in announcements.
The problem is not in the parser itself, but in the fact that encoding normalization is not moved into a separate step before writing to the infoblock.
Feed interruptions and stuck agents hurt stability. If the source responds slower than the timeout or drops the connection midway, the Bitrix agent remains in a working state and the next run overlaps the previous one.
The load on the database grows, the import "sticks," and the logs show a series of identical errors with no clear cause.
Load growth deserves separate mention: each run without limits pulls the full feed and writes everything indiscriminately. As tens of thousands of records accumulate, querying by date and sections starts to slow down, and the indexes do not cover the actual news output queries.
We break down how this is addressed: where to place normalization, how to build duplicate checking, and what to do with stuck agents so that auto-population works without daily supervision.
What Changes After Setting Up Auto-Filling
Before setting up auto-filling, the news section lives in fits and starts: the editor manually transfers materials from feeds, publication lags behind the event by hours, and half of the cards go into the archive as outdated.
After connecting the pipeline, the cycle changes — let's break down exactly what and by what magnitudes.
Publication Speed and Freshness
The flow from sources is processed on a schedule: the feed is polled, new materials enter the queue, are normalized, and go to publication.
The delay between a news item appearing at the source and its release on the site is reduced from hours to minutes — the section stops lagging behind the news agenda.
Share of Manual Work
Manual moderation remains where it creates value: the editor checks the headline, tags, category assignment, and decides whether to publish or reject.
Draft transfer, markup cleanup, deduplication, and section layout are automated — they take not the editorial team's time but the pipeline's work.
Resilience to Source Failures
A source may return an empty feed, change its layout, or become unavailable — the pipeline does not publish junk and does not crash.
Material with incomplete fields goes into quarantine, duplicates are filtered out by fingerprint, and when a source is unavailable, the queue waits for recovery and catches up on what was missed.
What Is Visible in the Numbers
The share of materials going to publication without editor edits grows; the time from a news item appearing to a card on the site drops; the number of empty and duplicate entries in the section trends to zero. Content freshness stops depending on the editorial team's workload.
We assemble the feed parsing scheme and publication rules for your section — on 1С-Битрикс this is done with built-in tools without external add-ons.
Methods of collecting feeds: comparison of approaches
The data source determines not only the format but also the update mode: the polling frequency, the load on the site, and the resilience to failures all depend on it. Below are the working options that are realistically chosen between for auto-population.
Comparison of approaches
| Option | When it fits | Source limitations |
|---|---|---|
| RSS/Atom feed | The source provides a standard feed; a title, date, link, and teaser are needed | It only provides what the owner has put in: no full text, teasers are sometimes truncated |
| Open API | There is documented access and a token; structured fields are needed | Limits on the number of requests, pagination, periodic changes to the response schema |
| Parsing HTML pages | There is no feed or API, but the data is open | The markup changes without warning — the selection markup breaks |
| Email newsletters | The source sends a digest, there is no other channel | It arrives with a delay, emails get lost in spam, attachments are not always readable |
| Manual export | A one-time archive transfer or infrequent updates | It does not scale: every new item is a manual operation |
Update mode and polling frequency
The polling frequency is selected to match the source's publication rate, not "the more often, the better." For a news feed, a reasonable interval is from 15 minutes to an hour: it catches fresh material but does not create unnecessary requests.
If the source provides a publication date, a check by unique identifier or link cuts off re-insertion — duplicates do not end up in the database.
For sources with strict request limits, we set up a queue and process the feeds sequentially, with a pause between calls. If the source is unavailable, the task does not fail but is moved to the next cycle — the site continues to work on previously collected data.
Parsing HTML pages requires separate control: we store the selection selectors in the configuration so that a markup fix is made in one place rather than in the task code.
For each source we keep a log: the time of the last successful collection, the number of new records, errors — from it you can see that a feed has stopped returning data before the editor notices it.
How we implement parsing: stages of work
We structure the process so that the output is a working auto-fill pipeline rather than a one-off script: first we sort out the sources, then we design the exchange, then we set up scheduled runs and protection against duplicates.
- Source audit. We compile a list of feeds and sections that the editorial team actually needs, check availability, delivery stability, and the structure of the materials. We record which fields are transferred — title, teaser, date, author, images — and which categories they correspond to.
- Exchange schema. We design how data gets into the information blocks: mapping of sections, rules for text transformation, image linking, behavior when required fields are missing. At this step we agree on what to do with material that failed validation — discard it or place it in a separate section for manual review.
- Agent setup. We implement regular traversal of sources at a specified interval, with error handling and retries on network failures. We log every pass so it is clear how many materials came in, how many were rejected, and for what reason.
- Duplicate check. We set up matching by external identifier and text fingerprints so that a repeated pass does not create copies. We separately check the behavior when a material is updated at the source — an edit should update the existing record rather than spawn a new one.
- Test run. We launch the pipeline on a limited set of sources, compare the result against a manual sample, and monitor the load on the site during traversal. We make adjustments before rolling out to all feeds.
- Handover of procedures. We provide a description of the sources, rules for mapping categories, the procedure for adding a new feed, and instructions for reviewing rejected materials. The editorial team receives a clear set of actions without having to contact the developer for every edit.
This order produces a predictable result: the content pipeline runs on schedule, duplicates do not accumulate, and maintenance does not require the constant involvement of an engineer.
What the client receives
We hand over not "a script running somewhere on a server," but a package that lets the exchange be operated, extended, and handed off to another contractor without digging through someone else's code.
- Exchange configuration — source settings files: the list of feeds, polling frequency, mapping of source fields to infoblock properties, binding to sections and news types.
- Data flow schema — where we pull from, where we normalize, which infoblock the material lands in and how often, and where the manual moderation points are.
- Processing rules — deduplication by canonical link and title, filters by category and stop words, text trimming and cleanup, behavior on an incomplete or broken feed.
- Maintenance regulations — what to do when a feed format changes, how to add a new source, how to roll back an erroneously uploaded batch, and who owns queue monitoring.
- Documentation — a description of the exchange structure, the purpose of each field and parameter, and an editor's guide: how to fix a rule without breaking the neighboring ones.
- Access and roles — which accounts the exchange needs, what rights the editor has, and what is blocked from publication without confirmation.
- Log and monitoring — where to view import statistics, how many materials passed and were filtered out, and how a source failure notification arrives.
All of this is committed to the repository together with the code, rather than left in a chat thread. A month after launch, you or your in-house developer open the documentation and change a rule yourselves — without contacting us.
Describe the task: we will select sources, assemble the exchange configuration, and come back with an estimate and a work plan.
Case Study: Automated News Feed Population Without Duplicates
The client is a regional industry portal on 1C-Bitrix with an editorial team of two. News was collected manually: the editor went through a dozen sources, copied text, cleaned up markup, and published.
The output was 8-12 items per day, with a third of the time spent on mechanical work, and the feed still lagged behind events by several hours.
What It Was Before
Publications depended on the editor's availability: on days with urgent tasks, the feed stalled. Duplicates appeared constantly — the same story came from several feeds with different headlines, and tracking repeats manually was impossible.
Some materials went to the archive with broken image links because the images remained on the source side.
What Was Changed
We set up collection from approved feeds on a schedule, with headlines and text normalized to a unified format.
The key element is deduplication: an item is checked against what has already been published by canonical link and text fingerprint, so a repeated story from another feed does not create a second entry.
Images are transferred to the portal's media library upon publication so that the feed does not depend on the source.
What It Became
The daily volume grew to 40-60 items without expanding the editorial team, and the delay from event to publication was reduced to minutes. The share of duplicates fell to nearly zero, and the editor switched to reviewing and editing instead of copying. Publication stability no longer depends on whether a person is available at a particular hour.
1C-Bitrix remains the publication and storage system in this setup — the collection and verification logic lives separately and does not interfere with platform updates.
If your feed is populated manually and you are hitting the same ceiling — describe the task, and we will come back with an estimate and a plan.
How does a one-time collection differ from continuous updating?
A one-time collection is the loading of an existing dataset: you provide a list of sources and a depth (for example, materials from the last two years), an engineer configures page parsing, brings the fields into the structure of information blocks, and performs a single run.
The output is a populated section that then lives on its own: new entries appear only when someone from the editorial team adds them manually.
This mode solves the starting task — the site no longer looks empty, search engines receive a ready archive of materials, and the editorial team works with an already collected base rather than from scratch.
Continuous updating works differently: the same parsing, but launched on a polling schedule — once an hour, once a day, depending on how often the source publishes new content.
The key part here is not the collection itself, but protection against duplicates and junk: each entry is checked by canonical link and text hash, repeat hits are filtered out, updated materials are overwritten rather than created anew.
Without this, within a month the information block will end up with three copies of the same news item with different dates.
The difference shows up in maintenance. A one-time collection is work with a fixed result: the source changes its markup, the run breaks, but the next launch is scheduled manually, and until then the section simply is not replenished.
Continuous updating is a service with a run log, failure notifications, and adaptation of the parsing to changes in the source.
The archive of materials remains intact: old entries are not overwritten but supplemented with new ones, so rolling back to a previous state is possible at any stage.
The practical criterion for choosing is simple. If the task is to quickly get a section up and then run it with the editorial team's own efforts, a one-time collection is enough.
If the news flow must proceed without manual involvement and you are ready to maintain a polling schedule and monitoring, a continuous mode is needed — it is what provides uninterrupted population without downtime.
Common technical concerns
Before starting, clients usually check not whether the parser can pull news, but what will happen to the system over the long run: whether the database will hold up, whether the markup will break, what happens on a re-run, and what if the source goes down. Below are answers to the concerns that most often come up during the evaluation stage.
Will auto-filling create extra load on the database?
The flow is spread across a schedule and queues: import runs in batches rather than in one burst, inserts are grouped into packets, and the uniqueness of material is checked by the hash of the title and link before writing.
A news item found again does not create a duplicate and does not hit the tables an extra time. During peak hours, the exchange can be shifted to a night window so it does not compete with visitor traffic.
Is this compatible with the current markup and site templates?
Material arrives in the same fields and property types as with manual publishing, so output templates, SEF URLs, and catalog sections are not rewritten.
Images are pulled into the media library while preserving proportions, and the markup is cleaned of third-party code before insertion. If the source structure changes, only the parsing layer is adjusted — the public-facing part remains untouched.
What happens when the task is re-run?
The task is idempotent: each record is tagged with a source identifier, and a repeated pass skips what has already been loaded. An interrupted import continues from the last processed position rather than from the beginning. This removes the risk of duplicate publications and unnecessary requests to the external site.
How does the system behave when the source is unavailable?
A source failure does not bring down the exchange: retries with increasing delay kick in, and after several failures the task is flagged and a notification is sent to the responsible person. Already collected materials remain in the database, and nothing is lost. When the source comes back, the import catches up on what was missed by dates.
Let's discuss your auto-fill task
To come back with an estimate and a plan rather than generalities, we need at minimum five points of input from you. Below is exactly what we capture at the start: that is enough to design collection, normalization, and publication for your environment.
- List of sources — the specific feeds and sections that fill the site: open news streams, industry platforms, partner newsletters, internal databases. For each source, it is important to understand whether it provides data machine-readable or only as pages, whether there is authentication and rate limits on requests.
- Content requirements — which fields are mandatory (title, date, author, body, images, tags), what we do with duplicates and repeats, whether text needs to be reworked to match your style or we publish as is, which topics we exclude.
- Section structure — where to place materials: mapping to information blocks, categories, and tags, rules for selecting a section, whether manual moderation is needed before publication or the material goes to the site automatically.
- Update mode — how often we check sources, whether we publish immediately or in batches on a schedule, what we do when a source fails and how we roll back an erroneously uploaded entry.
- Accesses — credentials for sources, write permissions to information blocks, a test environment for a trial run, access to logs and monitoring so that failures are visible before they reach visitors.
If some points are unclear — that is normal, we will sort it out on a call and record it in the technical specification. Describe the task in free form: which sources, what volume of materials per day, and what already exists on the site side. We will come back with a timeline estimate and a work plan.