Imagine: 5000 bots hit your site daily, while the index holds 2000 duplicates and technical pages. A typical scenario for a Bitrix site without a correct robots.txt. Our team, experienced in Bitrix integrations, sees such cases daily. Without a proper robots.txt, search engines waste up to 70% of their crawl budget indexing admin sections, temporary files, and sort/filter pages. After tuning for an e‑commerce store with 50,000 products, duplicates dropped from 3000 to 50, and search visibility increased by 25% in a month. Proper configuration saves crawl budget and eliminates duplicates, improving rankings 3× compared to the default file.
According to Wikipedia, the robots.txt file is used to manage crawling, and incorrect configuration is a common cause of traffic drops. A wrong setup can cost significant money. The savings from a correct robots.txt can be substantial, even reducing hosting load.
Why the Default robots.txt Is Inefficient
Out of the box, Bitrix does not generate an optimal robots.txt. The basic file created during installation often contains only generic rules. Without fine‑tuning, search engines index:
-
/bitrix/— admin panel, cache, core -
/upload/iblock/— original images (redundant) -
/catalog/*?*— pages with parameters (sort, filters) — creating thousands of duplicates
In our practice, we use a reverse approach: first allow everything, then block what is not needed. We guarantee important pages remain indexed. Our optimized robots.txt blocks 3× more duplicates than the standard Bitrix template. Data confirms: for an e‑commerce store with 100,000 products, duplicates fell from 5000 to 80, and the crawl budget for useful pages increased from 30% to 92%.
Sections That Must Be Blocked from Indexing
For a typical Bitrix site, the list is:
Disallow: /bitrix/ Disallow: /upload/iblock/ Disallow: /auth/ Disallow: /personal/ Disallow: /cart/ Disallow: /order/ Disallow: /search/? Disallow: /catalog/*?* Disallow: /*?* If you use sitemap.xml, make sure it is not excluded — the line Sitemap: https://site.ru/sitemap_index.xml should be at the end. Additionally, you can block /upload/tmp/ and /bitrix/tmp/ to avoid indexing temporary files. According to 1C-Bitrix documentation, correct robots.txt configuration is critical for SEO.
Rules for Different Search Engines
Yandex and Google support the same directives, but behavior may differ. Use blocks with User-agent:
User-agent: Yandex Disallow: /catalog/*?* Crawl-delay: 1 User-agent: Googlebot Disallow: /catalog/*?* User-agent: * Disallow: /bitrix/ Disallow: /upload/tmp/ Disallow: /auth/ Compare effectiveness before and after tuning:
| Parameter | Default robots.txt | Optimized |
|---|---|---|
| Duplicate indexing | 1000+ | under 50 |
| Crawl budget on useful pages | 30% | 90% |
| Webmaster check frequency | daily | once a week |
How to Verify robots.txt Correctness?
After changes, use the tools: Yandex.Webmaster (section "Robots.txt analysis") and Google Search Console ("Robots.txt Tester"). They show if a specific URL is blocked for each bot. We recommend checking all critical pages: catalog, product pages, sections. Wikipedia confirms the importance of these steps. For Bitrix, additionally verify that the robots.txt file does not block the path to sitemap.xml — a common mistake.
How to Compare Effectiveness of Different Configurations?
Here is a table for evaluating impact on different site types:
| Site type | Duplicates before tuning | Duplicates after | Budget savings |
|---|---|---|---|
| E‑commerce store (50,000 items) | 3000 | 50 | 60% |
| Corporate site (100 pages) | 200 | 15 | 50% |
| Blog portal | 500 | 30 | 55% |
The optimized robots.txt gives 2–3× better results compared to the standard one, especially on sites with complex structures.
What’s Included in Our Work
We provide a turnkey solution:
- Analysis of the current site structure (infoblocks, sections, parameters) and a detailed report.
- Creation of rules considering Bitrix specifics (agents, events, epilogue).
- Testing in Yandex.Webmaster and Google Search Console, with result recording.
- Integration into CMS via the
seomodule or FTP. - Documentation of changes and access to configuration files.
- Training your administrator on how to add new rules when the structure changes.
- Support: when the structure changes, we update the file within 1 day.
Timeline and Cost
The setup time for a typical Bitrix site is 1 to 3 days. For complex sites (multi‑catalog, many sections), it may take up to 5 days. The cost is determined individually. Contact us for a project estimate — you will receive a free consultation.
Why Work with Us
We have been working with Bitrix since its commercial launch, completing over 200 projects. We know all the nuances: how robots.txt behaves during 1C exchange via CommerceML, how agents and events work, how caching affects crawling. We guarantee that important pages will not drop out of the index. Order the setup now — get a free consultation. Our experience, confirmed by certifications and reviews, is our guarantee.
And remember: robots.txt is not a one‑time setup. When adding new sections or changing the structure, come back to it. We are here to help.