XML sitemap: why search engines don't see new pages
New pages are published, they're in the menu and internal links, but the search results show the old set. A familiar picture: the search engine simply doesn't know that you have new URLs.
The robot crawls the site by following links, and if a page is buried deep in the catalog or only opens through a filter, it will reach it in weeks, or even months.
An XML sitemap closes this gap: it's a file with a list of URLs that you give to the search engine directly, bypassing crawling by links.
The problem is almost never the absence of the file, but its condition. The sitemap was generated once when the site launched and hasn't been updated since — there are no new URLs in it.
Or it includes everything indiscriminately: pagination pages, filter parameters, duplicates with different endings. The robot spends its limited crawl budget on this junk and never gets to the real pages.
The third common scenario — the file exists, but it's served with an error, isn't located at the standard address, or is blocked in robots.txt, and the search engine simply doesn't read it.
A separate story is priorities and update frequency. If all URLs have the same weight, the search engine doesn't understand what matters more to you: a product page or a utility page.
Set priorities and correct last-modified dates direct crawling to where the content actually changes.
We start with diagnostics: we look at what the server actually serves, which URLs made it into the sitemap and which fell out, how the file relates to robots.txt and the site structure.
Then — generation and configuration for your type of resource, registration in webmaster panels, and verification that new pages enter the index within days, not months. We'll work out how this gets closed on your site.
What a correct XML sitemap delivers
An XML sitemap is not a formality for a report, but a working tool that sets the order of crawling for search engines: which addresses exist, when they changed, how important they are.
When the file is generated automatically from the database and served with correct headers, a new section or product page enters the crawl queue within hours, not weeks.
If the sitemap is assembled manually or is outdated, the robot walks through broken links, pulls duplicates, and leaves out exactly those pages that should be bringing traffic.
The first measurable effect is indexing speed. Fresh addresses that land in the sitemap with an up-to-date lastmod are discovered and crawled noticeably faster: the share of pages indexed within the first day after publication grows severalfold.
This directly affects how quickly a new product, service, or article starts appearing in search results.
The second effect is index coverage. When the sitemap contains no junk addresses (sorting parameters, pagination pages, duplicates with ?utm), the robot does not spend its crawl budget on them and reaches useful content.
The share of indexed meaningful pages grows, and sections that "dropped out" of search come back.
The third effect is fewer crawl errors. A valid sitemap with correct response codes removes 404s and "soft" errors from reports, while crawl priorities and update frequency distribute weight between sections: key ones are crawled more often, archival ones less often.
As a result, search engines get an unambiguous and honest picture from the site, rather than a set of contradictory signals.
Let's break down how to build the generation and registration of the sitemap so that these effects are stable rather than one-off.
Generation methods: manual, scripted, on the fly
The three approaches differ not by tool, but by the moment when an address enters the map: before publication, during the build, or at the moment of the request.
This determines how quickly a new section or product becomes visible to search engines and how much manual work each release entails.
A static file is the most predictable option: the map sits in the root and is served as an ordinary document. It is convenient when the structure rarely changes and the addresses are stable.
There is one limitation — every edit is made by hand, and on a catalog with thousands of items, a discrepancy with the actual addresses appears within a couple of weeks.
Scripted generation removes the manual work: the build reads the database or the page markup and rebuilds the map on every release.
A dynamic map is formed on the fly upon request and always reflects the current state of the catalog, but it requires the application to respond quickly and not serve utility addresses to the search robot.
| Option | When it fits | Limitations |
|---|---|---|
| Static file | Landing pages, small sites, rare structure edits | Manual edits, discrepancies with actual addresses as the catalog grows |
| Scripted generation | Catalogs and blogs where addresses come from a database or markup | A build step is needed in the release process, errors in the data break the map |
| Dynamic map | Marketplaces, frequently changing selections and filters | Load on the application, risk of serving duplicates and utility pages to the robot |
In practice, the approaches are combined: a static map covers rarely changing sections, a script covers the catalog, and dynamic generation covers what is updated daily.
This way automation does not become a single point of failure, and the map stays up to date without manual edits.
How we implement an XML sitemap: stages of work
We start with an indexing audit: we check what search engines already know about the site, which sections have dropped out of the index, where duplicates and broken URLs are.
At this stage it becomes clear whether an XML sitemap is needed at all or whether the problem can be solved with proper redirects and canonical URLs.
- Audit of the current state. We compile a list of pages, check server responses, find sections closed to crawling, duplicates, and "orphaned" URLs that have no internal links.
- Determining the sitemap composition. We decide which page types go into the file: products, categories, articles, pagination pages. We exclude service and technical URLs so as not to waste the crawl limit.
- Setting priorities and update frequency. We assign priorities based on the importance of sections and set the change frequency — this way the search robot returns more often to important pages and less often to archived ones.
- Generation and markup. We generate the file in the correct format, split it into multiple sitemaps when the volume is large, and link them with an index file. We check the markup for compliance with search engine requirements.
- Registration and markup verification. We specify the sitemap in the directives file for robots and submit it in the webmaster panel, then monitor how the search engine reads it and which URLs it accepts.
- Update automation. We tie generation to content changes: a new page or a URL edit — the sitemap updates itself, without manual upload.
- Handover and regulations. We hand over the configured sitemap, document the update procedure and control points so that support does not depend on a single person.
We close each stage with a check: the sitemap must return a correct server response, contain only URLs available for crawling, and match the actual site structure. If this is not the case, the search engine receives contradictory signals and crawling slows down.
What the client receives
We hand over not a "sitemap.xml file" but a complete package that your team can use to maintain the map independently, without us. Everything described below is aligned with the current site structure and verified against real indexing.
- Generated map files. The main map with all indexable URLs, separate maps by section, and an image map. Splitting by section is needed when there are many URLs: search engines are more willing to fetch small files in full rather than cut off the download halfway.
- Generation scheme. The scheme shows where URLs come from: from the database, from application routes, or from the publication queue. It shows at which step a URL enters the map and why some pages are not included in it.
- Priority rules. A mapping table: page type, priority, change frequency. The homepage, categories, product cards, articles, and utility pages receive different values — without manual editing on every publication.
- Sitemap documentation. What the files are, where they are located, who fetches them and at what address, what to do when the domain or section structure changes.
- Update regulations. Rebuild triggers: publishing a piece of content, changing a URL, deleting a page, scheduled nightly build. With an indication of who is responsible for each step.
- Configured submission to search engines. The map is registered, linked to the domain, and the processing status is visible in the panels. Plus a rule that reports an update automatically, rather than relying on someone's memory.
- Indexing report. A snapshot: how many URLs were submitted, how many were accepted, which were rejected and with what wording. It shows where the map and the actual pages diverge.
- Acceptance checklist. A short list of checks before each release: the map returns a correct response code, it contains no closed sections, and the URLs match the canonical ones.
Each item is a working artifact, not a description. Describe the task — we will come back with an estimate and a plan: what we do on your side, what we take on ourselves, and in what order.
Case study: how a growing share of indexed pages accelerated the launch of new products
The client — a chain of online stores on 1C-Bitrix — complained that new sections and product cards were "hanging" without traffic: search engines found them through internal links, crawling dragged on, and some pages remained out of the index altogether.
The catalog was growing faster than search could crawl it, and every new product waited weeks for its first impressions.
What it was before
The sitemap was generated manually: once a month a developer exported a list of URLs with a script, checked it by eye, and handed over the file. The sitemap had no modification dates, priorities were all the same, and new sections only got into it after the next export. Search engines saw an outdated snapshot of the structure and crawled the site on a leftover basis.
What was changed
We set up sitemap generation on the application side: it is built from the database on every catalog update, serves up-to-date addresses with the last modification date, correct priorities, and a split into several files with an index.
For product cards and articles we added separate sitemaps so that search would see fresh materials right away. We configured auto-updating and registration in webmaster panels, as well as monitoring: if the sitemap stops updating or returns an error, a notification is sent.
What it became
The share of indexed pages grew from 62% to 94% in a month and a half. New sections and cards began to get into the index within 2–4 days instead of 2–3 weeks — bringing new products into search sped up severalfold.
Search traffic to new sections started coming in during the very first week after launch, rather than a month later.
The effect on crawling was noted separately: search robots stopped spending resources on outdated addresses and visited fresh pages more often. This is a case where a technical fix is several times cheaper than the subsequent struggle for visibility of every new product.
How an XML sitemap differs from an HTML sitemap and robots.txt
An XML sitemap is a machine-readable list of URLs that a search robot parses line by line: <loc>, <lastmod>, <changefreq>, <priority>.
It is not for people but for crawling: the robot understands which pages exist, when they were edited, and how important they are relative to the rest.
An HTML sitemap solves a different task — it is a page for the visitor, where links are grouped by section so that a person does not get lost in the structure. Search engines read it too, but as an ordinary page with links, without priorities and dates.
robots.txt is not a sitemap but access rules. The Disallow directive closes service sections from crawling, Sitemap: indicates where the XML file is located, and Crawl-delay limits the request rate. robots.txt itself does not list URLs — it only allows or forbids their crawling.
Therefore the combination is built as follows: robots.txt directs the robot to the sitemap, the XML sitemap lists what to crawl and with what priority, and the HTML sitemap remains navigation for people.
The differences show up in format and purpose. An XML sitemap is a service file that is not visible in the interface and does not appear in search results as a page. An HTML sitemap is a full-fledged page with markup, indexed on par with the rest.
robots.txt is a text file in the domain root that the robot requests first. None of the three tools replaces another: robots.txt without a sitemap will not tell about new sections, and a sitemap without a correct robots.txt may go unnoticed.
When each is needed: an XML sitemap is mandatory for sites with dynamic sections, filters, pagination, and frequent updates — there the robot does not always reach the content via links.
An HTML sitemap is useful on large portals and in online stores, where the visitor needs an overview of categories. robots.txt is always needed — at least to close the admin panel, cart, and sorting parameters from unnecessary crawling.
If the XML sitemap and robots.txt contradict each other, the robot follows the prohibitions, and some URLs drop out of the index.
In practice, the three tools work in one bundle, and configuring them separately is pointless. We check that the sitemap serves only canonical URLs, that sections closed in robots.txt are not duplicated in the sitemap, and that the HTML version does not create competing pages.
This order eliminates typical indexing errors — duplicates, "orphaned" pages, and junk in search results.
Common technical concerns
A sitemap is an XML file that provides search engines with a list of URLs, the date of the last modification, priority, and update frequency. Most concerns around it boil down to four topics: limits, rebuild frequency, priorities, and compatibility with the current site. Let's go through each one.
Is there a limit on the number of URLs in a single file?
Yes, the format has an upper bound — 50,000 URLs and 50 MB per file. If there are more pages, the sitemap is split into several files and linked together by an index file, which the search engine reads as an entry point.
For large catalogs and e-commerce stores, this is a standard mode rather than a workaround: the index is assembled automatically during generation, and you don't need to manually monitor the structure.
How often should the sitemap be updated?
The sitemap should be rebuilt whenever the site structure changes — when pages are published, deleted, or moved. In practice, this is tied to an event: a new record in the database or a deploy triggers generation, and the file stays up to date without manual actions.
If the site is updated infrequently, a regular scheduled rebuild is sufficient — the main thing is that the dates in the file reflect actual edits rather than being automatically applied to all URLs.
What should be done about priorities and update frequency?
Priority is a hint of relative importance from 0.0 to 1.0, not a guarantee of ranking. It's reasonable to set 1.0 for the homepage, 0.8 for sections, and 0.5 for typical pages — that's enough for the search engine to understand the structure.
Update frequency describes how often a page changes: for news and product pages it's higher, for static sections it's lower. The values don't affect indexing directly, but they help the crawler plan its crawl.
Is this compatible with the current site, and what about validation errors?
The sitemap doesn't require reworking the site: the file is generated separately and served at a standard URL, and a link to it is specified in the rules file for crawlers.
Typical validation errors — incorrect date format, unescaped characters in URLs, broken links, and extra spaces — are caught by a check before publication.
We run the sitemap through a validator and verify the URLs against real pages so that the search engine doesn't receive error responses.
If your case isn't on the list — describe your task, and we'll get back to you with an estimate and a plan.
Let's discuss your XML sitemap task
Describe the current situation: how many pages the site has, what platform it runs on, how the XML sitemap is currently generated, and what exactly is not working — search engines don't see new sections, priorities are set formally, the file is updated manually after every publication.
That's enough for us to come back with a project estimate and a work plan, without lengthy back-and-forth and clarifying rounds.
What the client gets as a result:
- An analysis of the current state: how the XML sitemap is served now, which URLs make it in and which are lost, how priorities and update frequency are configured.
- A work plan with stages: generation, rule configuration, registration in webmaster panels, indexing control.
- An estimate of timelines and scope — based on the facts of your site, not a template price list.
- An automatic update scheme so the file doesn't have to be assembled by hand after every release.
- A checklist: which URLs we exclude, how we handle pagination, filters, and duplicates.
- A risk analysis: what will break when the URL structure changes and how to address it in advance.
- A consultation with an engineer on contentious points — which sections should be exposed to search engines and which are better closed off.
We don't take on a task "blindly": first we look at how the site is built, then we name the timelines and scope of work. If an XML sitemap already exists but works poorly, we'll figure out exactly where URLs are being lost and why new pages aren't getting indexed.
Let's discuss the task — tell us what's happening with indexing right now, and we'll come back with an estimate and a work plan.
