Two small files quietly control how search engines discover and crawl your entire website: the XML sitemap and robots.txt. Get them right and Google finds your new pages faster. Get them wrong — even by a single misplaced line — and you can accidentally hide important pages from search results without realizing it.
Quick answer: An XML sitemap is a list of URLs you want search engines to know about. Robots.txt is a set of rules that tells search engine crawlers which parts of your site they are allowed or not allowed to crawl. They serve opposite but complementary purposes — one invites crawling, the other restricts it.
What an XML Sitemap Actually Does
An XML sitemap is essentially a directory that lists the pages on your site you consider important enough to be indexed. It does not force Google to index every URL listed, but it does make discovery faster, especially for large sites, new sites, or pages that are not well linked internally.
A typical WordPress sitemap, generated automatically by Rank Math, groups content by type — posts, pages, categories — and updates automatically whenever you publish or update content. You do not need to manually edit this file in most cases, but you do need to check that it is accurate.
What Should — and Should Not — Be in Your Sitemap
- Include: service pages, blog posts, key landing pages, your core site pages (About, Contact).
- Exclude: tag archive pages, author archive pages, thank-you pages, internal search result pages, and any page marked “noindex.”
A sitemap that lists noindex pages, or pages that no longer exist, sends mixed signals to search engines and wastes crawl attention on pages you do not actually want ranked.
What Robots.txt Actually Does
Robots.txt sits at the root of your domain (yourdomain.com/robots.txt) and gives instructions to crawlers before they even start browsing your site. It can block entire folders — for example, your WordPress admin area — or specific file types from being crawled.
The critical thing to understand: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results (usually without a proper description) if other sites link to it, because Google never crawled it to know it should be excluded. To fully remove a page from search results, you need a “noindex” tag, not just a robots.txt block.
A Typical, Safe Robots.txt Structure
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://ecrystaltech.com/sitemap_index.xml
This example blocks crawlers from the WordPress admin area while still allowing the one admin-ajax file many plugins need, and points crawlers directly to the sitemap.
How the Two Files Work Together
Your sitemap says “these are the pages worth crawling.” Your robots.txt says “you are allowed to crawl these sections, but not those.” A sitemap referencing a URL that is simultaneously blocked in robots.txt sends a contradictory signal — Google sees the URL listed as important but is told not to crawl it.
| File | Purpose | Controls |
|---|---|---|
| XML Sitemap | Lists pages to discover | Discovery, not access |
| Robots.txt | Sets crawling permissions | Access, not indexing |
| Meta Robots / Noindex Tag | Controls search result inclusion | Indexing directly |
Different Types of Sitemaps on a Typical WordPress Site
Rank Math generates a sitemap index — a master file that links to several smaller sitemaps grouped by content type: one for posts, one for pages, one for each custom post type, and sometimes separate ones for images and categories. This structure keeps individual sitemap files manageable in size and makes it easier to spot which content type is missing pages if an issue arises. When checking your setup, review the sitemap index first, then drill into the relevant sub-sitemap for the content type you are troubleshooting.
Troubleshooting a Page That Won’t Get Indexed
If a specific page refuses to appear in search results despite being live, the sitemap and robots.txt are the first two places to check, in that order. Confirm the page’s URL actually appears in the relevant sitemap file — if it is missing, check whether it was accidentally set to “noindex” in Rank Math’s post settings, which automatically excludes it from the sitemap. If it is present in the sitemap but still not indexed, check robots.txt for any rule that might match that URL’s path, then verify there is no conflicting canonical tag pointing elsewhere.
Only after ruling out these technical causes does it make sense to consider content quality or lack of internal links as the reason a page is struggling to get indexed.
Special Considerations for Ecommerce Sitemaps
Ecommerce sites face a particular sitemap challenge: product availability changes constantly, and out-of-stock or discontinued products need to be handled deliberately rather than left indexed indefinitely. A discontinued product that remains in the sitemap and returns a normal page long after it is no longer available creates a poor experience for both search engines and shoppers who click through expecting to buy something that does not exist. Removing such pages from the sitemap, and either redirecting them to a similar in-stock product or returning an honest, helpful page explaining the discontinuation, keeps the sitemap an accurate reflection of what the business actually offers.
Common Mistakes That Hurt Rankings
- Accidentally disallowing the entire site with a line like
Disallow: /left over from a staging environment. - Blocking CSS or JavaScript files that Google needs to render the page properly, which can hurt how the page is understood and ranked.
- Forgetting to update the sitemap reference after migrating to a new SEO plugin.
- Listing noindexed or redirected URLs in the sitemap, confusing crawlers about which version of a page is canonical.
- Assuming robots.txt removes a page from Google when a proper noindex tag is what is actually needed.
Do Priority and Change Frequency Tags Still Matter?
Older sitemap standards included optional “priority” and “change frequency” fields, meant to hint at how important a page is and how often it updates. In practice, major search engines have publicly deprioritized reliance on these fields for years, largely because site owners set every page to maximum priority, making the signal meaningless. Modern sitemap tools, including Rank Math, still generate valid sitemaps without needing to fine-tune these fields manually, and spending time adjusting them rarely produces a measurable benefit today.
Coordinating Sitemap Changes With Your SEO Plugin
When switching SEO plugins — for example, moving from an older plugin to Rank Math — the old plugin’s sitemap files often remain accessible unless explicitly disabled, creating a confusing situation where two different sitemaps exist at different URLs simultaneously. Before submitting a new sitemap in Search Console, confirm the previous plugin’s sitemap generation is fully turned off and that any old sitemap URL either redirects to the new one or returns a proper 404, rather than continuing to serve outdated data indefinitely alongside the current one.
Step-by-Step: Checking Your Own Setup
- Visit yourdomain.com/robots.txt directly in a browser and read through every line.
- Visit yourdomain.com/sitemap_index.xml (or check Rank Math’s Sitemap Settings) to see what is currently included.
- In Google Search Console, confirm the sitemap is submitted under Indexing → Sitemaps and check for any reported errors.
- Search “site:yourdomain.com” in Google and compare what actually appears against what should be indexed.
- Fix any mismatch — remove old URLs from the sitemap, correct robots.txt rules, and resubmit.
Frequently Asked Questions
Do I need to manually create a sitemap on WordPress?
No. SEO plugins like Rank Math generate and update your XML sitemap automatically. You mainly need to verify it is submitted correctly in Google Search Console and periodically check its contents.
Can robots.txt block Google from indexing a page?
Not reliably. Robots.txt blocks crawling. If other websites link to a blocked page, Google can still list it in search results without crawling its actual content. Use a noindex meta tag for pages you truly want removed from search results.
How often should the sitemap be checked?
A quick check once a month is enough for most business sites, and especially after any major content restructuring, plugin change, or migration.
Should every page on the website be included in the sitemap?
No. Only pages you want indexed and ranked should be included — thank-you pages, internal search pages, and duplicate filtered URLs should typically be excluded or noindexed.
What happens if I accidentally block my whole site in robots.txt?
Search engines will stop crawling new content, and existing indexed pages may gradually drop out of search results over time. This is one of the fastest ways to lose organic visibility, so robots.txt changes should always be tested before publishing live.
Conclusion
Sitemap and robots.txt configuration is one of those technical tasks that takes only minutes to check but can cause serious, silent damage when misconfigured. A quick monthly review — confirming the sitemap reflects your real pages and robots.txt is not blocking anything important — prevents avoidable indexing problems long before they show up as a traffic drop.
If you want a second set of eyes on your current setup, eCrystal Digital Technology can review your sitemap, robots.txt, and indexing status and flag anything that needs correcting.
