XML Sitemaps

01 The Rule

Every site must provide an XML sitemap listing all canonical, indexable URLs. Sites exceeding 50,000 URLs must use a sitemap index. Only include URLs that return 200 and are not blocked by robots.txt or noindex.

02 Rationale

Sitemaps are the most direct way to tell search engines about your URL space. For large sites, sitemaps are often the primary discovery mechanism for deep content that isn't well-linked internally. Accurate lastmod values drive crawl prioritization.

03 Implementation

• Maximum 50,000 URLs per sitemap file, 50MB uncompressed
• Use <sitemapindex> for sites exceeding 50K URLs
• Include accurate <lastmod> dates — only update when content changes
• Segment sitemaps by content type (products, categories, blog)
• Gzip compress sitemap files
• Reference sitemap location in robots.txt

04 Common Violations & Consequences

05 The Fix

Generate sitemaps programmatically from your canonical URL database. Validate that every sitemap URL returns 200 and matches canonical tags. Segment by content type and update lastmod only on actual content changes.