Sitemap

A sitemap is a file that lists the pages, videos, images, and other content on a website, along with the relationships between them. Search engines like Google read this file to crawl a site more efficiently — discovering new pages, identifying recently updated content, and understanding which URLs the site owner considers most important. Think of it as a structured table of contents built for search engine crawlers rather than human readers.

Sitemaps come in two primary forms: XML sitemaps, machine-readable files designed for search engine bots, and HTML sitemaps, visible pages that help human visitors navigate a site. Both serve the same core purpose — making content findable — but they target different audiences.

What Is a Sitemap Used For?

A sitemap provides search engines with a structured roadmap of a website. Its primary function is to supplement the normal crawling process, in which bots follow links from one page to another. When internal linking is imperfect — and on large sites it often is — a sitemap fills the gaps.

Specifically, a sitemap helps search engines:

• Discover orphaned or deeply buried pages that lack internal links from other parts of the site.
• Identify new and recently updated content through the <lastmod> metadata tag, reducing the time between publication and indexation.
• Understand content types — video, image, and news sitemaps provide specialized metadata (running time, publication date, image location) that enriches how content appears in search results.
• Process large sites within crawl budget constraints by signaling which URLs matter most.

For a typical website, Google can discover most pages through normal link-following. The sitemap becomes critical when that assumption breaks down — on sites with thousands of product pages, dynamic URLs, limited navigation, or rich media that normal crawlers struggle to parse.

Launch your affiliate program today

Set up advanced tracking in minutes. No credit card required.

Request demo Start 30-days free trial

Types of Sitemaps

Not all sitemaps are the same file. The sitemap protocol, originally proposed by Google, Yahoo!, and Microsoft in 2005 and hosted at sitemaps.org, defines a family of formats, each tailored to a specific content type.

XML Sitemap

The <lastmod> tag is the most practically useful of the optional fields. Google has stated that it largely ignores <changefreq> and <priority>, but <lastmod> helps the crawler decide whether a page needs reindexing.

HTML Sitemap

An HTML sitemap is a visible, user-facing page that lists a site’s important sections and pages in a hierarchical layout — typically grouped by category. It functions as an alternative navigation aid, especially on large content sites where the main menu cannot link to every page.

Specialized Sitemaps

The sitemap index file is worth special attention. When a site outgrows the 50,000-URL or 50 MB limit (whichever comes first), the sitemap protocol requires splitting entries across multiple sitemap files and referencing them all from a single sitemap index. Most CMS plugins handle this automatically.

How Does a Sitemap Look?

An XML sitemap viewed in a browser appears as a structured but unstyled document — tags, URLs, and metadata in plain text. It is not meant to be read by humans in raw form. An HTML sitemap, by contrast, looks like any other webpage: headings, categorized links, and sometimes brief descriptions.

You can find most sitemaps by appending /sitemap.xml to a domain (e.g., https://example.com/sitemap.xml). Other common locations include /sitemap_index.xml, /sitemap-index.xml, or paths specified in the site’s robots.txt file under a Sitemap: directive.

Join our newsletter

Be the first to know about new features and product updates.

Email address

Subscribe

Why Do You Need a Website Sitemap?

The short answer: a sitemap improves the odds that search engines find and index your content, especially when your site is large, new, or media-rich. The longer answer depends on what kind of site you run.

When a Sitemap Is Necessary

• Large sites (1,000+ pages): Internal linking at scale is hard to maintain. A sitemap ensures that pages buried deep in the architecture — or added without being linked from a category page — are still discoverable.
• New sites with few external links: Google discovers pages by following links. A brand-new site with minimal backlinks relies on its sitemap as the primary discovery path.
• Sites with rich media: Video and image content often lives in players or galleries that crawlers cannot parse without explicit metadata in a specialized sitemap.
• Sites that update content frequently: News publishers, active blogs, and e-commerce stores with rotating inventory benefit from <lastmod> tags that tell crawlers what changed and when.
• Sites with JavaScript-rendered content: Pages that depend heavily on client-side rendering can be invisible to crawlers without a sitemap to point the way.

Do I Need a Sitemap for a Small Website?

That said, there is no downside to having one. Most content management systems, including WordPress, generate XML sitemaps automatically through plugins like Yoast SEO or Rank Math. The cost of having a sitemap is effectively zero, while the cost of missing one — even on a small site — is that a page you care about might go undiscovered.

How to Create a Sitemap

Using a CMS Plugin

If your site runs on WordPress, the easiest path is an SEO plugin. Yoast SEO and Rank Math both generate XML sitemaps automatically and update them whenever you publish, edit, or delete content. The sitemap regenerates on the fly — no manual intervention needed. Yoast SEO is free for basic sitemap functionality; Rank Math offers a free tier with sitemap support as well.

For other CMS platforms:

• Wix, Squarespace, Shopify: Sitemaps are generated automatically. No action required.
• Drupal: The Simple XML Sitemap module handles generation and submission.
• Concrete CMS: Built-in sitemap generation is available from the dashboard.

Manual Creation

For small, static sites, you can write an XML sitemap by hand. Create a file named sitemap.xml, place it in your site’s root directory, and follow the sitemap protocol structure. Ensure the file is UTF-8 encoded and that all special characters are properly escaped (use & for &, " for ").

How Sitemaps Optimize Crawling

Crawl budget — the number of pages Googlebot will crawl on your site in a given period — is a finite resource. On large sites, inefficient crawling means important pages wait in line behind low-value ones. A well-maintained sitemap optimizes crawling in three ways:

1. Prioritization: By including only canonical, indexable URLs (HTTP 200 status, no noindex directive), the sitemap tells crawlers which pages deserve attention.
2. Change detection: The <lastmod> tag allows crawlers to skip pages that have not changed since the last visit, conserving crawl budget for updated content.
3. Discovery of isolated content: Pages that exist outside the main navigation — landing pages, campaign-specific URLs, PDF resources — get surfaced through the sitemap rather than being missed entirely.

Using a CMS Plugin

If your site runs on WordPress, the easiest path is an SEO plugin. Yoast SEO and Rank Math both generate XML sitemaps automatically and update them whenever you publish, edit, or delete content. The sitemap regenerates on the fly — no manual intervention needed. Yoast SEO is free for basic sitemap functionality; Rank Math offers a free tier with sitemap support as well.

For other CMS platforms:

• Wix, Squarespace, Shopify: Sitemaps are generated automatically. No action required.
• Drupal: The Simple XML Sitemap module handles generation and submission.
• Concrete CMS: Built-in sitemap generation is available from the dashboard.

Using a Standalone Generator

For static sites or custom setups, online tools like XML-Sitemaps.com crawl your site and produce a sitemap file you can download and upload.

Manual Creation

For small, static sites, you can write an XML sitemap by hand. Create a file named sitemap.xml, place it in your site’s root directory, and follow the sitemap protocol structure. Ensure the file is UTF-8 encoded and that all special characters are properly escaped (use & for &, " for ").

How to Submit a Sitemap to Google

Creating the file is only half the task. Search engines need to know it exists.

1. Upload the sitemap to your site’s root directory (e.g., https://yoursite.com/sitemap.xml).
2. Reference it in robots.txt: Add the line Sitemap: https://yoursite.com/sitemap.xml to your robots.txt file. This is the first place crawlers look.
3. Submit via Google Search Console: Go to the “Sitemaps” section under “Indexing,” enter the sitemap URL, and click “Submit.” Google will process the file and report any errors.
4. Submit via Bing Webmaster Tools: Follow the same process for Bing, which also powers Yahoo and DuckDuckGo search results.

After submission, monitor the Search Console sitemap report periodically. It shows how many submitted URLs were indexed, flags errors (broken URLs, redirects, blocked resources), and confirms the last time Google processed the file.

How to Review Your Sitemap for Issues

A sitemap that contains errors is worse than no sitemap at all — it wastes crawl budget and can confuse crawlers. Review yours regularly by checking:

Run a review after any major site restructure, migration, or content audit. For most sites, a monthly check in Search Console is sufficient.

How Big Is Too Big for a Sitemap?

The sitemap protocol sets two hard limits per sitemap file:

• 50,000 URLs maximum
• 50 MB uncompressed file size maximum

If your site exceeds either limit, you must split the sitemap into multiple files and use a sitemap index file to reference them all. A sitemap index is itself an XML file that lists each child sitemap:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages-1.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-posts-1.xml</loc>
  </sitemap>
</sitemapindex>

What Should Not Be Included in a Sitemap

A sitemap should list only the URLs you want search engines to index. Exclude:

• Non-canonical URLs — only the canonical version of each page belongs in the sitemap.
• Redirected URLs (301, 302) — include the destination, not the redirect.
• Pages with a noindex meta tag — contradictory signals confuse crawlers.
• Blocked URLs — if robots.txt blocks a URL, including it in the sitemap creates a conflict.
• Duplicate or paginated pages — unless paginated pages have unique, indexable content worth surfacing independently.
• Admin pages, login pages, and thank-you pages — these have no search value.

A clean sitemap signals quality to search engines. A cluttered one signals neglect.

How Often Should You Update a Sitemap?

The sitemap should reflect the current state of your site at all times. For most sites, this means:

• Dynamic sites (blogs, news, e-commerce): The sitemap should regenerate automatically with every content change. CMS plugins handle this.
• Static sites: Regenerate and re-upload the sitemap after any structural change or content addition.
• Minimum cadence: Even on a static site, review and resubmit the sitemap at least once per quarter to catch any drift.

The <lastmod> tag should accurately reflect when each page was last meaningfully updated — not when the sitemap file itself was regenerated.

Sitemaps and Affiliate Marketing

For anyone running an affiliate program — whether as an advertiser managing a program, a publisher creating content, or a program manager overseeing operations — sitemaps play a specific, practical role in organic visibility.

For Advertisers (Brands Running Affiliate Programs)

Your affiliate program’s landing pages, signup forms, and resource hubs need to be indexed. An XML sitemap ensures that search engines discover:

• Program information pages — how your affiliate program works, commission structures, and terms.
• Affiliate resource centers — banners, product feeds, and promotional guidelines that publishers search for.
• Product pages — especially important if you run a large e-commerce site where new products are added frequently.

When publishers search for “[brand] affiliate program,” the pages you want them to find must be in Google’s index. A sitemap helps search engines discover those URLs, but submission does not guarantee indexing or a faster crawl.

For Publishers (Affiliates Creating Content)

Affiliate sites often have hundreds or thousands of content pages — product reviews, comparison tables, buying guides. These pages compete for organic traffic, and every unindexed page is a missed earning opportunity. A well-structured XML sitemap:

• Helps search engines discover new review and comparison content faster, reducing the time between publication and potential traffic.
• Ensures that deeply nested content — such as individual product reviews linked only from paginated category pages — is still findable.
• Provides <lastmod> signals when you update older content, which is common in affiliate marketing as products change, prices shift, and recommendations evolve.

An HTML sitemap can also serve publishers by giving visitors a browsable overview of all content categories — useful on review sites where users want to browse by product type rather than search.

For Affiliate Program Managers

Program managers sit between advertisers and publishers. Your role touches both sides: you need the program’s own pages indexed, and you benefit when your publishers’ content ranks well. Practical steps:

• Audit your program’s sitemap in Google Search Console to confirm all key pages are submitted and indexed.
• Provide publishers with a sitemap of your product feed — a structured list of product URLs they can reference when building content.
• Check that your affiliate platform’s tracking links are not included in the sitemap — redirect URLs and tracking parameters should never appear in a sitemap file. Only canonical, user-facing pages belong there.

Sitemap vs. Robots.txt: What Is the Difference?

These two files are often mentioned together but serve opposite functions:

A common mistake: blocking a URL in robots.txt while also including it in the sitemap. Crawlers encountering this contradiction will typically respect robots.txt and skip the URL — but the inconsistency signals poor site maintenance. Keep both files aligned.

Sitemap Best Practices

• Include only canonical, indexable URLs — status 200, no noindex, no redirects.
• Use the <lastmod> tag accurately — it is the only optional metadata field Google treats as a meaningful signal.
• Reference the sitemap in robots.txt — this is the first place crawlers check.
• Keep each sitemap file under 50,000 URLs and 50 MB — use a sitemap index file for larger sites.
• Regenerate dynamically — your sitemap should never be stale. CMS plugins should handle this automatically.
• Exclude paginated, filtered, and parameter-based URLs unless they represent unique, indexable content.
• Review the Search Console sitemap report monthly — catch and fix errors before they accumulate.