Sitemaps, Robots.txt, and Redirects
The three technical basics that control how search engines find and keep your pages.
Get these three right and crawling runs smoothly. Get one wrong and good pages can disappear from search. Here's what each one does.
The three crawl controls
🗺️
XML sitemapA list of the pages you want found. It helps search engines discover your content faster.
🚦
Robots.txtTells crawlers where they can and can't go. Powerful, and easy to get wrong.
↪️
RedirectsSend an old URL to a new one, so links and rankings aren't lost.
XML sitemaps
Your sitemap lists the pages you want search engines to find, usually at yoursite.com/sitemap_index.xml. It doesn't force anything into the index. It helps Google find your pages faster, which matters most on new or large sites. Most SEO plugins, like Yoast and Rank Math, build it for you, and you submit it in Google Search Console.
Robots.txt
Your robots.txt file, at yoursite.com/robots.txt, tells crawlers which parts of the site they can visit. It's useful, and it's the easiest way to break your SEO by accident.
Robots.txt and noindex are not the same
Robots.txt controls crawling, whether a bot can visit a page. A noindex tag controls indexing, whether a visited page shows up in results. To keep a page out of Google, use noindex. A robots.txt block can stop Google from even seeing the noindex.
Redirects
A redirect points an old URL to a new one. A 301 is permanent, and passes ranking value to the new URL, the right choice for pages you've moved or deleted. A 302 is temporary and doesn't pass full ranking value, so it's only for short-term changes.
📋 Redirect best practices
✅ Use 301s for permanent moves
✅ Point to the closest replacement
✅ Avoid redirect chains and loops
✅ Map every URL after a migration
✅ Test with Search Console
✅ Never redirect entire domains blindly
How they work together
Think of it as one system. The sitemap points crawlers to your pages, robots.txt sets the boundaries, and redirects make sure old links still land somewhere useful. A problem in one can undo good work in the others, so a full SEO audit checks all three. You can confirm indexing issues in Google Search Console.
Check your crawlability
Sitemap, robots, and redirect issues are easy to miss and costly to leave.
Run a free audit that flags them.
Run a Free SEO Audit →FAQs
Yes, it helps search engines find your pages faster, especially on new or larger sites. Most SEO plugins create one automatically, and you submit it in Google Search Console.
It tells search engine crawlers which parts of your site they can visit. It controls crawling, not indexing, and a mistake in it can accidentally hide important pages from Google.
Robots.txt controls whether a crawler can visit a page. A noindex tag controls whether a visited page shows up in search results. To keep a page out of Google, use noindex, not a robots.txt block.
A 301 is permanent and passes ranking value to the new URL. A 302 is temporary and doesn't pass full ranking value. Use 301s for pages you've permanently moved or renamed.
Migrations and rebrands change URLs. Without 301 redirects mapping old URLs to new ones, you lose rankings and visitors hit dead pages. Mapping every old URL is essential.