robots.txt is a small but powerful file that controls which pages search engines crawl on your website. A misconfigured robots.txt can accidentally block your entire site from Google — or waste crawl budget on pages that shouldn't be indexed. This guide covers everything from basic syntax to advanced crawl budget optimization.
⚠️ Common Mistake: Never add Disallow: / to your robots.txt in production — it blocks all crawlers from your entire site and can cause complete de-indexing. Always test changes before deployment.
Verify your robots.txt isn't blocking important pages
PageGuard checks if your key pages are accessible to search engines and flags technical SEO issues that prevent indexing.
robots.txt is a plain text file that lives at the root of your domain: https://yourdomain.com/robots.txt. It's part of the Robots Exclusion Protocol (REP), a web standard that defines how crawlers should behave when visiting websites.
When a crawler visits your site, it reads robots.txt first to understand which paths are allowed or disallowed. Following robots.txt instructions is voluntary — ethical crawlers like Googlebot, Bingbot, and Yandexbot respect it, while spam bots and scrapers often ignore it entirely.
Example robots.txt file:
User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /private/
Allow: /public/
User-agent: Googlebot
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlThe * wildcard in User-agent means "all crawlers." Specific crawlers like Googlebot can be given different rules.
The robots.txt format is simple but has important rules:
| Directive | Example | Meaning |
|---|---|---|
| User-agent: | User-agent: * | Which crawler the following rules apply to. * = all crawlers. |
| Disallow: | Disallow: /admin/ | Path the crawler must not visit. Empty Disallow means allow everything. |
| Allow: | Allow: /admin/public/ | Exception to a Disallow rule. More specific paths take precedence. |
| Sitemap: | Sitemap: https://domain.com/sitemap.xml | Full URL of your sitemap. Applies to all crawlers (not User-agent specific). |
| Crawl-delay: | Crawl-delay: 10 | Seconds between crawler requests. Supported by Bing but NOT Google (Google ignores it). |
Pattern matching wildcards:
* — matches any sequence of characters$ — matches the end of a URL patternDisallow: /*.pdf$ blocks all PDF filesDisallow: /*?* blocks all URLs with query stringsThese page types are good candidates for Disallow in robots.txt:
Disallow: /admin/
Admin panels and CMS backends — no SEO value, security risk
Disallow: /login
Login and authentication pages
Disallow: /dashboard/
User dashboards and private account pages
Disallow: /api/
API endpoints that return JSON/data (not human-readable pages)
Disallow: /search?
Internal search results — near-infinite duplicate content
Disallow: /cart
Shopping cart and checkout pages — no indexing value
Disallow: /wp-admin/
WordPress admin (though wp-login.php also needs protecting)
Disallow: /staging/
Staging or development sections accessible on production
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /wp-json/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Sitemap: https://yourdomain.com/sitemap_index.xmlUser-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout
Disallow: /orders/
Disallow: /account/
Disallow: /collections/*?*
Disallow: /products/*?*
Sitemap: https://yourdomain.com/sitemap.xmlUser-agent: *
Disallow: /dashboard/
Disallow: /login
Disallow: /register
Disallow: /api/
Disallow: /auth/
Disallow: /settings/
Disallow: /report/
Sitemap: https://yourdomain.com/sitemap.xml# Block OpenAI training crawler
User-agent: GPTBot
Disallow: /
# Block Anthropic training crawler
User-agent: ClaudeBot
Disallow: /
# Block Google AI training (not Google Search)
User-agent: Google-Extended
Disallow: /
# Block Common Crawl (used by many AI datasets)
User-agent: CCBot
Disallow: /
# Block Meta AI crawler
User-agent: Meta-ExternalAgent
Disallow: /Googlebot has specific behavior and several specialized sub-crawlers:
| User-agent | Purpose |
|---|---|
| Googlebot | Main Google web crawler for Google Search indexing |
| Googlebot-Image | Crawls images for Google Image Search |
| Googlebot-Video | Crawls video content for Google Video Search |
| Googlebot-News | Crawls news content for Google News |
| Google-Extended | Google AI training (Bard/Gemini) — separate from Search |
| AdsBot-Google | Google Ads quality checking — not controlled by Googlebot rules |
Important: Rules for Googlebot also apply to all Google sub-crawlers (Googlebot-Image, Googlebot-Video, etc.) unless overridden by a more specific rule. AdsBot-Google requires its own explicit rule to be blocked.
Understanding when to use each method is crucial for correct SEO:
| Scenario | Use | Why |
|---|---|---|
| Admin / private pages | robots.txt Disallow | Don't want crawlers accessing them at all |
| Pages you want crawled but not indexed | noindex meta tag | Crawler must read the page to see the noindex signal |
| Thin / duplicate content pages | noindex meta tag | Control indexing while allowing crawl for quality signals |
| API endpoints | robots.txt Disallow | No value to crawl, conserve crawl budget |
| Paginated pages (page 2, 3...) | Allow + canonical to page 1 | Let Google understand pagination, consolidate signals to page 1 |
| Faceted navigation (filters) | robots.txt Disallow (large sites) | Prevent crawl budget waste on near-infinite URL combinations |
Critical mistake to avoid: Do NOT add Disallow: for a page that has a noindex meta tag. If Google is blocked from crawling the page, it cannot see the noindex directive — the page may still appear in the index based on external link signals. Use noindex without Disallow, or Disallow without noindex. Not both.
Crawl budget is the number of pages Googlebot will crawl on your site during a given period. For most small and medium sites, crawl budget is rarely a limiting factor. But for sites with 100,000+ pages, it becomes critical:
?sessionid=abc123 creates millions of near-identical URLs
/products?color=red&size=xl&sort=price
/page?print=true duplicates every article
?utm_source=email&utm_campaign=...
Block these with robots.txt patterns or configure them in Google Search Console's URL Parameters tool (though the latter tool is being deprecated). For tracking parameters specifically, configure canonical tags to strip them.
Always test robots.txt changes before deploying to production:
In Google Search Console, navigate to Settings → robots.txt. The built-in tester lets you check if specific URLs are blocked or allowed for different Googlebot types. It also shows syntax errors and highlights problematic directives. This is the most accurate way to verify Google's interpretation of your file.
Fetch your robots.txt file directly: curl https://yourdomain.com/robots.txt. Verify: (1) The file is accessible (200 response); (2) The correct Content-Type header (text/plain); (3) No accidental Disallow: / rule; (4) All Disallow paths start with /.
Your staging environment robots.txt should contain Disallow: / to prevent accidental indexing of staging content. Ensure you switch this before going live.
Staging robots.txt (block everything):
User-agent: *
Disallow: /❌ Blocking CSS and JavaScript files
Old guidance was to block /wp-content/ or static asset folders. Modern best practice is to allow Googlebot to crawl CSS and JS — Google needs them to render your pages. Blocking these hurts your technical SEO scores and can prevent Google from seeing your content as users do.
❌ Using robots.txt to hide sensitive data
robots.txt is publicly visible — you're announcing which paths you want hidden. It's not a security mechanism. For truly private data, use server-side authentication. Malicious bots specifically look at robots.txt to find hidden admin paths.
❌ Wrong precedence: Disallow before Allow
When Allow and Disallow rules conflict, Google uses the more specific (longer) path. Bing uses the first matching rule. If you have both rules for the same path length, this can produce unexpected results. Test both with their respective webmaster tools.
❌ Using robots.txt to prevent indexing
Disallowing a page doesn't remove it from Google's index if it has external links pointing to it. Google can index a URL without crawling it based on anchor text from other sites. For true de-indexing, use noindex meta tag or request removal via Google Search Console.
❌ Forgetting the trailing slash on directories
Disallow: /admin only blocks the exact path /admin. Disallow: /admin/ blocks everything under /admin/. Use trailing slashes for directory exclusions, or Disallow: /admin with a wildcard like Disallow: /admin* to catch both.
| Platform | Where to Edit |
|---|---|
| WordPress | Yoast SEO → Tools → File Editor, or upload /robots.txt to public HTML root |
| Shopify | Online Store → Themes → Edit code → config/ → robots.txt.liquid (Shopify 2.0) |
| Squarespace | Settings → Advanced → External Services → Google Search Console (limited control) |
| Webflow | Project Settings → SEO → robots.txt text area |
| Wix | Marketing & SEO → Marketing Integrations → robots.txt (Wix manages defaults) |
| Next.js | Create /public/robots.txt or use next-sitemap to generate it, or app/robots.ts (App Router) |
| Netlify / Vercel | Place robots.txt in the /public folder of your project; it deploys to the root automatically |
| GitHub Pages / Hugo | Create static/robots.txt (Hugo) or put in root directory (GitHub Pages) |
Including your sitemap URL in robots.txt ensures any crawler that reads your robots.txt can also discover your sitemap automatically, without you needing to submit it to each search engine manually.
Add sitemap to robots.txt:
User-agent: *
Disallow: /admin/
Disallow: /login
Disallow: /api/
# Sitemap location (not User-agent specific)
Sitemap: https://yourdomain.com/sitemap.xmlIf you have a sitemap index file referencing multiple sitemaps, only list the index file URL. If you have multiple independent sitemaps, you can list multiple Sitemap directives:
Sitemap: https://yourdomain.com/sitemap-pages.xml
Sitemap: https://yourdomain.com/sitemap-products.xml
Sitemap: https://yourdomain.com/sitemap-blog.xmlAfter configuring your robots.txt, you need to verify that your important pages are still accessible to search engines and have the correct SEO signals. A page that's accidentally blocked or missing canonical tags won't get indexed regardless of how well it's written.
PageGuard scans individual pages and checks critical technical signals: whether the page returns 200, has correct canonical URLs, valid meta tags (title, description), structured data, and Core Web Vitals scores — everything Google evaluates when deciding whether to index and rank a page.
Verify your pages are crawlable and optimized
Enter any page URL to check its technical SEO health — canonical tags, meta tags, structured data, performance, and accessibility.
robots.txt is a plain text file at yourdomain.com/robots.txt that tells crawlers which paths to avoid. Crawlers read it before visiting any other page. It uses User-agent (which crawler), Disallow (blocked paths), and Allow (exceptions) directives. Following it is voluntary — Google, Bing, and reputable crawlers respect it; malicious bots may not.
robots.txt Disallow prevents crawlers from visiting the URL. noindex meta tag allows crawling but tells the crawler not to index the page. Critical: don't use Disallow to prevent indexing — Google can't read the noindex tag if it can't crawl the page. Use noindex for pages you want crawled but not indexed; use Disallow for pages that should never be crawled (admin, APIs).
Yes. robots.txt controls crawl budget allocation. For large sites (100K+ pages), blocking low-value pages (session IDs, filtered URLs, internal search) frees crawl budget for important content. For small sites, crawl budget is rarely a concern, but incorrectly blocking important pages in robots.txt is a significant SEO mistake that prevents Google from indexing your content.
Add a Sitemap directive with the full URL: 'Sitemap: https://yourdomain.com/sitemap.xml'. Place it at the end of your robots.txt. You can add multiple Sitemap lines for multiple sitemap files. The Sitemap directive is not User-agent specific — it applies to all crawlers regardless of which User-agent block it appears in (best practice: put it outside any User-agent block).
This is a business decision. Common AI training crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google AI training), CCBot (Common Crawl), Meta-ExternalAgent. Block them with 'User-agent: GPTBot' followed by 'Disallow: /'. Ethical AI companies respect these rules. Note: blocking AI search product crawlers (like Perplexity's search feature) may reduce AI-driven referral traffic.
Sitemap.xml Guide 2026
Create, format, and submit your sitemap to Google and Bing
Technical SEO Checklist 2026
Complete technical SEO audit covering crawlability, indexing, and Core Web Vitals
Google Search Console Guide 2026
Set up GSC, submit sitemaps, and diagnose indexing issues
On-Page SEO Checklist 2026
Optimize title tags, meta descriptions, headings, and structured data