How to Create a robots.txt File (With Copy-Paste Examples)
September 26, 2026
To create a robots.txt file, write plain-text rules in a file named exactly robots.txt and upload it to the root of your domain, so it loads at https://yourdomain.com/robots.txt. Each group of rules starts with a User-agent line naming a crawler (or * for all crawlers), followed by Disallow and Allow lines listing URL paths, and you can add a Sitemap line with the full URL of your XML sitemap. robots.txt controls crawling, not indexing, so it is not a way to hide pages or keep them out of search results.
What is a robots.txt file and what does it do?
robots.txt is a plain-text file that tells automated crawlers which parts of a site they may request. It follows the Robots Exclusion Protocol, standardized by the IETF as RFC 9309 in 2022. Well-behaved crawlers such as Googlebot and Bingbot read it before crawling and follow its rules; badly behaved bots can simply ignore it, so it is a request, not a security control. Its main uses are keeping crawlers out of low-value areas (internal search results, cart and filter URLs, staging folders), reducing unnecessary server load, and pointing crawlers to your sitemap.
Where do you put the robots.txt file?
In the top-level directory of the host it applies to, named in lowercase: https://example.com/robots.txt. A file at https://example.com/blog/robots.txt is ignored. Every host needs its own file — shop.example.com and example.com are read separately, and so are the http and https versions. Save it as UTF-8 plain text; Google only processes the first 500 KiB of the file, which is far more than a normal site needs. If your site runs on a CMS or framework, check whether it already generates a robots.txt before uploading your own.
Which robots.txt directives do you actually need?
Four directives cover almost every real-world file. Paths are case-sensitive and are matched from the start of the URL path. Google and Bing also support two wildcards: * matches any sequence of characters and $ marks the end of the URL. To skip writing it by hand, the free robots.txt generator builds a valid file from the options you pick.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Starts a group and names the crawler the group applies to; * means all crawlers | User-agent: Googlebot |
| Disallow | Path the crawler should not request; an empty value allows everything | Disallow: /admin/ |
| Allow | Re-opens a path inside a disallowed folder | Allow: /admin/help/ |
| Sitemap | Full URL of an XML sitemap; can appear anywhere and more than once | Sitemap: https://example.com/sitemap.xml |
How do you allow all crawlers to access everything?
Use a single group for all user agents with an empty Disallow, and add your sitemap. This is the right default for most small sites — it is equivalent to having no restrictions, but it makes your intent explicit and tells crawlers where the sitemap is.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlHow do you block a folder or a type of URL?
Add a Disallow line for each path you want crawlers to skip. A trailing slash blocks the folder and everything inside it; the * and $ wildcards let you block URL patterns such as parameters or file types. An Allow line can re-open one path inside a blocked folder — when rules conflict, Google applies the most specific (longest) matching rule.
User-agent: *
Disallow: /admin/
Disallow: /search
Disallow: /*?sort=
Disallow: /*.pdf$
Allow: /admin/help/
Sitemap: https://example.com/sitemap.xmlHow do you block AI crawlers like GPTBot, ClaudeBot and Google-Extended?
Give each AI crawler its own group with Disallow: /. GPTBot (OpenAI) and ClaudeBot (Anthropic) are the crawlers those companies document for collecting training data; Google-Extended is not a separate crawler but a token that tells Google not to use your content for training and grounding its Gemini models, and it does not affect Google Search. CCBot is Common Crawl, whose archive is widely used for AI training. Search-and-answer features often use different user agents (for example OAI-SearchBot for ChatGPT search), so blocking a training crawler does not necessarily remove you from AI search answers — check each provider's current documentation, because these names change.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlWhat are the most common robots.txt mistakes?
Most robots.txt problems come from a handful of errors that are easy to spot once you know them.
| Mistake | What happens | Fix |
|---|---|---|
| Disallow: / left over from a staging site | The whole site stops being crawled | Remove it or change it to an empty Disallow before launch |
| Using robots.txt to hide a page from Google | The URL can still be indexed without its content if other pages link to it | Allow crawling and add a noindex meta tag, or protect the page with a password |
| Putting noindex or crawl-delay in robots.txt for Google | Google ignores both (Bing does honor crawl-delay) | Use a noindex meta tag or X-Robots-Tag header instead |
| Blocking CSS and JavaScript folders | Google cannot render the page properly, which can hurt how it is understood | Allow the assets your pages need to render |
| Wrong location or file name | The file is ignored | Place robots.txt (lowercase) at the root of each host |
| Wrong capitalization in paths | Disallow: /Admin/ does not block /admin/ | Match paths exactly as they appear in URLs |
How do you test a robots.txt file?
First open https://yourdomain.com/robots.txt in a browser and confirm it loads with a 200 status and shows the rules you expect. Then, in Google Search Console, open the robots.txt report (under Settings) to see which version Google fetched, when, and whether it found errors; use the URL Inspection tool on a specific page to check whether it is blocked. Bing Webmaster Tools has an equivalent robots.txt tester. Google usually caches robots.txt for up to a day, so changes are not picked up instantly. Finally, spot-check a few important URLs against your Disallow patterns — a single wildcard in the wrong place can block far more than intended.
Does a robots.txt file improve SEO?
Not directly: no rule in robots.txt makes a page rank higher. It helps indirectly by keeping crawlers focused on the pages that matter on large sites, and by listing your sitemap. For a small site, the most important thing is simply not to block anything by accident. If a page must stay out of search results, use noindex rather than robots.txt, because Google has to be able to crawl a page to see its noindex tag.