Your Essential Guide: How to Create Robots.txt File

Only 37 % of the top 10 000 websites had a robots.txt file in 2025 (Cloudflare). Without that text file, you risk wasting crawl budget on pages that don’t matter. Learning how to create a robots.txt file is your first step toward guiding search engines to your most valuable content.

Key takeaway: a simple text file at your site’s root can direct crawlers, protect sensitive pages, and boost your SEO performance.

understand robots.txt basics

Before you jump into rules and uploads, it helps to know what a robots.txt file does and how it works.

what is robots.txt?

A robots.txt file is a plain-text document that lives in your site’s root directory (for example at example.com/robots.txt). It follows the Robots Exclusion Protocol, which most well-behaved crawlers check before they request pages. In short, the file tells bots which URLs they may crawl and which to skip.

crawl directives

Each rule group in robots.txt targets one or more user agents (bots). The basic directives are:

Directive	Description
User-agent	Specifies the bot (for example “Googlebot” or “*” for all bots)
Disallow	Prevents crawling of a path (for example /admin/)
Allow	Permits crawling of a path even if it falls under a broader block
Sitemap	Points to your XML sitemap to help crawlers find pages faster

Bots process these rules top to bottom, so order matters. If you list Disallow: / under User-agent: *, you block the entire site. A later Allow: /blog/ won’t override that block.

plan and write directives

A clear plan makes writing the file faster and safer. You’ll decide which content needs protection and which should shine in search results.

plan your crawl rules

Start by auditing your site structure. Identify low-value or duplicate pages such as:

• Staging or test environments
• Printer-friendly versions
• Tag or category archives
• Internal search results pages

Then list essential assets that bots need to render your pages properly (for example CSS or JavaScript files). Blocking these can hurt your SEO—Google recommends keeping critical resources accessible to ensure correct indexing.

create the file manually

If you prefer full control, a manual approach works well (you’ll learn how to create robots.txt file in just a few steps).

how to create robots.txt file

1. Open a plain-text editor that supports UTF-8 encoding.
2. Add your rule groups, for example:

   User-agent: *
   Disallow: /private/
   Allow: /public/
   Sitemap: https://example.com/sitemap.xml

3. Save the file as robots.txt.
4. Upload it to your site’s root directory (see robots.txt file location).
5. Check that the uploaded file matches your local copy.

For syntax reference, consult our guide on robots.txt file format. If you’d rather skip manual edits, you can try a simple robots.txt file generator to build your rules.

use CMS tools

Many content management systems let you edit robots.txt without FTP access.

WordPress

If you run WordPress, head to your SEO plugin’s file editor (for example in Yoast). You can view, edit, and save your robots.txt right from the Admin dashboard. For step-by-step instructions, visit robots.txt file wordpress.

other platforms

• Magento and Shopify often include a robots.txt editor under “Site settings” or “Preferences” (check your theme settings).
• If your CMS lacks a UI, look for an “advanced settings” or “file manager” option, or consider a plugin that adds robots.txt support.

test and deploy file

Writing rules is only half the battle, you also need to verify they work as intended before search engines catch crawlers ignoring important pages.

test and validate rules

Use tools to spot syntax errors and unintended blocks:

• Google Search Console’s robots.txt Tester (https://search.google.com/search-console) lets you fetch your file live and run URL checks.
• SEO platforms like SE Ranking (https://seranking.com) offer crawl simulations and highlight inaccessible assets.

A quick test can save hours of troubleshooting down the road.

avoid common mistakes

Keep an eye out for pitfalls that even seasoned pros slip into:

• Using robots.txt to hide sensitive data. (Bots can index a URL without crawling it if other sites link to it.)
• Blocking CSS or JavaScript needed for proper page rendering.
• Overly broad wildcards that catch more URLs than intended.
• Forgetting to remove old rules when you reorganize content.
• Relying on robots.txt alone to keep pages private (use password protection or a noindex meta tag).

maintain and update rules

Your site evolves. Your robots.txt file needs to keep pace.

schedule regular reviews

Aim to revisit your rules whenever you:

• Launch a major redesign or new section.
• Change your URL structure or permalink settings.
• Update your CMS version or switch platforms.

A quarterly check-in helps you catch broken links or outdated directives.

track changes and use generator

Store your robots.txt in version control if you can (for example in Git alongside your site code). That way you can roll back if something goes wrong. You may also lean on a robots.txt file generator when you need to build complex rule sets quickly.

Good news, keeping this file up to date is easier than it sounds once you have a process in place.

quick recap and next step

1. Understand the role and syntax of robots.txt.
2. Plan what to allow and what to block.
3. Create the file manually or via your CMS.
4. Test rules in Google Search Console or SE Ranking.
5. Review and update your directives regularly.

Pick one step today—maybe test your current file or schedule a review. You’ve got this, and a solid robots.txt file will help search engines focus on your best content.