Skip to content
FreeToolsHub

Robots.txt Generator

Build a valid robots.txt with rules and sitemap.

Loading tool…

Your text stays in your browser. It is not uploaded, saved or analysed by FreeTools Hub, and it disappears when you close or reload the page.

What is the Robots.txt Generator?

The Robots.txt Generator builds a correctly formatted robots.txt file with groups of rules for different crawlers, allow and disallow paths, an optional crawl delay and sitemap references.

It warns you about common mistakes, including accidentally blocking your entire site.

Why use it?

robots.txt tells well-behaved crawlers which parts of your site they may request. Keeping crawlers out of admin panels, internal search results and endless filter combinations saves server resources and helps search engines focus on useful pages.

How to use the Robots.txt Generator

  1. Edit the default rule group (User-agent: *) or add groups for specific crawlers.
  2. List paths to disallow, one per line, starting with /.
  3. Add allow rules for exceptions inside disallowed folders.
  4. Enter the full URL of your sitemap.
  5. Download robots.txt and upload it to the root of your domain.

Examples

Online shop

Disallow /cart/, /checkout/ and /search to stop crawlers wasting effort on pages with no search value.

Allow an exception

Disallow /private/ but Allow /private/press-kit.pdf to make one file crawlable.

Supported formats and inputs

  • Paths starting with / and optional * wildcards and $ end anchors, as supported by major search engines.
  • Sitemap lines as absolute URLs.

Limitations to know about

  • robots.txt is a request, not access control. Malicious bots ignore it, and the file itself is public.
  • Disallowing a page does not guarantee it is removed from search results; use a noindex meta tag on a crawlable page for that.
  • Google ignores Crawl-delay; set crawl rate preferences in Search Console instead.

Privacy

Your text stays in your browser. It is not uploaded, saved or analysed by FreeTools Hub, and it disappears when you close or reload the page.

Frequently asked questions

Where do I put robots.txt?

At the root of the host: https://www.example.com/robots.txt. A file in a subfolder is ignored.

What does an empty Disallow mean?

“Disallow:” with no path allows everything for that user-agent.

Can I block AI crawlers with robots.txt?

Many AI crawlers publish their user-agent names and say they respect robots.txt. Add a group for each user-agent with Disallow: /. Compliance is voluntary, so this is not a guarantee.

Sitemap XML Generator

Create an XML sitemap from a list of URLs.

Meta Tag Generator

Title, description, Open Graph and Twitter tags.

Slug Generator

Turn titles into clean, URL-friendly slugs.

URL Encoder/Decoder

Percent-encode or decode URLs and query parameters.

Browse all text & seo tools →