Back to SEO & Web Tools

Free Robots.txt Generator

Choose a preset and generate a valid robots.txt file for your website in seconds.

About This Tool

Free robots.txt generator online — choose to allow all crawlers, block all crawlers, or block specific paths, add a sitemap URL and an optional crawl-delay, and instantly get a valid, ready-to-upload robots.txt file. Useful for site owners and developers setting up a new site, blocking staging/admin areas from search engines, or double-checking that a hand-written robots.txt follows the correct syntax. Runs entirely in your browser — download the file directly, no server round-trip needed.

How to Use

  1. 1

    Choose a preset or customize

    Pick allow-all, block-all, or add specific disallowed paths.

  2. 2

    Add your sitemap URL

    Optionally include your sitemap.xml location and a crawl-delay.

  3. 3

    Copy or download

    Copy the robots.txt content or download it, then upload it to your site's root.

Frequently Asked Questions

Where does robots.txt need to be placed?+
It must be uploaded to the root of your domain, e.g. https://example.com/robots.txt — placing it in a subfolder means search engines won't find it.
Does robots.txt stop a page from appearing in Google?+
Not reliably — it only tells crawlers not to fetch a URL. A blocked page can still be indexed (without content) if other pages link to it. Use a noindex meta tag to reliably keep a page out of search results.
Should I include my sitemap in robots.txt?+
Yes — adding a Sitemap: line is a simple way to point every search engine crawler to your sitemap.xml file.

Related Tools

Sitemap Validator

Try it free →

Meta Tag Generator

Try it free →