🤖 Robots.txt Generator
Create a robots.txt file to control search engine crawling
User-agent Rules
How to Use the Robots.txt Generator
Add rules to control which search engine bots can access which parts of your site. Select a user-agent, choose Allow or Disallow, and enter the path. When done, click Generate and copy the output to a file named robots.txt in your website root directory.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot access. It must be placed in the root directory of your website.
Is robots.txt a security tool?
No. Robots.txt is a directive, not a security measure. Malicious bots may ignore it. Never use robots.txt to hide sensitive pages — use proper authentication instead.
Where should I put my robots.txt file?
Place it at the root of your domain: https://example.com/robots.txt. It will not work in subdirectories.