5 min read.htaccess / robots.txt Generator
robots.txt and .htaccess Basics
Generate a robots file with a sitemap URL, and Apache rules for HTTPS, cache, and security headers.
Using the free tool while you read? Open .htaccess / robots.txt Generator.
robots.txt tells crawlers where to go
Allow the public site, and disallow admin or API paths you do not want indexed. Add the full sitemap URL at the bottom so crawlers can find sitemap.xml.
robots.txt is not access control. A disallowed URL can still be opened by anyone who has the link.
Use .htaccess on Apache only
The HTTPS, www, gzip, cache, and security-header rules are for Apache. Nginx and many static hosts ignore .htaccess. Turn on only the rules you actually want before uploading the file.