SEO Instrumenti

Technical SEOAvailable

Robots.txt Generator

Create a robots.txt file with current crawler presets, Allow and Disallow rules and sitemap locations, with an explanation of each directive.

Runs in your browserNo permanent storage by default

Workspace for Robots.txt Generator

Request scope

Complete only the required fields. The method and limits are explained below the workspace.

The token is written exactly into the User-agent directive. Crawler policies change; confirm them in the official documentation.

One URL path per line, up to 200.

One URL path per line, up to 200. This does not protect sensitive files.

Complete public URL of an XML sitemap.

About Crawl-delay

The generator does not add Crawl-delay. It is not part of the robots.txt standard and does not control Googlebot crawl rate.

The check starts only after you press the button.

About this check

What the tool checks

Prepare a readable robots.txt policy for search and AI crawlers without treating the file as access control or a way to remove URLs from an index.

Related tools
Robots.txt and Indexability Tester

Check robots.txt, meta robots, X-Robots-Tag, HTTP status and canonical signals together for a specific public URL.

 XML Sitemap Generator

Crawl a limited sample of public same-origin pages and create a valid XML sitemap without inventing lastmod dates or promising indexing.

 AI Crawler Access Checker

Check robots.txt policies for training and search crawlers, product-control tokens and user-triggered AI fetchers with their roles kept distinct.

 JavaScript SEO Dependency Audit

Inspect which important SEO signals exist in server HTML and identify structures that are likely to depend on JavaScript, without an external browser service.