Build a robots.txt file that tells search crawlers which paths they may fetch and where to find your sitemap.
Format
The generator writes one User-agent group with optional Allow and Disallow lines (one path per input line) and an optional Sitemap directive. Paths are normalized to start with / so /admin and admin behave the same.
When to use it
Launch a new site, block staging areas from indexing, or document crawler policy before uploading robots.txt to your domain root. Pair with a sitemap generator so bots discover your URL list.
Limitations
robots.txt is a voluntary hint ā malicious bots may ignore it. It does not authenticate users or replace server-side access control. Validate URLs manually; this tool does not ping search consoles.
Example
User-agent *, allow /, disallow /admin, and sitemap https://example.com/sitemap.xml produces a four-line policy file ready to deploy.