Robots.txt Tester
100% PrivateSEO ToolsCheck whether a specific crawler is allowed to fetch a specific path under a robots.txt file.
Runs entirely in your browser. Your input is not uploaded to Ovatly or anywhere else.
Allowed for Googlebot
No Allow or Disallow rule in the Googlebot group matches /admin/public/help, so the path is allowed by default.
Matching follows RFC 9309: the group naming the crawler wins over the * group, the longest matching pattern wins, and Allow wins a tie.
- 1Paste the robots.txt content (a sample file is preloaded).
- 2Type the crawler name, for example Googlebot or Bingbot.
- 3Type the path or full URL you want to check.
- 4Read the verdict and the explanation of the matching rule.
With the preloaded sample file, testing "/private/data" as Googlebot returns Blocked, decided by "Disallow: /private/" in Googlebot's own group. Testing "/admin/settings" as Googlebot returns Allowed, because when a crawler has its own group, the * group's rules no longer apply to it.
- Matching follows RFC 9309: the group naming the crawler wins over the * group, the longest matching pattern wins, Allow wins a tie, and an empty pattern matches nothing.
- Paths are matched exactly as written, including case. Percent-encoded paths are compared in their encoded form, as the standard specifies.
- The tester evaluates one file and one crawler at a time; it does not fetch your live robots.txt.
robots.txt blocks fetching, not indexing. If other pages link to it, Google can list the URL without crawling its content. This tester tells you about crawling only.
About this tool
The Robots.txt Tester checks whether a specific crawler is allowed to fetch a specific path under a given robots.txt file, and shows exactly which rule decided it. It implements the RFC 9309 matching rules, including wildcards, end anchors, crawler-specific groups and longest-match wins.
Related tools
Part of Ovatly's free tools in SEO Tools.