Robots.txt Tester
Client-sideNewTest whether a path is allowed by robots.txt for a user-agent — local Allow/Disallow matching.
Robots.txt Tester checks whether a path is ALLOWED or BLOCKED by a robots.txt for a chosen user-agent — matching runs locally in your browser on Toolverse by RS.
Examples
Key takeaways
- Paste robots.txt + path (or full URL) + optional user-agent
- Reports ALLOWED/BLOCKED with the winning rule and specificity
- Supports * and $ wildcards in Allow/Disallow patterns
- No network fetch — your staging robots.txt stays private
What does a robots.txt tester do?
It evaluates Allow/Disallow rules against a URL path and user-agent so you can see which rule wins before you deploy.
This tool parses User-agent groups, supports * and $ wildcards, and picks the most specific matching rule (with Allow winning ties at equal specificity — a Google-compatible subset). It does not fetch live robots.txt files; paste the file contents to keep testing private and offline-friendly.
Features
Group-aware matching
Picks the best User-agent group (* or a more specific bot name).
Specificity report
Shows which Allow/Disallow won and why.
Wildcard support
Handles * and $ patterns used in modern robots.txt files.
Local paste workflow
No fetch — test drafts before they are public.
How it works
Paste robots.txt
Include the User-agent groups you care about.
Enter a path or URL
Full URLs are normalized to pathname + search.
Test
Read ALLOWED/BLOCKED plus the matched rule.
Example walkthroughs
Broad disallow with exception
More specific Allow wins — result ALLOWED.
User-agent: * Disallow: /private Allow: /private/public Path: /private/public/page
Blocked admin path
URL normalizes to /admin/users — result BLOCKED.
User-agent: * Disallow: /admin Path: https://example.com/admin/users
Use cases
Pre-deploy robots review
Verify staging rules will not block indexable marketing URLs.
Debug sudden deindexing
Check whether a new Disallow explains a drop in coverage.
Code snippets
Typical production robots.txt
txtPrefer allow-by-default; block only what must stay private.
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Common errors
Everything is blocked
Cause: Disallow: / with no Allow exceptions.
Fix: Remove the blanket disallow on production or add specific Allow rules.
Rule seems ignored
Cause: Wrong User-agent group or a more specific competing rule.
Fix: Check the matched group in the report and compare specificity scores.
About Robots.txt Tester
Robots.txt Tester is a private Allow/Disallow workspace for SEO engineers and developers. Paste a robots.txt, enter a path or full URL, optionally set a user-agent (Googlebot, * , etc.), and read a clear ALLOWED or BLOCKED result with the matched group and rule. Matching stays in your browser — useful when the file is not public yet.
Most “robots.txt checker” searches happen after a scare: a staging Disallow: / that leaked to production, or a too-broad /api rule that blocked a marketing page. Longest-match precedence is easy to get wrong by eye when Allow and Disallow compete. This tool surfaces the winner and its specificity so you can adjust before crawl budget is wasted.
The matcher implements a practical Google-compatible subset: User-agent groups, Allow/Disallow, * wildcards, and end anchors with $. Sitemap and Crawl-delay lines are ignored for the allow decision (they do not change Allow/Disallow). Empty Disallow means allow all — a classic footgun this report calls out.
It will not replace Search Console’s robots tester for live verification, and it will not claim to simulate every bot’s quirks. Use it to reason about rule conflicts quickly, then confirm in GSC after deploy. Pair with Meta Tag Generator when you unblock a page and need fresh head tags.
Best practice: keep production robots.txt minimal, never Disallow: / on prod, test both Googlebot and * groups, and prefer Allow exceptions only when a parent Disallow is required.
Frequently asked questions
Does this fetch my live robots.txt?
No. Paste the file contents. Testing stays in your browser on Toolverse by RS — ideal for unpublished drafts.
Which matching rules do you follow?
A Google-compatible subset: most specific Allow/Disallow wins; at equal specificity Allow beats Disallow; * and $ wildcards are supported.
What about Sitemap lines?
They are ignored for the allow/block decision. They do not change whether a path is crawlable.
Can I test Googlebot specifically?
Yes. Set the user-agent field to Googlebot (or another bot). The longest matching agent token in a group wins over *.
Related tools
Meta Tag Generator
Generate title, description, canonical, Open Graph, and Twitter meta tags as paste-ready HTML.
SERP Preview
Preview title, URL, and meta description as a search snippet with character budgets — private and local.
Open Graph Debugger
Inspect Open Graph and Twitter Card tags from pasted HTML — find missing fields locally, no URL fetch.
Popular tools
JSON Formatter
Format, beautify, and minify JSON in your browser. Free, private, and instant — no upload to a server.
JSON Validator
Validate JSON instantly in your browser. See type, size, and node counts — or exact parse errors with line and column.
JSON Viewer
Explore JSON as an interactive tree with paths, types, and expandable nodes — all processed locally in your browser.
Base64 Encode / Decode
Encode UTF-8 text to Base64 or decode Base64 back to text — instantly in your browser.
Newest tools
Case Converter
Convert text between UPPERCASE, lowercase, Title Case, camelCase, snake_case, and more.
Word Counter
Count words, characters, sentences, and paragraphs with reading time — live and private.
Text Diff
Compare two texts line by line — see added and removed lines with a unified-style diff.
Slug Generator
Turn titles into clean, SEO-friendly URL slugs — lowercase, hyphenated, accents stripped.
Explore more
Guides & articles
SEO checklist before launch: meta, robots, SERP, OG
Generate meta tags, test robots.txt, preview SERP snippets, and debug Open Graph HTML locally before a page goes public.
Robots.txt and crawl control notes on the RS Roshi blog
Practical write-ups around robots.txt and crawl control.
Platform documentation
Shortcuts, limits, and how client-side tools work.