robots.txt is a public plaintext file at yourdomain.com/robots.txt. Every path listed under Disallow is visible to any person, bot, or scanner that visits it. The file was designed to coordinate with compliant crawlers, not to protect content. Using it to hide sensitive paths is not a security control. It is a directory listing.

Most SEO teams add Disallow entries for legitimate crawl reasons. Suppress duplicate parameter URLs. Keep staging out of the index. Tell Googlebot to skip an internal search results page. The intent is fine. The side effect is that the file slowly accumulates a map of every directory the team would prefer attackers not find. Nobody on the security side ever reads it, because it is filed under "SEO". Nobody on the SEO side ever reads it from an attacker's point of view, because that is filed under "security". The blind spot sits between two job descriptions.

This post argues a single point: robots.txt is the cleanest example of what happens when SEO and security operate as separate disciplines on the same domain.

What robots.txt is, and why attackers read it first

robots.txt tells compliant crawlers which URLs to skip. That is the entire feature set. There is no authentication layer, no server-side enforcement, and no mechanism that prevents a browser, a scanner, or a person from visiting any listed path directly. Disallow is a polite request to well-behaved bots. It is not an access rule.

The file is governed by the Robots Exclusion Protocol, not by any HTTP access control specification. Disallow does not even prevent indexing reliably; if another site links to a "disallowed" URL, search engines can index the URL without crawling it. The file is served at a predictable, permanent location on every domain that uses it.

Which is exactly why attackers retrieve it first. PortSwigger documents the pattern explicitly: the file "is often used to identify restricted or private areas of a site's contents" and "may therefore help an attacker." Automated tooling like Nikto, OWASP ZAP, and Burp Suite pulls robots.txt during passive recon. A single GET request can reveal admin panels, backup paths, API routes, and the underlying CMS in one document. As Baeldung's analysis of attacker workflows notes, the file exposes "resources that cannot be reached just by repeatedly crawling through the hyperlinks." That is the entire value to an attacker. They do not need to crawl. You handed them the index.

Paths that actually leak: patterns from real audits

The disclosures we see most often fall into four categories: admin panels, backup directories, internal APIs, and staging environments. Every one of those paths was added by someone with good intent. The team wanted them out of search results. The team did not realise they were also adding them to a public reconnaissance document.

Real examples are easy to find without breaching anyone's confidentiality. The InfoSec community has catalogued the typical WordPress pattern on live sites: /wp-admin/, /wp-content/plugins/, and /archive/ listed under Disallow. The presence of /wp-admin/ confirms the CMS in one line. Black Duck's review describes the consequence: once the CMS is identified, an attacker can "focus their attack, enumerating specific version number vulnerabilities." Plugin paths multiply the surface, since outdated plugins are a primary WordPress attack vector. Red Secure Tech's analysis identifies /backup/, /config/, /private/, and /database/ as the directory names most commonly exposed in misconfigured files, which is exactly what you would expect. Those are the words a developer reaches for when naming sensitive folders.

In our own audits, the most painful finds are not the obvious admin paths. They are paths that look harmless until you visit them. Old export endpoints. Internal tools subdirectories named after a former product. Bulk download routes left over from a migration three years ago. The Disallow line was added once, copy-pasted into every subsequent robots.txt, and never re-evaluated.

The ownership gap nobody audits

SEO agencies add Disallow entries to manage crawl budget, suppress staging URLs, and stop parameter pollution. Security teams audit authentication headers, cookies, session handling, and APIs. Neither owns robots.txt from both angles. That gap is where the file accumulates risk over years of incremental edits.

The SEO rationale is genuinely valid in places. Internal search results (/search?q=), faceted navigation parameters, and pagination URLs belong in robots.txt because indexing them wastes crawl budget and creates duplicate-content noise. The problem starts when the same logic is applied to /admin/ or /staging/ without a parallel question: does this path have its own authentication layer? In most audits we run, the answer for staging is "HTTP basic auth was removed for a sprint demo and never re-added." The robots.txt entry is the only thing left.

Security audits, for their part, treat robots.txt as out of scope because it is "an SEO file." So the file gets updated quarterly by the marketing team and read carefully by no one. If you want a single artefact that captures the cost of siloed disciplines, it is the Disallow list of a five-year-old B2B site. Our technical SEO audit process treats robots.txt as a security artefact by default, because that is what it has become in practice.

What belongs in robots.txt, and what does not

robots.txt should contain crawl efficiency directives only. Parameter URLs. Internal search results. Faceted navigation. Pagination. Anything you list because someone wants it "hidden" should be protected at the server, not the crawler. MDN's guidance is unambiguous on this: robots.txt "should not be used as a way to prevent the disclosure of information."

Here is the split we apply in audits.

The single rule: authentication on the path itself must exist independently of whether the path appears in robots.txt. If removing the Disallow line would expose the path, the path was never protected. It was just quiet. Quiet is not a control. We make a similar argument about visibility versus protection in our writeup on on-page SEO and content optimisation, where the same "appearance of control" pattern shows up in canonical tags.

How to audit your robots.txt in under 20 minutes

You do not need a tool. You need a spreadsheet and a willingness to ask one question of every line.

1. Retrieve the file: curl https://yourdomain.com/robots.txt, or open it in a browser.

2. Copy every Disallow entry into a spreadsheet, one path per row.

3. Classify each path as either "crawl efficiency" (parameters, pagination, search) or "path concealment" (admin, backup, staging, internal tools).

4. For every concealment entry, implement server-level auth before changing anything else. htaccess authbasic, Nginx authrequest, or application middleware. Whatever your stack uses.

5. Once auth is verified, decide whether to keep the Disallow line for crawl reasons or remove it. The point is to stop relying on it for security.

One critical clarification: removing a path from robots.txt does not expose it. The path was already publicly visible at the URL. You are removing false security, not creating new risk. Re-submit your XML sitemap through Google Search Console after structural changes, and move on.

Frequently asked questions

Is robots.txt a security risk?

robots.txt is not a vulnerability by itself. The risk is using it as access control. PortSwigger classifies it as an information disclosure issue when sensitive paths appear in it. Because the file is public by design, every path you add becomes a publicly documented entry point for reconnaissance.

What should I not put in robots.txt?

Do not list admin panels, backup directories, staging environments, internal APIs, database paths, or any URL you want kept private. Disallow applies only to compliant crawlers. It has no effect on browser navigation, direct URL requests, or scanner enumeration. Authentication controls the path. robots.txt does not.

Can hackers use robots.txt?

Yes, and they do as a matter of routine. Automated reconnaissance tools retrieve robots.txt as a first step in web application assessments. The file gives attackers site-structure intelligence without active crawling, which both saves them time and reduces the noise their tooling generates against your logs.

The cross-discipline angle

robots.txt became a security file by accident, because teams used it to hide things it was never built to hide. The SEO rationale (suppress admin paths from indexing) is often defensible. The implementation (using robots.txt as the only control) is not. The structural problem is split ownership: SEO adds paths, security never reads them.

This is the class of issue a single-discipline agency misses every time, because it falls between two scopes. Most B2B sites we audit have at least one path in robots.txt that should not be there, and at least one path with no server-side protection beyond the Disallow line. If your robots.txt has been edited by your SEO team without a security review, or your security team has never looked at it, the cost of checking is twenty minutes. Book a free SEO audit call and we will go through it as part of the technical review. No slide decks, no 200-page PDFs. Specific findings, in writing.