IT Glossary
What is robots.txt?
A plain text file at the root of your site telling search engine robots which areas they may crawl and which they must leave alone.
Before a search robot reads a single page of your website, it asks for one file: robots.txt, always at the same address — yourdomain.com/robots.txt. Think of it as house rules posted at the entrance for machines: which directories crawlers may enter, which they should skip (admin screens, internal search results, the shopping cart, endless filter combinations) and where the sitemap lives. It is a few lines of text with wildly disproportionate power. One wrong directive — the notorious ‘Disallow: /’ left switched on after a launch — blocks every robot from the entire site, and the business quietly disappears from search while the pages still load perfectly for human visitors. That failure mode is common enough to deserve a routine: configure the file once, correctly, then re-check it after every relaunch, redesign or migration, because staging environments ship with the blocking rule by default and somebody has to remember to remove it. One further distinction is worth carrying into any conversation with a developer or agency: robots.txt governs crawling, not secrecy and not indexing. The file itself is public — anyone can read yours — so it is the wrong place to hide sensitive folders, and a blocked address can still surface in results when other sites link to it. Keeping a page out of the index is a different job, done with a noindex tag or a password.
Let’s talk about your project
Message us on WhatsApp or send an email — you talk directly to a developer.
office@northdan.com · +40 752 070 247
Why it matters for your business
Crawl budget spent on pages that sell
Blocking zero-value areas — internal search, filter permutations, cart steps — points a search engine’s limited crawling effort at your products and money pages.
Technical corners stay out of results
Admin panels, test builds and internal scripts sit off the robots’ route, so the version of your site that reaches Google is the one you meant to show.
A one-minute safety net at every launch
Checking the file after a go-live or migration prevents the classic disaster: a new site shipped with crawling blocked and organic traffic collapsing for no visible reason.
Frequently asked questions
How do I check my own robots.txt file?
Type your domain into a browser followed by /robots.txt — the file is public by design, on your site and on every competitor’s. The alarm signal is the line Disallow: / sitting under User-agent: *, which shuts out every crawler at once. Google Search Console carries a dedicated report showing how Google actually reads the file and which addresses it currently blocks, and that report is the version that matters when the two disagree.
If I block a page in robots.txt, does it disappear from Google?
Not necessarily. The file stops crawling, not indexing: when other pages link to the blocked address, Google may still list it, usually with no description because it was never allowed to read the content. Genuine removal from the index uses a noindex tag or password protection — and noindex only works when the page is not blocked here, since a robot that cannot fetch the page never sees the tag telling it to stay away.
What belongs in the robots.txt of an online store?
Typically open access to product and category pages, blocks for cart, checkout, account areas, internal search results and the filter combinations that spawn infinite addresses, plus a Sitemap line pointing at your XML map. The exact rules depend on the platform — most ship sensible defaults that need adjusting rather than rewriting — and the filter question deserves a genuine look, because faceted navigation is where wasted crawling concentrates.
Let’s talk about your project
Message us on WhatsApp or send an email — you talk directly to a developer.
office@northdan.com · +40 752 070 247