If you run a website in Singapore, understanding robots.txt is one of the quickest wins you can get for your technical SEO. It takes five minutes to set up, costs nothing, and gives you direct control over how Googlebot and other crawlers interact with your pages. Yet I still audit sites every week where this tiny file is either missing, misconfigured, or actively sabotaging their rankings.
This guide goes beyond the basics. I’ll show you exactly how robots.txt works, walk you through real syntax examples, and flag the mistakes I see most often on Singapore business websites. Whether you’re running an e-commerce store or a professional services site, you’ll walk away knowing how to write, test, and maintain this file yourself.
What Exactly Is a Robots.txt File?
A robots.txt file is a plain text file that sits in the root directory of your website. Its sole purpose is to communicate with search engine crawlers, telling them which parts of your site they’re allowed to access and which parts they should skip.
When Googlebot (or Bingbot, or any well-behaved crawler) arrives at your domain, the very first thing it does is request yoursite.com/robots.txt. It reads the instructions in that file before it crawls a single page. Think of it like the “Authorised Personnel Only” sign at the back of a hawker centre kitchen. It doesn’t physically stop anyone from walking in, but legitimate visitors will respect it.
The file uses a protocol called the Robots Exclusion Protocol, which has been a web standard since 1994. It’s not a programming language. There are no loops, no variables, no functions. Just a handful of directives that are surprisingly powerful when used correctly.
Why Robots.txt Matters for Your SEO
You might think, “I want Google to see everything on my site. Why would I block anything?” That’s a fair question. Here’s why this file deserves your attention.
Crawl Budget Management
Google allocates a finite crawl budget to every website. For a small 50-page site, this rarely matters. But if you’re running a Singapore e-commerce store with 10,000 product pages, filtered category pages, and session-based URLs, crawlers can waste enormous time on pages that add zero SEO value. I’ve seen sites where Googlebot spent 60% of its crawl budget on faceted navigation pages that were never meant to rank.
By using robots.txt to block these low-value paths, you redirect crawl activity toward your money pages. On one client’s site, we blocked four unnecessary URL patterns and saw their new product pages getting indexed 3 days faster on average.
Preventing Duplicate Content Signals
Many CMS platforms generate duplicate or near-duplicate pages automatically. Print-friendly versions, URL parameters from internal search, paginated archives. If crawlers index these, you end up competing against yourself in search results. Robots.txt lets you cut off these paths before crawlers even discover them.
Protecting Staging and Admin Areas
Your /wp-admin/, /staging/, or /dev/ directories have no business appearing in Google’s index. Neither do thank-you pages, cart pages, or internal dashboards. Blocking these keeps your search presence clean and professional.
Controlling Third-Party Bot Access
It’s not just Google visiting your site. AI training bots, SEO tool crawlers, and aggressive scrapers all consume your server resources. A well-configured robots.txt file lets you selectively allow or deny access to specific bots by name.
Robots.txt Syntax: The Complete Breakdown
The file uses only a few directives, but the way you combine them matters. Let me walk you through each one.
User-agent
This specifies which crawler the following rules apply to. Use an asterisk to target all bots, or name a specific one.
User-agent: *This targets every crawler. If you want to write rules specifically for Google, you’d use:
User-agent: GooglebotYou can stack multiple rule sets for different bots in the same file. Googlebot will only follow the rules under its own User-agent block. If no specific block exists for it, it falls back to the wildcard (*) rules.
Disallow
This tells the specified bot not to crawl a particular path.
Disallow: /admin/
Disallow: /cart/
Disallow: /internal-search/An empty Disallow directive means “allow everything”:
User-agent: *
Disallow:Allow
This is where things get more nuanced. You can use Allow to create exceptions within a broader Disallow rule. For example, if you want to block an entire directory but keep one specific page accessible:
User-agent: *
Disallow: /resources/
Allow: /resources/seo-checklist/This tells crawlers to skip everything in /resources/ except the SEO checklist page. The Allow directive takes precedence over Disallow when both match a URL, as long as the Allow path is more specific.
Sitemap
You can (and should) reference your XML sitemap directly in your robots.txt file:
Sitemap: https://www.yoursite.com/sitemap.xmlThis helps crawlers discover your sitemap even if it’s not submitted through Google Search Console. It’s a small detail, but it ensures every crawler that reads your robots.txt also knows where your full site map lives.
Wildcard Patterns
Googlebot supports two wildcard characters that aren’t part of the original standard but are widely respected:
*matches any sequence of characters.Disallow: /*?sessionidblocks any URL containing the parameter “sessionid”.$anchors the match to the end of the URL.Disallow: /*.pdf$blocks all PDF files but won’t accidentally block a URL like/pdf-guide/.
These wildcards are incredibly useful for Singapore e-commerce sites that generate hundreds of parameterised URLs from product filters, sorting options, and currency selectors.
A Real-World Robots.txt Example for a Singapore Business Site
Here’s a robots.txt file I’d consider reasonable for a mid-sized Singapore business website running WordPress:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?s=
Disallow: /*?add-to-cart=
Disallow: /tag/
Disallow: /author/
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://www.yoursite.com/sitemap_index.xmlLet me explain the thinking behind each block.
The /wp-admin/ block is standard for WordPress, but you need the Allow exception for admin-ajax.php because many themes and plugins use that file to load content on the front end. Block it, and you might break functionality that Googlebot needs to render your pages properly.
The ?s= pattern blocks internal search result pages. These are thin content pages that dilute your crawl budget. The ?add-to-cart= pattern blocks URLs generated when someone clicks “Add to Cart” on WooCommerce sites.
The GPTBot and CCBot blocks prevent AI training crawlers from scraping your content. This is increasingly relevant for Singapore businesses that invest heavily in original content and don’t want it feeding large language models without consent.
Common Robots.txt Mistakes I See on Singapore Websites
Accidentally Blocking Your Entire Site
This happens more often than you’d think. A developer adds Disallow: / during staging and forgets to remove it at launch. I audited a Singapore F&B chain’s website last year that had been live for four months with zero organic traffic. The culprit was a single line in robots.txt. Four months of content, four months of link building, all invisible to Google.
Always check your robots.txt immediately after any site migration or redesign.
Treating Robots.txt as a Security Tool
Robots.txt does not hide content from humans. Anyone can type yoursite.com/robots.txt into their browser and read it. In fact, malicious actors sometimes use robots.txt files as a roadmap to find sensitive directories you’ve tried to hide. If you need to protect a page, use server-side authentication or password protection. Not robots.txt.
Blocking CSS and JavaScript Files
Years ago, it was common practice to block /wp-content/ or /wp-includes/ directories. Don’t do this. Google needs access to your CSS and JavaScript files to render your pages the way a real user sees them. Blocking these files can cause Googlebot to see a broken, unstyled version of your site, which directly hurts your rankings.
Confusing Disallow with Noindex
A Disallow directive stops crawling, not indexing. If other websites link to a page you’ve blocked in robots.txt, Google may still index the URL. It will show up in search results with a bare title and no description snippet. If you want a page completely removed from search results, you need a noindex meta tag or an X-Robots-Tag HTTP header. You can’t use both robots.txt blocking and noindex on the same page, because Googlebot needs to crawl the page to see the noindex tag.
How to Test and Validate Your Robots.txt File
Never push a robots.txt change to production without testing it first. Here’s my process.
Step 1: Use Google Search Console’s URL Inspection Tool
Go to Google Search Console, enter a URL you’ve blocked, and check whether Google reports it as blocked by robots.txt. This is the most authoritative test because it uses Google’s actual parsing logic.
Step 2: Check the Robots.txt Report in Search Console
Navigate to Settings > Crawling > robots.txt in Search Console. You’ll see the last cached version of your file and any errors Google detected. If Google can’t fetch your robots.txt (returns a 5xx error), it will temporarily stop crawling your entire site as a precaution. Make sure your server reliably serves this file.
Step 3: Test with Screaming Frog or Sitebulb
Run a crawl of your site with your SEO crawler of choice. Check which URLs are being blocked by robots.txt and verify they match your intentions. I’ve caught several cases where wildcard patterns were accidentally blocking high-value product pages.
Step 4: Monitor Crawl Stats
After making changes, monitor your crawl stats in Search Console for 2 to 4 weeks. You should see crawl requests shift away from blocked paths and toward your priority pages. If total crawl requests drop significantly, something may be misconfigured.
Robots.txt vs. Meta Robots vs. X-Robots-Tag: When to Use Each
This is where many site owners get confused. Here’s a quick comparison.
Robots.txt controls crawling at the directory or URL-pattern level. Use it when you want to save crawl budget or prevent discovery of entire sections.
The meta robots tag (placed in your page’s HTML head) controls indexing at the individual page level. Use it when you want a page crawled but not indexed, or when you want to prevent link equity from flowing through a page’s outbound links.
The X-Robots-Tag HTTP header does the same thing as the meta robots tag but works for non-HTML files like PDFs, images, and videos. Use it when you need to noindex a file type that doesn’t have an HTML head section.
For most Singapore business websites, you’ll use a combination of all three. Robots.txt handles the broad strokes. Meta robots and X-Robots-Tag handle the fine details.
Keeping Your Robots.txt File Maintained
Your robots.txt file isn’t a “set and forget” asset. Here’s a maintenance checklist I recommend reviewing quarterly:
- Verify the file is accessible at your root domain (check both www and non-www versions).
- Confirm no critical pages are accidentally blocked after any CMS update or plugin change.
- Review crawl stats in Search Console for unusual patterns.
- Update rules if you’ve added new site sections, changed URL structures, or migrated platforms.
- Check that your sitemap reference URL is still valid.
I add robots.txt review to every quarterly SEO audit we run. It takes five minutes and has saved clients from some very expensive mistakes.
Take Control of Your Crawl
A properly configured robots.txt file won’t single-handedly rocket you to page one. But a misconfigured one can absolutely keep you off it. This is foundational technical SEO, the kind of work that separates sites that rank consistently from sites that struggle despite having great content.
If you’ve read this far and realised your robots.txt might need attention, start by pulling it up in your browser right now. Type your domain followed by /robots.txt and see what’s there. You might be surprised.
If what you find looks confusing, or if you’re planning a site migration and want to make sure nothing breaks, we’re happy to take a look. You can reach out to us here and we’ll review your robots.txt file as part of a no-obligation technical SEO check.
Frequently Asked Questions About Robots.txt
Does robots.txt stop Google from indexing my pages?
No. It stops Google from crawling them. If other sites link to a blocked page, Google can still index the URL without its content. To prevent indexing, you need a noindex meta tag on the page itself, which means the page must remain crawlable.
Can I use robots.txt to block specific AI crawlers?
Yes. Many AI companies have published their crawler names. For example, you can block OpenAI’s GPTBot or Anthropic’s ClaudeBot by adding a User-agent block with a Disallow directive. Whether these bots fully respect the protocol is another question, but major AI companies have publicly committed to honouring robots.txt.
What happens if my robots.txt file has a server error?
If your server returns a 5xx error when a crawler requests robots.txt, Google will treat it as a temporary failure and may reduce or pause crawling of your site until the file becomes accessible again. This is why reliable hosting matters for technical SEO.
Should I block my WordPress tag and author archive pages?
In most cases, yes. Tag archives and author pages on small to mid-sized sites tend to create thin, duplicate content that wastes crawl budget. Blocking them in robots.txt is a quick fix, though combining this with noindex directives through your SEO plugin gives you more complete control.
How quickly does Google respond to robots.txt changes?
Google caches your robots.txt file and refreshes it roughly once every 24 hours, though it can take longer. After making a change, you can request a re-crawl through Google Search Console to speed things up. Don’t expect instant results. Give it a few days before checking whether the new rules have taken effect.