Web Crawlers, Robots.txt, AI Scraper Directives & Search Indexing Architecture Runbook

curl -sI -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/robots.txt \
  && curl -s https://example.com/robots.txt | head -n 30

Validates that robots.txt returns an unambiguous HTTP 200 OK with text/plain Content-Type and no caching or redirect loops.

# Example matching order against URI: /photos/nature/mountain.jpg
# Rule 1: Disallow: /photos/
# Rule 2: Allow: /photos/nature/
# Result: ALLOWED (Length 15 > Length 8)

Demonstrates that the longer string pattern takes absolute precedence regardless of which line appears first in the file.

# Block Model Training Bots while permitting Search & Retrieval
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: CCBot
User-agent: Bytespider
User-agent: PerplexityBot
User-agent: Diffbot
User-agent: Applebot-Extended
Disallow: /

Explicitly excludes training crawlers from ingesting platform assets while leaving standard search engine indexers unaffected.

map $http_user_agent $is_ai_bot {
    default 0;
    ~*(GPTBot|CCBot|Bytespider|ClaudeBot|PerplexityBot) 1;
}

limit_req_zone $binary_remote_addr zone=ai_limit:10m rate=1r/s;

Network-level rate-limiting preventing rogue AI scrapers that ignore RFC 9309 from overwhelming origin database pools.

# Block indexing of PDF manuals, CAD files, and internal JSON endpoints
location ~* \.(pdf|doc|docx|json)$ {
    add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
}

Prevents non-HTML documents from appearing in search results where HTML meta tags cannot be rendered.

<head>
  <!-- Prevent indexing and pass zero link equity -->
  <meta name="robots" content="noindex, nofollow, noarchive, max-image-preview:none" />
</head>

Standard document-level directive instructing search bots to index nothing and discard cached snippet previews.

# Block faceted sorting and pagination sprawl while allowing canonical URLs
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*page=*
Disallow: /search?
Disallow: /calendar/*

Eliminates crawler loops by pruning search and filter URL parameters before requests hit the origin application server.

rg "Googlebot" /var/log/nginx/access.log \
  | awk '{print $7}' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -n 25

Audits top requested URIs by search bots to detect runaway parameter combinations consuming server resources.

# Signal temporary maintenance so bots return without dropping indexed URLs
error_page 503 @maintenance;
location @maintenance {
    add_header Retry-After "3600" always; # Retry in 1 hour
    return 503 "Service temporarily unavailable. Please retry shortly.";
}

Guarantees search bots pause crawling without de-indexing pages during scheduled maintenance or database migrations.

# Instruct crawlers to immediately remove defunct legacy URLs from index
location ~* ^/(legacy-v1|outdated-promo|deleted-catalog)/ {
    return 410 "Content permanently removed.";
}

Accelerates index purging compared to 404 Not Found, which search engines retry repeatedly for several weeks.