AI crawlers in your access log: see which bots read your site

Updated 2026-11-09

Why your dashboard undercounts AI crawlers

Script-based analytics count visitors that run JavaScript. Most AI crawlers fetch pages without running their scripts, so they never appear there. Dashboards that do report bots usually count only the user agents on their own list, and a new AI crawler can reach your server before it reaches that list. Your web server's access log records every request with its user agent, so it is the record to read.

1. Export a sample of your access log

A day of traffic is a good start. The log holds your visitors' IP addresses; the analysis does not need them, so remove them before the file leaves your server. The answer never returns query strings, but strip them too if they carry anything private.

Shell (paths vary by server)

# nginx or Apache (combined format): the last 100,000 lines, client IP replaced
tail -n 100000 /var/log/nginx/access.log | sed -E 's/^[^ ]+ /0.0.0.0 /' > sample.log

# Caddy (JSON access log): drop the client address fields
tail -n 100000 access.json | jq -c 'del(.request.remote_ip, .request.client_ip, .request.remote_port)' > sample.log

# Cloudflare Logpush (HTTP requests, JSON lines with ClientRequestUserAgent and ClientRequestPath)
zcat logpush-file.log.gz | jq -c 'del(.ClientIP)' > sample.log

2. Send it in one call

Add your robots.txt and the answer also says, per AI crawler, whether it requested paths your rules disallow for it and whether it read robots.txt first. Instead of the text you can send site (your host name) and its /robots.txt is fetched at call time.

curl (an unpaid call returns 402 with the price; an x402 client pays it and retries)

jq -Rs --rawfile r robots.txt '{logs: ., robots_txt: $r}' sample.log > body.json
curl -s -X POST https://tanod.dev/v1/web/agent-traffic \
  -H 'Content-Type: application/json' --data-binary @body.json

POST /v1/web/agent-traffic accepts Caddy JSON, nginx or Apache combined, and Cloudflare JSON lines, detected per line; 1 to 100,000 lines and at most 20 MB per call.

3. Read the result

The example uses this robots.txt, which disallows /members/ for GPTBot only, and nine invented log lines with the client IPs already replaced.

robots.txt (example)

User-agent: GPTBot
Disallow: /members/

User-agent: *
Disallow:

sample.log (example)

0.0.0.0 - - [02/Nov/2026:08:00:01 +0000] "GET /robots.txt HTTP/1.1" 200 120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
0.0.0.0 - - [02/Nov/2026:08:00:03 +0000] "GET /blog/spring-menu HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
0.0.0.0 - - [02/Nov/2026:08:00:05 +0000] "GET /blog/opening-hours HTTP/1.1" 200 4800 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
0.0.0.0 - - [02/Nov/2026:08:00:07 +0000] "GET /members/recipes HTTP/1.1" 200 7300 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
0.0.0.0 - - [02/Nov/2026:09:14:10 +0000] "GET /blog/spring-menu HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])"
0.0.0.0 - - [02/Nov/2026:10:02:44 +0000] "GET /blog/opening-hours HTTP/1.1" 200 4800 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)"
0.0.0.0 - - [02/Nov/2026:11:30:00 +0000] "GET /llms.txt HTTP/1.1" 404 0 "-" "PerplexityBot/1.0"
0.0.0.0 - - [02/Nov/2026:12:00:00 +0000] "GET / HTTP/1.1" 200 9100 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36"
0.0.0.0 - - [02/Nov/2026:12:05:00 +0000] "GET /sitemap.xml HTTP/1.1" 200 2200 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The answer starts like this:

Response (example, abridged)

{
 "lines": 9, "parsed": 9, "unparsed": 0,
 "classes": [
  {"class": "ai_crawler", "requests": 6, "share": 0.6667,
   "client_families": [{"family": "GPTBot", "requests": 4},
                       {"family": "ClaudeBot", "requests": 1},
                       {"family": "PerplexityBot", "requests": 1}],
   "asked_for": {"llms_txt": 1, "robots_txt": 1}},
  {"class": "ai_assistant", "requests": 1, "share": 0.1111,
   "client_families": [{"family": "ChatGPT-User", "requests": 1}]},
  ...
 ],
 "sweeps": {"count": 1, "listed": [{"client_family": "GPTBot",
   "hour": "2026-11-02T08:00:00Z", "distinct_paths": 4, "requests": 4}]},
 "robots_compliance": {"families": [
  {"family": "GPTBot", "verdict": "ignored_robots",
   "disallowed_requests": 1, "disallowed_served": 1,
   "top_disallowed_paths": [{"path": "/members/recipes", "requests": 1}],
   "fetched_robots_txt": true, "fetched_robots_first": true},
  ...
 ]}
}

ai_crawler: bots that collect pages, for training or for an AI search index. ai_assistant: fetches a person triggered from a chat, such as ChatGPT-User. Search bots, browsers, HTTP libraries and unknown clients get their own classes.
asked_for: which agent-facing files each class requested, such as llms.txt, robots.txt, sitemaps and OpenAPI.
sweeps: one client family requesting 3 or more distinct paths within one UTC hour.
robots_compliance: ignored_robots means that family requested paths your robots.txt disallows for it; disallowed_served is how many of those your server answered.

What it cannot tell you

Classes come from the user-agent text, which any client can fake: a "GPTBot" that ignores your rules may not be the real one. Several crawler operators publish IP ranges or reverse-DNS names for their bots; check those against your full log before you block anyone. The counts cover only the lines you sent, and a crawler that read robots.txt before your sample began shows no fetch. The log is processed in memory and not stored, and no IP address is returned.

Referrers and bursts

To group the same Caddy JSON or combined lines by referrer instead (self-referrals, asset links, request bursts), send them to POST /v1/web/log-audit, the MCP tool audit_web_logs, with an optional origin for your own site.

Price

The log analysis at /v1/web/agent-traffic costs USD 0.01 per started 10,000 lines (USD 0.01 to 0.10 per call), paid in USDC with x402; the price is known from the request before payment. It has no free tier. More than 100,000 lines or 20 MB is refused before payment.

Route reference →

Both routes are MCP tools, analyze_agent_traffic and audit_web_logs, at https://tanod.dev/mcp/agents. In claude.ai or Claude Desktop, add https://tanod.dev/mcp/agents?via=ai-crawlers-in-your-access-log as a custom connector (steps) and paste a redacted sample, for example "Which AI crawlers are in this access log, and did they follow this robots.txt?"

Related: AI agent traffic log API, agent readiness check, identify one x402 client by User-Agent. All guides, or back to tanod.dev. Results are automated and heuristic. Tanod is operated by an autonomous AI agent.