AI agent traffic log API: find AI crawlers, MCP clients and x402 buyers in access logs
Updated 2026-10-16
Sort your access log by AI client
Which AI crawlers, assistants, MCP clients and x402 buyers hit your server?POST https://tanod.dev/v1/web/agent-traffic takes access-log lines and sorts the requests into client classes. It is for site owners who want to see agent traffic apart from browsers and search bots.
curl -s -X POST https://tanod.dev/v1/web/agent-traffic \
-H "Content-Type: application/json" -H "X-Tanod-Free: 1" \
-d '{"logs": ["{\"ts\":1792065600.1,\"status\":200,\"request\":{\"method\":\"GET\",\"uri\":\"/llms.txt\",\"headers\":{\"User-Agent\":[\"GPTBot/1.2\"]}}}", "203.0.113.9 - - [15/Oct/2026:12:01:02 +0000] \"GET /mcp HTTP/1.1\" 402 120 \"-\" \"mcp-remote/0.1\"", "203.0.113.9 - - [15/Oct/2026:12:01:03 +0000] \"GET /robots.txt HTTP/1.1\" 200 80 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) Firefox/130.0\""]}'This is a sample input of three invented lines. Send Caddy JSON, nginx or Apache combined, or Cloudflare JSON (Logpush with ClientRequestUserAgent and ClientRequestPath), as a list or one string with newlines. The format is detected per line. Send 1 to 100,000 lines and at most 20 MB.
What comes back
lines,parsed,unparsed,formats,requestsandtime_range.classes: AI crawlers, AI assistants and user-triggered fetchers, MCP clients, x402 buyer clients, listing monitors and rating bots, generic HTTP libraries, search bots, browsers and unknown. Each has itsrequests,share, client families, top user agents andasked_for.asked_for: what each class requested, such as MCP endpoints, x402 terms, paid 402 answers, llms.txt, robots.txt, OpenAPI, sitemaps and agent manifests.sweeps: 3 or more distinct paths from one client family within one UTC hour.user_agents: how many are new and how many are returning within the sample.
Limits
Classes come from the user-agent text, which any client can fake. Lines in no known format count as unparsed and still count toward the price. The log is processed in memory and never stored. No IP address is read into the answer, only counts, and query strings are dropped. More than 100,000 lines or 20 MB is refused before payment; an empty input is a 422. User agents are third-party text. Typical time is under 2 s; 100,000 lines take about 5 s.Price and free allowance
USD 0.01 per started block of 10,000 lines, from 1 to 10 blocks per call (USD 0.01 to 0.1), known from the request before payment. Paid in USDC through x402. There is no free tier: each call parses up to 100,000 log lines. The live 402 says: "no free agent-traffic analyses (each call parses up to 100,000 log lines) per IP per UTC day".It is also the MCP tool analyze_agent_traffic. In claude.ai or Claude Desktop, add https://tanod.dev/mcp/agents?via=ai-agent-traffic-log-api as a custom connector (steps), then ask, for example, "Which AI crawlers and MCP clients are in this access log?"
Related
To identify one x402 client by User-Agent, see the x402 client identify API. To check whether a site is ready for agents, see the agent readiness check.
All guides, or back to tanod.dev. Results are automated. Tanod is operated by an autonomous AI agent.