Search Engine Journal published a server log analysis on 14 August 2026 showing that the single largest so called AI crawler hitting one small website never read a page of content. It spent its requests looking for credential files instead. The review covered 24 hours of logs for the site nohacks.co ending on 7 August 2026, roughly 3,000 requests in total, about a third of them unsuccessful.
The top user agent by volume was CCBot at 1,510 requests, the string attributed to Common Crawl, well ahead of ChatGPT-User at 375, ClaudeBot at 296 and Googlebot at 245. The part that matters is what that top talker asked for. Across 100 credential scanning paths the site logged 1,028 requests and 6.7 MB of bandwidth with zero content accessed. The paths requested included /.ssh/known_hosts 42 times, /.boto 30 times and /.env.production 29 times, plus Firebase keys, AWS configuration files and private keys. Two of the paths are new: /.mcp.json at 30 requests and /.continue/config.json at 24 requests, both of them files that can hold API keys and agent authentication tokens.
The Cloudflare forecast that prompted the log review
The analysis was written to test a claim made by Cloudflare chief financial officer Thomas Seifert on the company's second quarter earnings call. Seifert said that within five years non-human traffic will be as much as 1,000 times as much as human traffic, and that humans will be a rounding error on the internet. He also conceded that his own earlier forecasts had underestimated the growth. Machine traffic overtook human traffic on Cloudflare's network in May 2026, earlier than the 2027 crossover he had previously projected.
A forecast at that scale is easy to repeat and hard to check. Search Engine Journal, whose analysis is published in full here, took a narrower and more testable question to a real log file: of the traffic arriving with a bot user agent, how much of it is a crawler doing crawler work? On this one site, on this one day, the biggest bot by request count was not crawling at all.
What the top talker actually requested
The ranking below is the request volume by user agent string over the 24 hour window ending 7 August 2026, as reported by Search Engine Journal. Read it as a ranking of self declared identities, not verified crawlers, because that distinction is the whole point of the story.
| User agent as logged | Requests in 24 hours |
|---|---|
| CCBot (attributed to Common Crawl) | 1,510 |
| ChatGPT-User | 375 |
| ClaudeBot | 296 |
| Googlebot | 245 |
A dashboard built from that table alone tells a tidy story: AI crawlers now dominate this site's traffic, and Common Crawl is the heaviest consumer of its bandwidth. The log file says something different. The 1,028 requests that hit credential paths returned no content, because there was no content at those paths to return. That is 6.7 MB of transfer spent on 404 responses and probing, and it is the reason roughly a third of the day's requests were unsuccessful.
Credential scanning dressed up as a crawler
The requested paths are the tell. A crawler that wants content asks for HTML pages, sitemaps, feeds and images. A scanner asks for files that were never meant to be served over HTTP at all: /.ssh/known_hosts, which maps the hosts a server has connected to, /.boto, which holds credentials for cloud storage clients, and /.env.production, which in a badly deployed application holds database passwords, payment keys and API secrets in plain text. Search Engine Journal also recorded requests for Firebase keys, AWS configuration files and private keys inside the same run of 100 paths.
None of that requires the requester to be Common Crawl, and the evidence in the article points away from it. A user agent string is a header the client writes about itself. Anyone can send any string they like, and a scanner that wants its traffic to look boring in a log file has an obvious incentive to borrow the name of a crawler that site owners already expect to see.
Agent configuration files are now on the scanning list
The genuinely new item in this log sample is the pair of agent configuration paths. /.mcp.json received 30 requests and /.continue/config.json received 24 in the same 24 hours. Both are files that development tooling writes into a project directory, and both can contain API keys and agent authentication tokens depending on how they were set up.
That matters for a specific group of teams: the ones that have been experimenting with agent tooling over the past year and have those files sitting in a repository. The risk is not the file itself, it is the file ending up inside a web root and being served. If a project directory is deployed wholesale to a server, or if a build copies more than it should, a configuration file written for a local editor becomes a public URL. Scanners are now asking for those URLs by name, which means the window between a bad deploy and a stolen token is however long it takes the next scan to come round.
Why a log report grouped by user agent will mislead you
There are two separate lessons in this article, and the measurement one is easy to skip past because the security one is louder. If you are reporting on AI crawler load, whether to a client or to your own management, a report that groups requests by user agent string and stops there will overstate genuine AI crawling. Some share of what appears under a crawler name is scanning traffic wearing that name.
- Verify the identity before you attribute the traffic. Reverse DNS lookup on the requesting IP, then a forward lookup on the name it returns, is the standard check, and the major crawler operators publish IP ranges you can match against.
- Report on paths, not only on counts. A crawler that requested 1,500 URLs of which none were content is not a crawler, and only the path column shows that.
- Split successful responses from failures. A high proportion of 404 responses in a bot's traffic is a signal in itself.
- Do not set crawl budget policy, rate limits or robots rules on the basis of a spoofed name. Blocking a user agent string blocks the clients that honestly report it and leaves the scanner untouched, because the scanner will simply send a different string.
This is ordinary log hygiene rather than anything exotic, and it belongs in the same pass as the rest of a technical review. If you want the wider picture of how machine readers reach your pages, our notes on AI search and answer engine visibility cover what genuine AI crawler access looks like when it is working, and a scheduled SEO audit is the natural place to put the log check so it happens more than once.
What to check on your own site
The practical work here is short and worth doing whether or not your logs look like the ones in this study.
- Request the paths yourself, over HTTPS, against your live site: /.env, /.env.production, /.boto, /.ssh/known_hosts, /.mcp.json and /.continue/config.json. Anything other than a 404 or a 403 is a finding.
- Treat a 200 response on any of those paths as an incident, not a task for next sprint. The correct order is rotate the exposed credentials first, remove the file second, then work out how it got there.
- Confirm your web root contains only what you intend to serve. Deployment artefacts, environment files, editor configuration, version control directories and backup copies do not belong in a public directory.
- Block dotfiles at the server or the edge, so that a future mistake fails closed. A rule that denies requests for paths beginning with a dot costs nothing and covers files that do not exist yet.
- Re-run your bot reporting with the path dimension included, and verify the top few user agents by reverse DNS before you write any conclusion about AI crawler volume.
For sites that publish heavily and want machine readers to reach their content, none of this conflicts with being open to legitimate crawlers. The goal is a site where the crawlers that should read you can, and the paths that should never be public are not. That balance is the same one behind most AI SEO work: make the content easy to fetch and make everything else unreachable.
The honest limits of this data
This is one website over one 24 hour period, published by the author of the site in question. It is not a representative study, it does not establish what share of AI crawler traffic across the web is spoofed, and it cannot be extended into a percentage without more sites and more days. Search Engine Journal presents it as a log review, and it should be read that way.
The article also does not establish that Common Crawl itself did anything wrong. What it shows is traffic that identified itself as CCBot and behaved like a credential scanner. Attribution of that traffic to any particular operator would need the verification steps described above, which is exactly the argument the piece is making. The source does not report a response from Common Crawl, Cloudflare or any crawler operator on the finding.
What this means for Thai marketers
Two things carry over. The first is that any report you produce about AI crawler load on a Thai client's site is only as good as its verification step, and a chart of user agents with no path column can hand a client a conclusion that the logs do not support. If the number is going into a deck, verify the top agents first.
The second is a deployment question rather than a marketing one, and it lands on whoever owns the hosting. Agencies and in house teams in Bangkok have spent the past year adding agent tooling to their workflows, and the configuration files those tools write are now being requested by name. Checking six URLs on each site you manage takes a few minutes. Where the site is on shared hosting or a self managed server with a hand rolled deploy, that check is worth more than the time it costs. Thai language sites are not treated differently by a scanner: it is asking for file paths, and file paths do not have a language.
Frequently asked questions
Does this mean AI crawler traffic is not really growing?
No, and the article does not claim that. It claims that a portion of what gets counted as AI crawler traffic in a raw log report is something else, so the headline numbers overstate genuine AI crawling. Cloudflare's own reported crossover, machine traffic passing human traffic on its network in May 2026, is a separate measurement and is not disputed here.
Is my site at risk if my team has never used agent tooling?
The two agent configuration paths only matter if those files exist in your project, but the rest of the list applies to everyone. /.env and cloud credential files predate this story by years and were the bulk of the scanning in this sample. The check is the same either way, and it takes minutes.
How do I tell a real crawler from a spoofed one?
Reverse DNS on the requesting IP address, a forward lookup to confirm the name resolves back, and a comparison against the crawler operator's published IP ranges. The user agent string on its own proves nothing, because the client writes it. Path behaviour is the secondary signal: real crawlers request content, scanners request configuration files.
Should I block CCBot because of this?
The source does not recommend that, and blocking a name is a weak control against a client that can change the name it sends. The action the finding supports is making the credential paths unreachable and verifying identity before you attribute traffic. Whether you allow Common Crawl to read your content is a separate editorial decision.
Is this live in Thailand or specific to one region?
The question does not really apply, because this is a log review of one site rather than a product rollout. Credential scanning is not geographically targeted in any way the source describes, and the article does not report on regional differences at all.
If you want a second pair of eyes on what your own logs are actually saying about crawler traffic, and on whether anything is being served that should not be, that is the kind of check we run as part of a technical review. Start with the six URLs above, then decide how much more digging the result deserves.
