Nick Gray, founder of the Austin, Texas site PatronView, blocked Amazon's Amzn-SearchBot on 5 August 2026 after logging roughly 117,000 requests a day from it, and PPC Land reported the case on 8 August 2026. The block came after an earlier one against Anthropic's Claude-SearchBot in June 2026, a crawler Gray measured at 35,000 page requests for every one human visitor it sent him.
What makes the case worth reading is not the block itself. It is that Gray published server side numbers. Most discussion of AI crawling runs on estimates and vendor claims. Here there are request counts, byte counts, challenge solve rates and a before and after on one specific site.
What PatronView measured on its own server
PPC Land reported on 8 August 2026 that PatronView is built from philanthropic disclosure documents, runs to about 1.5 million pages and serves roughly 3.5 million page views a month. Across one week the server processed 2.5 million requests and delivered 1.28 million full pages, while the site's Plausible analytics recorded 5,977 human page views over the same period. That is 214 non-human page loads for every human one. Gray's own summary of the situation was blunt: "99% of my traffic is from bots".
The bandwidth split says the same thing in a different unit. In that week the site sent 4.63 gigabytes to one search crawler and 175 kilobytes to human readers. Anthropic's Claude-SearchBot requested 420,680 pages during the week. Claude-User, the agent that fetches a page because a person asked a question, referred 12 human visitors in the same window.
Infrastructure cost tracked the load. PPC Land put PatronView's normal monthly hosting at about 90 US dollars, with peaks running around 500% higher during heavy scraping periods. The Cloudflare plan doing the filtering is the Pro tier at 25 US dollars a month.
The measured ratios, crawler by crawler
Gray did not report a single blended figure. He reported per crawler ratios, and the spread between them is the most useful part of the whole case. The table below carries only the figures PPC Land published on 8 August 2026.
| Crawler | Volume measured | Crawl to referral ratio |
|---|---|---|
| Claude-SearchBot (Anthropic) | 420,680 pages in one week | 35,000 to 1, measured June 2026 |
| Amzn-SearchBot (Amazon) | About 117,000 requests a day | Not given by the source |
| Bingbot (Microsoft) | 158,610 pages crawled | 406 to 1, from 680 referred visitors |
| Googlebot | Not given by the source | 46 crawls per referred visitor |
For context, PPC Land noted that Cloudflare has disclosed a crawl to referral range running from 118 to nearly 50,000 across the crawlers it observes. PatronView's Anthropic figure sits near the top of that published range rather than outside it.
What Gray configured, and what changed after
PPC Land described nine Cloudflare firewall rules running on the 25 dollar Pro plan. They cover country blocks, blocks on named crawlers, a skip rule so Cloudflare verified bots pass through, continent level challenges, challenges on datacentre ASNs including an extension to Azure, a stale browser version check, and a rate limit on requests for extensionless page URLs.
The country blocking came first and came from a specific event. Between 21 and 23 April 2026 the site took 3.6 million requests from 361,844 unique IP addresses, mostly originating in China. China was blocked on 23 April 2026, with Vietnam and Singapore blocked afterwards. Baseline unique IP volume was around 18,000 a day; on 31 July 2026 it spiked to 124,000.
The enforcement numbers are where the challenge model breaks down. Cloudflare issued 106,437 challenges over 48 hours and 252 of them were solved, a solve rate of 0.24%. A separate window recorded 63,969 challenges against 552 solves. In the 24 hours after the newest rules went live, 46,729 requests were blocked outright, 43,150 of them from Amazon's crawler.
Blocking a crawler that authenticates and honours robots.txt works almost completely. After the Anthropic crawler was blocked, its daily request volume fell from about 60,000 a day to roughly 25 attempts a day, a drop of 99.96%. That is the shape of a well behaved crawler backing off, and it is a different problem from unidentified scraping traffic, which is what the country rules and datacentre challenges exist to handle.
One side effect is worth flagging because it cuts the other way. Gray disabled Cloudflare's JavaScript Detections feature on 5 August 2026 and the site's Lighthouse mobile score moved from 58 to 99. Bot defence had been costing measurable front end performance.
Why PatronView is not a normal site, and why that matters
This is one publisher with an unusual profile. A 1.5 million page archive of public record documents is close to an ideal crawl target: high page count, stable URLs, structured factual content, and very little reason for a human to read page 900,000 of it. A 300 page services site or a 2,000 post blog does not present the same surface area, and the source does not claim that Amazon's or Anthropic's crawlers behave this way anywhere else.
The blanket country blocks are the operator's own decision, made for his own traffic mix. PPC Land reports them as fact. It does not present them as advice, and for most commercial sites, blocking Singapore outright would remove real customers alongside the scrapers.
How to measure your own crawl to referral ratio
The measurement is not difficult and it does not need new tooling. It needs two numbers pulled from two different systems for the same date range.
- Crawl side. Pull raw access logs from your host, or use your CDN's bot analytics if you have one. Group requests by user agent string and count hits for each named AI crawler over a fixed window, for example 30 days. On Cloudflare this lives under the security and bot analytics screens.
- Referral side. In GA4, open Reports, then Acquisition, then Traffic acquisition, and switch the dimension to session source or medium. Look for assistant hostnames in your referral sources. In Plausible or a similar tool, the referrers report does the same job.
- Divide. Crawler requests divided by referred sessions for the same window gives your ratio per crawler. Run it per crawler, never blended, because the PatronView numbers show the spread between crawlers is larger than the average is informative.
- Cost side. Check bandwidth and origin requests in your hosting bill for the same period, so the argument is about money rather than about principle.
Googlebot's own volume can be checked separately in Google Search Console under Settings and then Crawl stats, which gives request counts and response sizes without touching server logs. Anyone doing this exercise as part of a wider technical SEO audit should log the baseline before changing anything, because without a before figure the after figure proves nothing.
What a block does, and what it does not do
A robots.txt directive is a request. A firewall rule is enforcement. The two are not interchangeable, and the PatronView numbers show why: the authenticated crawler that reads robots.txt dropped by 99.96%, while unidentified traffic had to be met with challenges that were almost never solved.
What a block does not do is protect content that has already been fetched. It also does not distinguish between a crawler building a search index that may cite you and a crawler collecting training data. Where a company runs separate user agents for those jobs, blocking the wrong one can remove a site from assistant answers while leaving the heavy fetching untouched. That trade off is the real decision, and it is a different decision for a documents archive that gets no citations than for a business whose visibility inside AI answers is a lead source.
What the source did not say
PPC Land's report does not include comment from Amazon or Anthropic on the figures. It does not say whether either crawler produces similar ratios on other sites, so nothing here can be read as a general rate. It does not say whether PatronView lost citations, referral traffic or revenue after the blocks, only that request volume fell. It does not quantify how much human traffic the country blocks removed alongside the bot traffic. It gives no figure for how the blocks changed the roughly 90 dollar monthly infrastructure cost afterwards. And it says nothing about Thailand or any Thai site.
What this means for Thai marketers
The source carries no Thai data, so what follows is reasoning rather than reporting. A Thai site with a few hundred pages will not see 117,000 requests a day, and the honest first step is measuring rather than assuming a problem exists. Where the case does read across is on cost structure: sites hosted on metered plans, or sites running heavy server side rendering on every request, pay per fetch in a way that a static site does not. A Thai ecommerce catalogue with faceted filters can generate a very large crawlable URL space from a modest product count, and that is the profile most likely to produce a bill nobody planned for.
The second read across is about what a block costs. Thai language content is thinner on the ground than English, which arguably makes each Thai page more likely to be cited when an assistant answers a Thai query. Blocking a search crawler to save origin cost may be a poor trade for a site competing in Thai organic search, and a reasonable one for an archive nobody cites. That judgement depends on measured numbers from the specific site, not on the PatronView figures.
FAQ on AI crawler blocking and crawl to referral ratios
Does this apply to my site?
Probably not at this scale, and the source does not claim it does. PatronView is a 1.5 million page public records archive, which is an unusually attractive crawl target. The measurement method applies to any site; the volumes reported do not.
Do I need to do anything right now?
No, unless your hosting costs or origin load have moved without an explanation. The reasonable action is to pull 30 days of logs and calculate your own ratio per crawler before deciding anything. Nothing in the PPC Land report is a directive to block.
Will blocking an AI crawler hurt my search rankings?
The source does not address ranking effects at all. What it reports is request volume, not ranking or citation outcomes, and it does not say whether PatronView lost anything after blocking. Google's crawler and AI assistant crawlers are separate user agents, so the answer depends on which ones a rule actually catches.
Is a 35,000 to 1 ratio normal?
The source does not say it is normal. It reports that figure for one crawler on one site in June 2026, and notes Cloudflare has disclosed an observed range from 118 to nearly 50,000, which puts the PatronView measurement near the high end of published figures rather than beyond them.
Can I block bots without a paid plan?
The source does not test free plans. It documents nine rules running on Cloudflare's Pro tier at 25 US dollars a month, and does not say which of those nine are available on lower tiers.
The measurement matters more than the block
The block is one operator's answer to one site's problem. The number is the transferable part: a crawl to referral ratio, calculated per crawler from your own logs against your own referral data, turns an argument about AI crawling into a line item you can price. Relevant Audience works with Thai and regional sites on exactly that kind of question, where crawl budget, hosting cost and assistant visibility have to be weighed against each other rather than treated separately. The original reporting is available from PPC Land.
