Gary Illyes, a Search Relations analyst at Google, was asked on LinkedIn whether repeatedly pushing an unavailable_after date forward causes problems. His answer, reported by PPC Land on July 28, 2026, was that he did not know. He offered that moving the date forward is "probably acceptable", then added the part that matters operationally: Google has to recrawl the page before any change to the tag registers at all.
Take the uncertainty at face value. Illyes said he did not have the answer, so this is a qualified reply from a Google spokesperson rather than settled guidance. The recrawl requirement is the part that is not a matter of opinion.
What the unavailable_after directive does
The unavailable_after robots directive tells Google when a URL should stop appearing in search results. It is a visibility instruction with a date attached. Once the date passes, the URL drops out of the results.
The catch is in delivery. Google only learns the date, or a changed date, by crawling the page. Illyes also noted that Google rarely issues HEAD requests, so the crawler downloads the full page even when the only thing worth checking is a date. Change the date on ten thousand listings and nothing happens until ten thousand full page fetches happen first.
The crawl waste it does not fix
The exchange was covered by PPC Land, which frames the limitation around sites with high-turnover inventory. That framing is the useful part of the story. The directive controls whether a URL shows in search. It does nothing about 404 errors, and nothing about crawl budget spent on pages that no longer exist. On sites where more than half of Googlebot's requests land on expired pages, the tag leaves that untouched.
So the assumption that tagging expiring pages handles crawl waste is wrong twice over. The tag needs a crawl to take effect, and the crawl is the thing you were trying to save.
What to do with expiring pages instead
- Decide what an expired URL should return. A 404 or 410 removes it, though Googlebot still has to request it to find out. A redirect to a live category or search page keeps the link value and gives visitors somewhere to land.
- Drop expired URLs from the XML sitemap on the same schedule the listings expire. Leaving them there is an active invitation to recrawl.
- Read the server logs before deciding anything. The share of Googlebot hits landing on expired URLs is measurable, and it is the number that tells you whether this is a real problem or a tidy one.
- Cut the internal links pointing at expired listings. Most crawl waste on inventory sites is self-inflicted through pagination and related-item modules.
- Use unavailable_after for what it is: control over how long a page stays visible, on pages that will be recrawled anyway.
What this means for Thai marketers
This lands hardest on Thai property portals, job boards, and travel and deal sites, where large volumes of listings expire every week. A condo listing that sold, a job that closed, a package that ran out of dates: each leaves a URL behind, and those URLs keep getting requested long after anyone would want them served.
If expired-listing 404s dominate your crawl logs, unavailable_after is not the lever. The lever is the status code you return, the sitemap you publish, and the internal links that keep pointing at dead inventory. That is ordinary technical SEO work, and it is the sort of thing an SEO audit should quantify before anyone touches a robots directive. For an ecommerce catalogue with seasonal SKUs, the same question applies to out-of-stock product URLs.
Common questions
Does unavailable_after prevent 404 errors?
No. It controls whether a URL appears in search results. Expired pages still return whatever status code your server returns, and Googlebot still requests them.
Is it safe to keep pushing the expiry date back?
Illyes said he did not know, and offered that moving the date forward is probably acceptable. Treat that as an unresolved question rather than a green light, and remember the change only takes effect after a recrawl.
How quickly does a change to the tag take effect?
As fast as Google recrawls the page, and no faster. For pages that are crawled rarely, a date change can sit unnoticed for a long time.
If your crawl logs are full of expired inventory, the fix is a crawl-and-status-code exercise, not a robots tag. Relevant Audience can run that analysis on your site and tell you which of the two you actually have.
