Noindex Tag: What It Does and When to Use It

A noindex tag is a rule that tells search engines to keep a page out of their results. You add it either as a robots meta tag in the page's HTML head or as an X-Robots-Tag HTTP header, and once Googlebot crawls the page and reads the rule, Google drops that page from Google Search even if other sites link to it.

That last condition matters more than it looks. The rule only works if Google can fetch the page and see it. Most noindex problems come from that gap: a rule that was never read, a rule that was left on by accident, or a rule used for a job it was not built to do. This guide covers how the noindex tag works, how it differs from blocking a URL in robots.txt, which pages deserve it, and how to confirm in Search Console that it is doing what you meant.

What a noindex tag does

Search engines handle a page in stages. They discover the URL, crawl it, process what they find, and then decide whether to add it to the index, which is the database results are drawn from. A noindex rule acts at the indexing stage. Google still crawls the page, still reads it, and can still follow the links on it, but it will not show the page in search results.

If the page is already indexed, adding noindex does not remove it on the spot. Google has to come back, crawl the page again, see the new rule and then drop it. If the page is not yet indexed, the rule stops it from being added in the first place.

Noindex is also page-level. It applies to the single URL that carries it, not to a folder, a subdomain or a whole site. To keep a section out of search, every URL in that section needs the rule, which is usually handled by a template, a CMS setting or a server rule rather than by hand.

Two ways to add it: meta robots tag vs X-Robots-Tag

Google documents two methods, and they have the same effect. Pick the one that fits how your site is built.

The robots meta tag

This is a single line placed inside the <head> of an HTML page:

<meta name="robots" content="noindex">

The name "robots" addresses every search engine that supports the rule. If you only want to address Google, use name="googlebot" instead. Most CMS platforms expose this as a checkbox. In WordPress, SEO plugins usually add a per-page "allow search engines to show this page" option, and Shopify themes can output the tag through their template code. In those cases you change the setting, not the HTML.

The X-Robots-Tag HTTP header

The same rule can be sent by the server as part of the HTTP response instead of inside the page:

X-Robots-Tag: noindex

The header is the only option for files that have no HTML head, such as PDFs, images, video files and other downloads. It is also handy when you need to apply the rule to many URLs at once, for example every file in a directory, through a server or CDN rule. The value "none" is shorthand for noindex plus nofollow and works in both the header and the meta tag.

If a page sends conflicting robots rules, for example an "index" meta tag and a "noindex" header, Google applies the more restrictive one. That is a common source of confusion when a CMS and a server config both set rules without anyone knowing.

Noindex vs robots.txt disallow

This is the mistake that causes the most trouble. A Disallow line in robots.txt tells crawlers not to fetch a URL. A noindex rule tells search engines not to show a URL. They answer different questions, and combining them on the same page cancels the noindex.

Here is why. If robots.txt blocks a page, Googlebot never downloads it, so it never sees the noindex tag inside it. Google's documentation is explicit that a blocked page can still appear in results if other pages link to it. It usually shows up with the URL and no description, because Google knows the address exists but was not allowed to read the content.

So the order of operations for removing a page is:

1. Make sure robots.txt does not block the URL.
2. Add the noindex rule, either as a meta tag or a header.
3. Wait for Google to recrawl, or request a recrawl with the URL Inspection tool.
4. Only after the page has dropped out, and only if you also want to save crawl activity on it, consider blocking it in robots.txt. Bear in mind that once you block it, Google can no longer see the noindex, so any new links pointing at it can bring the bare URL back into results.

One more point: Google does not support a noindex line inside robots.txt itself. Some older guides still recommend it. It has no effect in Google, so do not rely on it.

noindex, follow vs noindex, nofollow

You can combine noindex with other rules, separated by a comma. The two combinations people ask about most are:

• noindex, follow: keep the page out of results but let crawlers follow its links. Since "follow" is the default, plain noindex already behaves this way. Writing "follow" adds nothing technically, though some teams like it for clarity.
• noindex, nofollow (or "none"): keep the page out of results and ask crawlers not to follow the links on it. This suits pages whose links you do not want crawled at all, such as an internal search results page that links to endless filter combinations.

Be careful with nofollow on pages that are part of your navigation. If a noindexed category or tag page is the main route to your products or articles, nofollow cuts that route. Do not use a noindexed page as the only path to content you want indexed. Link to important pages from indexable ones as well.

When to use a noindex tag

Noindex fits pages that have a reason to exist for users but no reason to appear in search. Typical candidates:

• Thank-you and confirmation pages after a form submission or purchase. A visitor landing on one from search would be confused, and it can pollute conversion tracking.
• Internal site search results. These pages are generated on demand, can be created in unlimited numbers and rarely add anything a dedicated page does not.
• Thin tag and archive pages on blogs that list a handful of posts with no unique text.
• Faceted filter combinations on ecommerce sites, such as color plus size plus price range, where each combination makes a near-empty URL. Whether noindex, canonical or crawl controls is the better fix depends on how many combinations exist and whether any of them match real searches.
• Login, account, cart and checkout pages.
• Staging copies, test pages and campaign landing pages that duplicate the main site or exist only for paid traffic.
• Downloads you do not want found in search, such as a price list PDF, using the X-Robots-Tag header.

When not to use noindex

Noindex is the wrong tool in a few situations that come up often.

• Choosing between duplicate versions of the same page. If two URLs show the same content and you want one of them to rank, a canonical tag or a redirect consolidates signals onto the preferred version. Google specifically advises against using noindex to pick a canonical within a site.
• Hiding private or sensitive content. Noindex keeps a page out of results, but anyone with the link can still open it. Use a password or login for anything confidential.
• Pages you want to rank later. A noindex left in place "for now" is easy to forget. If a page is weak, improving or merging it is usually better than hiding it.
• Removing a page urgently. Noindex waits for a recrawl. For something that must disappear fast, Search Console's Removals tool hides a URL temporarily while the permanent fix takes effect.

How long until a noindexed page drops out

There is no fixed time. Google has to recrawl the page before it can act on the rule, and how often a page is recrawled depends on how important and how frequently updated Google considers it. Google's own documentation says it may take months for Googlebot to revisit some pages. A homepage or a popular category page can be recrawled quickly, while a deep, rarely linked page can sit in the index for a long time after you add the tag.

You can speed this up for individual URLs. Open the URL Inspection tool in Search Console, run a live test to confirm Google sees the noindex, then click Request Indexing. That asks Google to recrawl the page, and the recrawl is what lets it apply the rule. For many URLs, keeping them in your XML sitemap with an updated lastmod date for a short period can prompt recrawling, but remove them from the sitemap once they have dropped out, since a sitemap should list pages you want indexed.

How to find noindexed pages in Search Console

Search Console gives you two views that cover most checks.

The Page indexing report. Under Indexing, open Pages and look at the reasons pages are not indexed. "Excluded by 'noindex' tag" lists URLs where Google found the rule. Read through this list carefully. Every URL on it should be one you meant to hide. If you see product pages, service pages or blog posts there, something in a template or plugin is adding the rule where it should not. You can also filter the report by sitemap. A URL that appears in your sitemap and is also marked noindex is sending Google two opposite instructions, and it is worth fixing whichever of the two is wrong.

The URL Inspection tool. Paste any URL to see whether indexing is allowed and, if not, whether the noindex came from the meta tag or the HTTP header. The live test shows the HTML Googlebot actually received, which is the version that counts.

Outside Search Console, a site crawler will list every URL with a noindex in either location. That is the fastest way to audit a large site before and after a migration or a redesign.

Common noindex mistakes

• Staging settings carried into production. Many CMS platforms have a site-wide "discourage search engines" switch for development sites. If it travels with the launch, the whole site can fall out of search. Check this on every launch day.
• Relying on JavaScript to remove noindex. Google's JavaScript guidance says that when Googlebot finds noindex in the original HTML, it may skip rendering the page, so a script that later changes the tag to "index" may never run. Serve the correct rule in the initial HTML.
• Noindex plus robots.txt block on the same URL, covered above. The block hides the rule.
• Noindexed pages kept in the XML sitemap long after they were hidden.
• Plugin conflicts. Two SEO plugins, or a plugin and a theme, each writing their own robots tag can produce conflicting rules. Google then applies the more restrictive one.
• Noindex on paginated series. Hiding page 2, 3 and onward of a category can cut crawlers off from the products listed there. Keep pagination crawlable and make sure items are linked.

Noindex tag compared with other controls

What this means for Thai sites

Bilingual Thai and English sites need one extra check. If the Thai and English versions are linked with hreflang, both versions should be indexable. A noindex on the Thai template, which can happen when a translation plugin sets its own robots rules, removes the Thai pages from Google while the English ones stay, and the language pairing stops working for Thai searchers. Inspect a Thai URL and its English pair side by side after any plugin update.

Thai ecommerce sites built on WooCommerce or Shopify tend to generate many filter and sort URLs, and URLs containing Thai characters are percent-encoded, which makes them hard to read in reports. Export the "Excluded by 'noindex' tag" list and decode the URLs before reviewing it, so you can tell a filter page from a real category. Campaign landing pages built for LINE, Facebook or Google Ads traffic are also common candidates for noindex when they duplicate an existing service or product page.

If you are unsure which pages on your site carry the rule, an SEO audit crawls every URL and lists the noindex, robots.txt and canonical signals in one place, so conflicts become visible before they cost traffic.

FAQ about the noindex tag

Does a noindex tag stop Google from crawling a page?

No, a noindex tag does not stop crawling; Google has to crawl the page to read the rule. To limit crawling you use robots.txt, but blocking a page there also hides its noindex, so do not combine the two on a page you want removed.

Can I put noindex in robots.txt?

No, Google does not support a noindex rule inside robots.txt. Use a robots meta tag in the page or an X-Robots-Tag HTTP header instead.

Should I noindex tag and category pages?

It depends on whether they carry unique value. A category page with its own description and a useful product list can rank well and should usually stay indexable, while thin tag pages that list two or three posts are reasonable candidates for noindex.

How do I noindex a PDF?

Send an X-Robots-Tag: noindex HTTP header with the file, because a PDF has no HTML head for a meta tag. This is set on the server, in a CDN rule, or through your hosting control panel.

Will noindex hurt the rest of my site's rankings?

Noindexing pages that should not be in search does not harm the pages that remain indexed. The risk is applying it by mistake to pages that matter, which is why the Page indexing report is worth checking after every template or plugin change.

Next step

A noindex tag is a small line of code, but on a large site it is easy to lose track of where it is set and why. If you want indexing controls reviewed alongside the rest of your technical setup, Relevant Audience's SEO services cover crawl and index audits, and online stores can start with the ecommerce SEO team, who work with filter, pagination and product URL structures. For businesses targeting Thai and English searchers, see our SEO in Thailand page.