In short: Shopify gives you three separate mechanisms for hiding something, they behave differently, and a merchant who uses one thinking it is another removes products from retrieval by accident.
A noindex meta tag, the unlisted product status, and the seo.hidden metafield. The last two also remove the product from your sitemap, which is how a system finds it in the first place.
The fourth thing to understand is robots.txt.liquid, where the common customisation quietly freezes rules that Shopify otherwise keeps current.
Three Mechanisms, Three Reaches
Shopify's own documentation separates them, and the separation is the part worth learning.
A robots meta tag added to the theme head marks a page as not for indexing while leaving it publicly reachable and still in your sitemap. It is the narrowest of the three.
Unlisted product status is the broadest. An unlisted product is reachable only by its direct URL: it is out of internet search, out of the Shopify catalogue, out of your sitemap, and out of your own collection pages, search results and product recommendations.
The seo.hidden metafield, set to 1 on a page, product or blog post, removes it from sitemaps, search engines and your store's own search. Shopify notes it is weaker than unlisted: a product hidden this way can still appear in storefront collections, recommendations and third-party channels.
The failure this causes: a merchant wants a product out of a seasonal collection and reaches for whichever control is nearest. Two of the three take it out of the sitemap, which is the list a retrieval system reads to learn the product exists. The product is then absent from answers for a reason nobody connects to a merchandising decision made months earlier.
The robots.txt Template
Shopify generates a robots.txt by default and it works for most stores. Customising it means creating a robots.txt.liquid template, which themes do not include until you add it.
Shopify's documentation is specific about how to edit it, and the reason matters more than the instruction: use the Liquid robots object and add to it, rather than replacing the output with plain text, because the default rules are updated regularly to keep current with search practice.
Replace the object with a static file and you have taken ownership of a list that used to maintain itself. It will be correct on the day you write it and progressively less so afterwards, and nothing will tell you.
What To Check Before You Edit It
Fetch what is actually being served, and read it against the agents in Meet the AI Crawlers:
curl -s https://example.com/robots.txtThen ask the business question before the technical one. Blocking a live retrieval agent removes you from answers being generated today, and access restored takes as long as the crawler takes to return. Managing AI Crawler Access is the decision underneath the file.
A store on a platform still owns this decision. It is easy to assume the platform has made it for you because the default exists and works. The default is a reasonable starting position rather than a policy, and an inherited one is a policy nobody made.
The Sitemap Is The Discovery Path
Everything above matters for one reason: a retrieval system has to learn a URL exists before it can fetch it, and on a store with thousands of products the sitemap is the main way that happens.
So the practical rule is to treat sitemap presence as the thing you are protecting. Before using any hiding mechanism, ask whether you want the product gone from search or merely out of a collection, because those are different requests and only one of them costs you retrieval.
And check the result rather than trusting the setting: fetch your sitemap and confirm the products you expect are in it.
Key Takeaways
- Shopify has three hiding mechanisms with different reaches: a robots meta tag, unlisted status, and the seo.hidden metafield.
- Unlisted and seo.hidden both remove a product from the sitemap, which is how a retrieval system learns the URL exists at all.
- Unlisted is the stronger of the two; a product hidden by the metafield can still appear in collections, recommendations and third-party channels.
- Customise robots.txt by adding to the Liquid robots object rather than replacing it with static text, because Shopify keeps the default rules current.
- Fetch your robots.txt and your sitemap and check them, rather than trusting what a settings screen says.
Check yourself
Before you move on
Not scored, not recorded, and not part of the certificate. Both answers are settled by a sentence in this lesson, and the reasoning appears whichever option you pick.
- 01
A merchant wants a product out of one seasonal collection and sets it to unlisted. What else has changed?
- 02
Why does Shopify recommend building robots.txt.liquid around the Liquid robots object rather than replacing it with plain text?