TL;DR: On September 15, 2026, Cloudflare is changing how it handles AI crawlers by default and if you don’t check your settings, you could accidentally block Google and the AI platforms your customers are starting to shop through. Cloudflare is splitting bots into three buckets (Search, Agent, and Training) instead of one blanket “allow or block” switch, and the new defaults will restrict Training and Agent bots on any page that runs ads unless you tell it otherwise. The fix isn’t complicated, but it does require you to actually go check your settings before the deadline. Below, we break down what’s changing, why it matters for a DTC brand specifically, and exactly what to check this week.
Quick note before you dive in: this post is specifically about Cloudflare’s settings. If your store isn’t running on Cloudflare, none of the action items below apply to you, and you can stop reading here.
You should also note that Shopify runs through Cloudflare, but you don’t have access to these settings. It will be interesting to see how they adapt.
Why You Should Care About a CDN Setting
If you’re running an ecommerce brand, you’ve probably never thought twice about how Google or ChatGPT actually “sees” your website. You just assume it works. For most of the last decade, that assumption has been safe. You’re busy running Meta and Google ads and organic search is something that’s just there and doing its thing.
That’s starting to change.
Shoppers aren’t only Googling “best stainless steel water bottle” anymore, they’re asking ChatGPT, Claude, and Gemini to recommend one, compare options, and sometimes even complete the purchase for them. If the bots powering those answers can’t actually crawl your site, your brand simply isn’t part of the conversation. Not ranked lower. Not deprioritized. Completely gone.
That’s why a technical change buried in Cloudflare’s dashboard is worth five minutes of your time.
What’s Actually Changing on September 15
Cloudflare sits in front of a massive chunk of the internet (upwards of 60% – nuts, right???), protecting sites from bad bots, DDoS attacks, and scrapers. For the past year, site owners have had a single toggle: block AI bots, or don’t.
Starting September 15, that toggle is retiring. I
n its place, Cloudflare is sorting crawlers into three categories based on what they’re actually doing:
- Search – bots that index your content so it can show up in search results and AI-generated answers
- Agent – bots completing a specific task on behalf of a real person, like an AI assistant checking your return policy for someone who asked
- Training – bots scraping content to train the underlying AI models
Here’s the part that catches people off guard: the new default settings will block both Training and Agent bots on any page that carries advertising, while Search bots stay allowed. That default only applies automatically to new Cloudflare accounts, newly added sites, and existing free-plan customers, but if that’s you and you don’t change anything, the switch flips on its own.
There’s also an issue worth flagging for anyone who assumes Google is exempt: Googlebot itself is a mixed-use crawler, meaning it handles both search indexing and AI training through the same bot. If your settings are configured to block Training crawlers, Cloudflare applies the strictest rule that fits, which means Googlebot gets caught in the net too.
You could unintentionally block Google Search while trying to keep AI models away from your content. DO NOT DO THIS!!!
A Word on Cloudflare’s “Get Paid for Your Content” Pitch
Part of Cloudflare’s push here is the idea that publishers will finally get compensated when AI platforms use their content, instead of having it scraped for free. It’s a nice pitch, but it’s really a pipe dream.
In my opinion, here’s the more likely outcome: if a site tries to charge an AI platform for access, the AI platform isn’t going to pay you. They’ll laugh and move on. It will just end up dropping that site from its index and routing the answer, the citation, and the traffic to a competitor who left the door open. So the brand holding out for a payday doesn’t get paid; it gets replaced, which is far more costly than losing a few bucks in “royalties” from your content. It’s quite possibly the stupidest thing I’ve ever heard. The visibility, the clicks, and the sales that would have come from being cited simply go to whoever didn’t try to charge for access. Being precious about “protecting” your content ends up handing your competitors the exact traffic and leads you were trying to keep to yourself.
This Isn’t Happening in a Vacuum
The Cloudflare deadline is really just the latest chapter in a trend that’s been building for a while. More and more sites have quietly been blocking AI crawlers, sometimes on purpose, often by accident, as brands get nervous about AI companies using their content without sending anything back in return.
We get why. It’s a fair concern. But it’s also where a lot of brands are getting the strategy wrong.
What Our Head of Digital Wants Every Brand to Understand
Our Head of Digital, Kristen Ravesloot, put it best when we were talking through this internally:
“A lot of brands that we talk to are leaning more and more into the idea that Googlebot and the AI crawlers are the bad guys and that they are stealing your content, period. This couldn’t be further from the truth. By allowing them access to crawl, parse, and comprehend your content, you give them enough information to be able to cite your brand in their results, which in turn gives you brand awareness, impressions, clicks, website traffic, and sales. If you block them, you will find that your results will start to slip and potentially go to zero overnight.”
That last line is the one to sit with. This isn’t a slow fade. Block the wrong bot, and your visibility can disappear practically overnight in Google, in AI answers, or both.
What This Means for Your Store
If your site runs through Cloudflare and you’re monetizing any pages with display ads or ad partnerships, think blog content, resource pages, or affiliate placements, those are exactly the pages the new default targets. Product pages without ads may be unaffected by the specific September 15 default, but the broader crawler-blocking trend can still touch them depending on how your settings are configured.
The bigger opportunity here isn’t just “don’t get blocked.” It’s making sure your brand is set up to actually get cited when someone asks an AI platform for a recommendation in your category. That’s a new discovery channel, and it rewards brands that make their content easy to access and easy to understand.
Your Action Checklist
- Log into Cloudflare and check your bot settings. Under Security, look for AI bot policies and see whether you’re still on the old all-or-nothing toggle.
- Set your Search, Agent, and Training preferences deliberately rather than letting the default decide for you.
- Pull your robots.txt file and look for a blanket block. A stray “disallow everything” line is more common than you’d think.
- Make sure your CDN settings and your robots.txt actually agree with each other. A firewall rule can override robots.txt without you realizing it.
- Check your server logs to confirm Googlebot and the major AI crawlers are getting through and not hitting error pages.
- Flag any ad-supported pages on your site — those are the ones most exposed by the September 15 default.
None of this takes a developer team. It takes about twenty minutes and a checklist. Make the time this week to get it done.
FAQs
Do I need to do anything if I’m not on Cloudflare?
No. The September 15 deadline and default changes are specific to Cloudflare’s platform. If your store isn’t running through Cloudflare, this particular checklist doesn’t apply to you.
Will this affect my Google rankings?
It can, indirectly. If your settings end up blocking Googlebot because it’s classified as a mixed-use crawler, your Google visibility can take a real hit. That’s exactly the scenario worth checking for before the deadline.
Should I just block all AI bots to protect my content?
No, it’s a terrible idea. Blocking bots that train AI models is a reasonable call for some brands, but blocking the bots that let AI platforms cite and recommend you removes you from a growing discovery channel entirely.
How do I know if my site is already blocking crawlers by accident?
Check your robots.txt file directly, and review your Cloudflare (or other CDN) bot settings. If you’re not sure how to read either one, that’s exactly the kind of quick audit we can run for you.
Is this a one-time fix?
Not really. New AI crawlers launch regularly, and providers update how their bots identify themselves. Treat this as something to revisit quarterly, not a box you check once and forget.
Greg is the founder and CEO of Stryde and a seasoned digital marketer who has worked with thousands of businesses, large and small, to generate more revenue via online marketing strategy and execution. Greg has written hundreds of blog posts as well as spoken at many events about online marketing strategy. You can follow Greg on Twitter and connect with him on LinkedIn.