To block, or not to Block AI – Coudflare’s, new tools give the choice

To Block AI Bots or Not – That Is the Question

That’s probably my favourite Shakespeare quote, second only to:

At Glass Mountains, we’re big proponents of Cloudflare, often recommending it to clients because of the ease with which it adds a valuable layer of security and performance to a website or domain.

Now, as the internet infrastructure giant powering roughly one-fifth of all live websites, Cloudflare has just taken a major step in giving site owners more control over how AI companies access and use their content.

 

Cloudflare’s Expanding AI Arsenal

Cloudflare isn’t anti-AI, far from it. They’ve been steadily adding AI-friendly tools like Workers AI, which lets developers run AI models directly on Cloudflare’s edge network.

But they are positioning themselves in an interesting middle ground: supporting the growth of AI while also protecting the rights of publishers, creators, and businesses.

One of the newest additions to their toolkit is a set of AI-blocking features that can stop AI crawlers (the bots that collect data for training models) from scraping your site without permission.
This isn’t all about blocking of course, using the tool gives you the ability to explicitly “allow” AI crawlers—so instead of AI crawlers potentially just assuming they have access, or being unsure, they now have clarity of what they should or should not be crawling.

Whether you run a blog, an ecommerce store, or a business site, your content may already be fuelling AI tools without your knowledge. Cloudflare’s new features let you take back control.

The Catch-22 of AI Visibility

Here’s the tricky bit: as a business, you may want to appear in AI-generated answers or recommendations, after all, these tools are fast becoming part of how people search for and discover information.

But at the same time, you may not want your content copied, repurposed, and served up without traffic, acknowledgement, or payment.

It’s a balancing act between visibility and protection, and Cloudflare’s tools aim to give you more say in where that line is drawn.

Step One: The AI Crawl Control

If you’re wondering just how much of your website is already being scraped by AI bots, Cloudflare’s AI Crawl Control is the place to start.

It scans your traffic logs and identifies which AI crawlers are visiting, how often, and what pages they’re targeting.

This is often eye-opening, especially if you assumed only search engines like Google were indexing your site. In reality, AI crawlers are often far more aggressive, and unlike search engines, they don’t send visitors back in return.

 

Step Two: Choose what to block

Once you know what’s hitting your site, you can take action with Cloudflare’s AI bot blocking tools.

Through the Bot Fight Mode and AI-specific rules in the Cloudflare dashboard, you can:

• Block all AI bots by default — using Cloudflare’s regularly updated list of known AI crawlers.
• Create exceptions for bots you trust, such as Google’s crawler.
• Log and challenge suspicious traffic before it can reach your content.

The system works at the network level, meaning AI bots are stopped before they even begin scraping, far more effective than relying on robots.txt alone.

Further reading on the Cloudflare blog →

 

Step Three: The AI Labyrinth – Fighting Back Against Rogue Bots

Not all bots play fair. Some AI crawlers ignore robots.txt and other “do not crawl” signals entirely.

For those, Cloudflare offers something straight out of a digital Greek myth, The AI Labyrinth.

If a bot is caught ignoring the rules, it can be redirected into a deliberately confusing rabbit hole of:

• Fake content
• Resource-heavy pages
• Endless, crawlable links

The goal? Waste the bot’s time, feed it fake data, and slow down its operation, while also collecting valuable info on its behaviour.

The interesting part here is Cloudflare is using the AI crawlers techniques to train their own AI detection, it really is evolving battle that seems it will only get more fierce over time:

When bots hit these URLs, we can be confident they aren’t actual humans, and this information is recorded and automatically fed to our machine learning models to help improve our bot identification.

Read more about The AI Labyrinth →

 

Pay Per Crawl – Putting a Price on Access

An interesting feature currently in closed beta is Pay Per Crawl.

The idea is simple: if an AI company wants to use your content, they can—but they’ll have to pay for the privilege.

For websites that produce valuable data, articles, or creative work, this could open up a small but meaningful revenue stream, or at least help fund the cost of producing that content in the first place.

Further reading on Pay Per Crawl → 

 

Why This Matters

In the UK, the tension between AI companies and creators is heating up. The BBC has already threatened legal action against US-based Perplexity AI for using its content without permission, while artists like Sir Elton John (BBC Article: https://www.bbc.co.uk/news/articles/c8e6z590wweo →) have been vocal about protecting copyright in the age of machine learning.

What’s at stake isn’t just fair payment, it’s the future of the creative economy. As Cloudflare CEO Matthew Prince put it:

“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone.”

Cloudflare blog post regarding their new AI blocking stance →

The Bigger Picture

To block, or not to block—at least now you can decide—Cloudflare’s tools aren’t a silver bullet, they can only protect content on sites that use their services, and legal protections will still be essential.

But these measures do offer something valuable: immediate, practical ways to push back against unwanted AI scraping and to create a framework for a fairer, more balanced AI economy.

It also sets a precedent and raises awareness, by making AI blocking the default option, one of the largest names in the industry is setting a clear tone and baseline.

One thing that’s still clear to me: these tools are relatively easy for tech-savvy users, but for the average person; setting up a Cloudflare account, implementing DNS changes, and knowing which AI blocking options to enable/configure isn’t an everyday skill. This makes it all too easy for AI scrapers to take advantage of those not in the know.
At Glass Mountains, we can release you of this burden of course, get in with us contact today if you’d like us to help configure your site for the age of AI.

No Comments