For years, “block the bots” was sensible shorthand for protecting a website. Automated traffic can scrape content, probe for vulnerabilities, test stolen passwords, hoard inventory and consume enough server capacity to degrade the experience for everyone else.
That remains true. The difference in 2026 is that bot is no longer a useful enough category for making access decisions.
Some automated visitors are still plainly harmful. Others are search crawlers, advertising-review systems, monitoring tools or AI agents acting on a person’s request: comparing products, checking an order, researching a supplier or preparing a booking. They may look automated in a log, but the commercial intent behind them may be very human.
This creates an awkward risk for website owners. An overly aggressive bot rule can protect infrastructure while quietly blocking a prospect’s chosen route to the business. Worse, the team may not notice until traffic, visibility, ad approvals or conversions fall.
The aim is not to open every door. It is to stop treating every machine as the same visitor.
Why the traffic spike is real
AI crawlers and browser agents have added a new layer of demand to the web. Unlike a conventional search crawler making predictable, scheduled requests, an agent may visit several sites in quick succession while carrying out a user’s task. It may load pages, follow links, compare options and revisit a page when an answer changes.
There are good reasons for site owners to be cautious. DataDome’s 2026 research, based on traffic across tens of thousands of sites, reported that bad-bot traffic grew far faster than human traffic. It also found AI-related requests increasingly reaching login pages, forms, carts and payment journeys — precisely where abuse is most costly. DataDome’s 2026 bot and agent security report makes the key point: a genuine assistant researching a purchase and an automated credential-testing attack are not the same thing. (datadome.co)
That distinction is difficult because attackers can impersonate familiar user-agent strings. A rule that trusts a request merely because it says Googlebot, ChatGPT-User or PerplexityBot is not a security control; it is an invitation to spoofing. Google explicitly advises site owners to verify its crawler through reverse DNS or published IP ranges rather than relying on the user-agent header alone. (developers.google.com)
So the security team is right to care about bot traffic. The problem starts when “automated” becomes a proxy for “unwanted”.
The important difference between crawlers and agents
Not every non-human request serves the same purpose. A practical policy begins by separating at least four types of traffic.
| Traffic type | Typical purpose | Sensible default |
|---|---|---|
| Search crawlers | Index public pages | Allow verified crawlers |
| Training crawlers | Collect material for model training | Decide by content policy |
| User agents | Research or act for a user | Permit low-risk public journeys |
| Malicious automation | Fraud, scraping or account abuse | Block or challenge |
A training crawler that downloads large volumes of editorial content creates a legitimate commercial and capacity question. A customer’s browser agent checking whether a product is in stock is different. So is a verified advertising crawler trying to validate the landing page attached to a paid campaign.
OpenAI, for example, warns advertisers that bot mitigation, CAPTCHAs, web application firewalls and rate limits can return 403 Forbidden responses to legitimate crawlers. Its guidance recommends checking every layer — robots.txt, CDN, firewall, application logic and rate limiting — rather than assuming a single setting controls access. OpenAI’s advertiser crawler guidance is a useful reminder that a blocked machine can have a direct marketing consequence. (help.openai.com)
This is an extension of a familiar customer-service lesson. Businesses have long used chatbots to help people get answers and complete straightforward tasks more quickly, as we covered in Utilising Chatbots for Customer Service. The new issue is that the assistant is increasingly on the customer’s side of the browser, not solely on the company’s website.
How blanket blocking loses business
The most obvious impact is discovery. If a site blocks legitimate search and assistant crawlers, its products, services and expertise may be less available when people ask AI-driven search tools for recommendations or comparisons.
But the risks extend beyond visibility.
A customer may be using an agent, not abandoning the purchase
A person may ask an agent to shortlist hotels, compare insurance policies, find compatible parts or check a return policy. If the agent is denied access, the user may never see the site’s offer. They are not necessarily willing to repeat the work manually in another browser.
This does not mean every agent should be allowed through checkout, account settings or payment flows. Those are high-risk areas and should retain stronger controls. It does mean that public product, pricing, availability, support and policy pages deserve a more considered policy than a universal denial.
Your own paid activity may fail validation
Ad platforms and related services need to inspect destination pages for policy, safety and relevance. If an anti-bot rule blocks verified review or search crawlers, campaigns can be delayed, disapproved or poorly understood by the system assessing them.
This is particularly frustrating because the marketing team often sees an ad problem while the cause sits in a security configuration, CDN rule or CAPTCHA introduced elsewhere.
You can unintentionally damage search access
The overlap between crawler identities and purposes is increasingly messy. Google’s documentation notes that controls for Google-Extended govern model-training and grounding choices, while Google Search continues to rely on Googlebot access. Google’s crawler documentation explains these distinctions, but they are easy to lose in a broad “AI bot” setting. (developers.google.com)
Cloudflare has also moved away from a single AI-bot switch towards separate controls for Search, Agent and Training traffic. Its 2026 configuration changes reflect the fact that site owners need purpose-based choices, not one blunt category. For new domains, some AI traffic can be blocked by default on ad-supported pages, making it especially important to review inherited or default settings rather than assume robots.txt tells the whole story. Cloudflare’s AI traffic options outline the categories and defaults. (blog.cloudflare.com)
A better policy is verify, segment and limit
The answer is not a giant allowlist. It is a layered policy that grants the minimum useful access to traffic whose identity and purpose you can establish.
Start with these steps:
- Audit what is being blocked. Review CDN, WAF, bot-management and application logs together. Look for
403,429and challenge events by user agent, source, URL path and rule ID. A robots.txt file may say “allow” while an edge rule blocks the request before it reaches the site. - Verify before you trust. Use provider-supported verification, published IP information, reverse DNS where appropriate and signed-request schemes. Do not allowlist on user-agent strings alone.
- Set rules by purpose and path. Public information pages may support verified search and research agents. Login, account recovery, basket, checkout, payment and administrative routes need stricter rate limits, session checks and fraud controls.
- Prefer throttling to permanent denial where possible. Rate limits, caching, quotas and managed challenges can protect capacity without making the site disappear to every automated visitor. Return useful
429 Too Many Requestsresponses where a client can retry responsibly. - Test the journey from outside your network. Test key URLs as verified crawlers or through your provider’s bot-verification tools. Include product pages, campaign landing pages, help content, robots.txt, XML sitemaps and structured-data endpoints.
- Give security and marketing a shared dashboard. Track blocked verified bots, challenged requests, crawl errors, ad-review failures, agent referrals and conversion performance. A security rule should be reviewed like any other change that can affect acquisition.
Block abuse, not intent
A website does not have to welcome every crawler, scraper or autonomous agent. Content rights, operating costs, fraud exposure and customer safety all matter.
But the old binary choice — humans in, bots out — no longer maps neatly to how people use the web. The machine at the door may be an attacker. It may also be a customer’s delegate, a search route, an ad validator or a service helping someone decide whether to buy from you.
In 2026, the useful question is not “Are we blocking bots?” It is: which automated visitors are we blocking, how have we verified them, and what customer or commercial journey could that decision interrupt?
