wiseseo:~/news/allow-google-agent-waf-cdn $

date
category
[CRAWLING]
impact
MED
status
OFFICIAL DOC
read
4 min

Google publishes a guide to stop your WAF blocking Google-Agent

Google's new crawling doc explains how to verify and allow Google-Agent, its user-triggered browsing agent, through CDNs, WAFs and bot tools.

On 17 November 2026, Google added a page called Allow Google-Agent traffic to its crawling infrastructure documentation. The changelog entry for that day describes it as a technical guide on making sure Google-Agent can reach your website, written for publishers that want to enable it. Search Engine Roundtable flagged the page the same day.

Google-Agent itself is not new. It already sits in Google's list of user-triggered fetchers, where Google describes it as the name used by agents hosted on Google infrastructure to navigate the web and perform actions when a user asks them to. What is new is a step-by-step explanation of how to let it through security layers that were built to stop bots.

What the guide says

Google gives three ways to recognise a genuine Google-Agent request:

  • The User-Agent header contains Google-Agent. The fetcher list shows separate mobile and desktop strings, both Chrome-based.
  • The request comes from an IP address listed in user-triggered-agents.json. That file is separate from the Googlebot ranges. When we fetched it, its creationTime read 18 November 2026, so the list is being refreshed.
  • Optionally, a cryptographic signature. Google says it is experimenting with the Web Bot Auth protocol under the identity https://agent.bot.goog, signing requests with HTTP Message Signatures (RFC 9421). Participating requests carry a Signature-Agent: g="https://agent.bot.goog" header plus Signature and Signature-Input headers labelled g. The public keys are published at https://agent.bot.goog/.well-known/http-message-signatures-directory.

For requests that pass those checks, the guide's instruction is to skip bot challenges. Google notes that some CDNs, firewalls and bot-detection services can already verify Google-Agent, and points you to your provider for the exact setting.

The troubleshooting section is the most useful part for technical teams. It lists the places where an allow rule tends to fail:

  • Rule order. The allow rule has to run before broad cloud or data-centre ASN blocks, managed bot rules and category-level "AI agent" block policies.
  • Interactive challenges. Skipping HTTP blocks is not enough. CAPTCHAs, JavaScript challenges and in-page CAPTCHA widgets on search or form submission also need to be skipped.
  • The whole journey. Allowing the homepage is not enough. On-site search, filters, detail pages, add-to-cart, booking and the XHR, GraphQL or api.* endpoints behind them all need to be covered.
  • Shared IPs. Many users' agent sessions share the same egress pool, so per-IP rate limits can throttle legitimate traffic.
  • Stale ranges. IP-based rules need to stay in sync with the published JSON.
  • Proxies. Load balancers must preserve the client IP, the User-Agent and the signature headers.

Why it matters

Google-Agent is listed as a user-triggered fetcher. Google's fetcher documentation says fetchers in this group generally ignore robots.txt because a user requested the fetch. In practice, robots.txt is not how you control this traffic. Your CDN, WAF and bot-management rules are. Many sites now block "AI agents" as a category, or challenge anything from cloud IP ranges, and both would catch Google-Agent.

It landed the same day Google made Tasks in Gemini free in six countries, which we covered in Gemini Spark is now Tasks, and it's free. Google has not said in the new doc which products send Google-Agent traffic, so do not assume every Gemini task arrives under this name.

Confirmed vs. not confirmed

Confirmed: the guide, the identification methods, the IP file and the Web Bot Auth experiment, all in Google's own documentation. Not confirmed: whether allowing or blocking Google-Agent has any effect on Google Search rankings. The guide is about access for agents, not about ranking, and Google does not mention Search ranking at all. Whether to allow it is a business decision. Google frames the page as help for publishers that want to enable Google-Agent.

what to do

  • Decide whether you want agents completing actions on your site. If you do, ask your CDN or bot-management provider whether it already verifies Google-Agent.
  • If not, build an allow rule on User-Agent plus IP match against user-triggered-agents.json, and add signature checks if your stack supports Web Bot Auth.
  • Put that rule above ASN, managed-bot and "AI agent" category rules, and make it skip CAPTCHAs and JavaScript challenges.
  • Cover search, filters, cart, booking and API endpoints, not just page loads. Review per-IP rate limits for the shared egress pool.
  • Automate syncing of the IP ranges instead of hard-coding them.
  • Check server logs for requests containing Google-Agent and confirm they get 200 responses, not 403 or 429 errors or challenge pages.
  • After changing firewall rules, make sure Googlebot is unaffected. Run a URL Inspection live test on key templates in Search Console and watch the host status in the Crawl stats report for new availability problems. For how status codes during downtime affect crawling, see why serving a 200 during an outage backfires.

refs

  1. [1] Allow Google-Agent traffic (Google Crawling infrastructure docs)https://developers.google.com/crawling/docs/crawlers-fetchers/allow-google-agent
  2. [2] Crawling infrastructure documentation changelog, 17 November 2026https://developers.google.com/crawling/docs/changelog
  3. [3] List of Google user-triggered fetchershttps://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
  4. [4] user-triggered-agents.json (Google IP ranges)https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json
  5. [5] New Google Search Help Document On How To Allow Google-Agent Traffic (Search Engine Roundtable)https://www.seroundtable.com/google-agent-bot-42559.html