Allow Claude, ChatGPT, and Gemini to crawl your Next.js site without triggering edge 429 rate limits or exposing…
In-depth Next.js guides covering App Router, RSC, ISR, and deployment. Get code examples, optimization checklists, and prompts to accelerate development.
No spam. Unsubscribe anytime.
I was testing how AI models like Claude and ChatGPT index content on my Next.js site when I hit an unexpected wall. I had carefully implemented Content Signals in a dynamic Next.js robots.ts file, created an llms.txt route for AI search discovery, and exposed clean markdown mirrors for my articles. Yet, whenever automated scanning tools or live chat browser agents attempted to fetch pages, they were slammed with an HTTP 429 Too Many Requests error.
When I opened the Vercel dashboard and checked the Firewall Rate Limiting tab to see what was getting blocked, the interface displayed a confusing message: "There's no data available for your selection."
After diving deep into Vercel's edge architecture and the Vercel CLI, I realized that application-level files like robots.txt never even get evaluated if Vercel's automated edge mitigations intercept requests first. In this guide, I will walk you through how Vercel's edge firewall handles automated bots, why standard allowlisting rules fail for live AI browser agents, and how to configure edge bypass rules so AI tools can read your public content cleanly without compromising sensitive routes.
Before writing any configuration rules, it helps to understand the execution hierarchy of the Vercel edge network. When an incoming request reaches your domain, it does not immediately execute your Next.js route handlers. Instead, it passes through three distinct layers in strict sequential order.
The first layer is Platform System Mitigations. This is an always-on, automated DDoS and bot protection layer managed by Vercel. It tracks volumetric request patterns across cloud provider IP pools. If an automated script or bot from AWS or Google Cloud sends rapid bursts of requests, this layer intercepts the traffic and returns an HTTP 429 or a JavaScript challenge before any custom rule is evaluated. This also explains why the Rate Limiting tab in the dashboard displays no data. That dashboard tab only logs custom user-created rate-limiting rules, not platform-level mitigations.
The second layer consists of static IP blocks, and the third layer runs your Custom WAF Rules.
This creates a major problem for AI search engines and browsing assistants. When a user asks Claude or ChatGPT to summarize an article by URL, the request does not come from a traditional web crawler. Background crawlers like ClaudeBot or GPTBot operate on scheduled indexing runs. Live user browsing tools, however, run on ephemeral cloud instances with generic or rotating headers. Because these requests originate from shared cloud hosting subnets, Vercel's system mitigations treat them as suspicious automation and throttle them immediately.
To solve this, we need to create explicit bypass rules in our custom firewall configuration and place them high enough in the execution chain to exempt trusted traffic from edge scrutiny.
Our first implementation step is to ensure our Next.js application explicitly declares which automated agents are welcomed and what content signals we expect them to follow.
In the Next.js App Router, we handle this by generating our robots.txt programmatically using a dedicated TypeScript route file.
// File: src/app/robots.ts
import { MetadataRoute } from 'next'
const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://www.buildwithmatija.com'
const adminPaths = ['/studio', '/api/', '/admin', '/dashboard', '/wp-admin']
const llmAgents = [
'GPTBot',
'ChatGPT-User',
'Claude-Web',
'ClaudeBot',
'anthropic-ai',
'PerplexityBot',
'Applebot-Extended',
'Google-Extended',
]
export default function robots(): MetadataRoute.Robots {
return {
sitemap: `${baseUrl}/sitemap.xml`,
rules: [
{
userAgent: '*',
allow: '/',
disallow: adminPaths,
other: {
'Content-Signal': 'ai-train=no, search=yes, ai-input=no',
},
},
...llmAgents.map((userAgent) => ({
userAgent,
allow: '/',
disallow: adminPaths,
other: {
'Content-Signal': 'ai-train=no, search=yes, ai-input=no',
},
})),
],
}
}
This route exports a default robots function adhering to Next.js metadata specifications. It explicitly identifies primary AI agents, separates administrative paths from public content, and attaches modern Content-Signal headers indicating that search and citation are welcome while raw training is restricted.
While this file establishes proper crawler governance at the application level, it cannot protect requests from being throttled at Vercel's CDN edge. To further protect origin compute and bandwidth as automated scrapers query your content, pairing edge bypass rules with strategies to reduce Vercel Fast Origin Transfer using ISR ensures that crawler bursts do not spike infrastructure invoices. Now that our application baseline is defined, we must configure Vercel's edge firewall to mirror these permissions.
Rather than clicking through multiple screens in the Vercel dashboard, managing firewall configurations via the Vercel CLI provides complete visibility into rule priority, status, and staged drafts.
Open your terminal in your project root and inspect the live status of your firewall configuration:
# File: terminal
npx vercel firewall overview --non-interactive
When you execute this command, the CLI outputs a comprehensive summary of your edge environment:
Firewall: Enabled
Custom Rules: Active
IP Blocks: 0
System Bypass: 0 IPs
Attack Mode: Off
System Mitigations: Active
Notice the System Mitigations: Active line. This confirms that Vercel's automated edge traffic scrubbing is actively filtering requests before they reach your code. Next, list your existing custom rules to inspect their execution order:
# File: terminal
npx vercel firewall rules list --non-interactive
In Vercel WAF, rules execute sequentially from index 1 to N. When an incoming request matches a rule with a Bypass action, rule processing halts immediately, and the request is permitted through to the application without being subjected to any subsequent custom rate-limiting rules.
Now that we know how our rules are evaluated, we can begin adding targeted bypass rules.
A common mistake when configuring firewall rules is creating an allowlist rule only for ClaudeBot or GPTBot. As established earlier, real-time browsing requests from Claude Chat or ChatGPT Search often use identifiers like Claude-Web, anthropic-ai, or ChatGPT-User.
Vercel WAF rule conditions use a sub (substring / contains) operator that evaluates strings in a case-insensitive manner. By targeting the root vendor names, a single rule will cover every crawler, search bot, and user-facing tool operated by that provider.
Run the following commands in your terminal to stage bypass rules for Claude, OpenAI, Google, and Bing:
# File: terminal
# Stage bypass for all Claude and Anthropic agents
npx vercel firewall rules add "Rule for User Agent: Claude" \
--condition '{"type":"user_agent","op":"sub","value":"Claude"}' \
--action bypass --yes --non-interactive
npx vercel firewall rules add "Rule for User Agent: Anthropic" \
--condition '{"type":"user_agent","op":"sub","value":"Anthropic"}' \
--action bypass --yes --non-interactive
# Stage bypass for all OpenAI and ChatGPT agents
npx vercel firewall rules add "Rule for User Agent: OpenAI" \
--condition '{"type":"user_agent","op":"sub","value":"OpenAI"}' \
--action bypass --yes --non-interactive
npx vercel firewall rules add "Rule for User Agent: ChatGPT" \
--condition '{"type":"user_agent","op":"sub","value":"ChatGPT"}' \
--action bypass --yes --non-interactive
# Stage bypass for Googlebot and Gemini
npx vercel firewall rules add "Rule for User Agent: Google" \
--condition '{"type":"user_agent","op":"sub","value":"Google"}' \
--action bypass --yes --non-interactive
npx vercel firewall rules add "Rule for User Agent: Gemini" \
--condition '{"type":"user_agent","op":"sub","value":"Gemini"}' \
--action bypass --yes --non-interactive
These commands stage six discrete custom rules. Each rule instructs Vercel's edge to check the incoming User-Agent header. If the string contains the target term anywhere in the header, Vercel grants a full bypass.
However, relying strictly on user-agent strings is only half the battle. Some automated tools and cloud scanners do not transmit recognizable bot headers. To ensure our reading surfaces remain completely accessible, we must also apply path-based bypasses.
On a modern content-driven Next.js site, public blog articles, markdown endpoints, and hub pages are static or incrementally regenerated. Serving these cached pages from Vercel's edge cache consumes negligible server compute. Rate limiting these routes provides very little security benefit while frequently penalizing legitimate AI readers.
We will create path-based bypass rules covering our homepage, all blog routes regardless of locale, and our primary content hubs.
# File: terminal
# Stage bypass for all blog content (HTML and Markdown, all locales)
npx vercel firewall rules add "Rule for Path: All Blog Pages" \
--condition '{"type":"path","op":"sub","value":"/blog"}' \
--action bypass --yes --non-interactive
# Stage bypass for the homepage across localized roots
npx vercel firewall rules add "Rule for Path: Homepage" \
--condition '{"type":"path","op":"eq","value":"/"}' \
--or --condition '{"type":"path","op":"eq","value":"/de"}' \
--or --condition '{"type":"path","op":"eq","value":"/sl"}' \
--or --condition '{"type":"path","op":"eq","value":"/en"}' \
--action bypass --yes --non-interactive
# Stage bypass for core hubs and discovery routes
npx vercel firewall rules add "Rule for Path: Content Hubs" \
--condition '{"type":"path","op":"pre","value":"/llms"}' \
--or --condition '{"type":"path","op":"pre","value":"/cms"}' \
--or --condition '{"type":"path","op":"pre","value":"/topics"}' \
--or --condition '{"type":"path","op":"pre","value":"/insights"}' \
--action bypass --yes --non-interactive
The first command evaluates whether the request path contains /blog. Because it uses the substring operator, it matches /blog, /blog/my-post, /blog/md/my-post, and localized paths such as /de/blog/my-post in a single rule.
The second command utilizes the --or argument to group homepage entry points into one logical rule. The third command targets discovery endpoints like /llms.txt and core topic hubs using the prefix (pre) operator.
With these rules in place, our public content paths are exempt from custom rate limits, allowing our sensitive endpoints (like /api/auth or form handlers) to remain strictly protected.
One crucial characteristic of the Vercel Firewall system is that creating rules only stages them in draft mode. A staged rule has zero impact on live edge traffic until you explicitly publish it.
Before deploying, run the diff command to verify the staged changes:
# File: terminal
npx vercel firewall diff --non-interactive
Review the pending additions in the output:
Pending changes (9):
+ Added rule "Rule for User Agent: Claude"
+ Added rule "Rule for User Agent: Anthropic"
+ Added rule "Rule for User Agent: OpenAI"
+ Added rule "Rule for User Agent: ChatGPT"
+ Added rule "Rule for User Agent: Google"
+ Added rule "Rule for User Agent: Gemini"
+ Added rule "Rule for Path: All Blog Pages"
+ Added rule "Rule for Path: Homepage"
+ Added rule "Rule for Path: Content Hubs"
Once confirmed, deploy the staged rules live to production by running the publish command:
# File: terminal
npx vercel firewall publish --non-interactive --yes
The CLI pushes the changes to Vercel's global edge nodes within hundreds of milliseconds. Now, we need to verify our live endpoints against real bot user-agents.
To verify that your bypass rules are operating correctly and that Vercel is returning clean HTTP 200 responses, send test requests through curl using representative user-agent headers.
Run the following commands against your live production domain:
# File: terminal
# Test using Claude live browser agent signature
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude/1.0" \
https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation
# Test using Gemini crawler signature
curl -s -o /dev/null -w "%{http_code}\n" -A "Gemini/2.5" \
https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation
# Test using ChatGPT live user browsing signature
curl -s -o /dev/null -w "%{http_code}\n" -A "ChatGPT-User/1.0" \
https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation
Each command should return an immediate 200 status code. You can also verify that the full HTML or raw markdown content is delivered properly without edge interception:
# File: terminal
curl -ILs https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation | grep -E "HTTP/|content-type|x-vercel-id"
The response headers will confirm that the request was processed directly by Vercel without challenge screens or rate-limit restrictions:
HTTP/2 200
content-type: text/html; charset=utf-8
x-vercel-id: arn1::iad1::xdnvz-1789566490335-b16e8d61a905
If an automated scanner ever runs an intensive security audit and triggers platform-level DDoS scrubbing regardless of rules, remember that you can temporarily pause automatic system mitigations for 24 hours using the CLI:
# File: terminal
vercel firewall system-mitigations pause
This provides a clean window for testing before you re-enable protection with vercel firewall system-mitigations resume.
In this guide, we resolved the disconnect between application-level crawler directives and edge firewall rate limits on Vercel. We looked at how Vercel's multi-tier firewall processes requests, why live AI browsing agents get blocked by platform system mitigations even when standard crawlers are permitted, and why dashboard analytics often show zero rate-limiting events.
By combining a structured robots.ts configuration with targeted substring bypass rules in Vercel WAF, you can give Claude, ChatGPT, Gemini, and search crawlers completely unhindered access to your blog, homepage, and markdown endpoints while preserving strict rate limits on your sensitive API routes.
For more on edge caching, Next.js metadata routes, and AI discovery:
other metadata map.Let me know in the comments if you have questions, and subscribe for more practical development guides.
Thanks, Matija