---
title: "Configure Vercel Firewall for AI Bots: Stop 429 Rate Limits at the Edge"
slug: "configure-vercel-firewall-ai-bots-edge-rate-limits"
published: "2026-09-16"
updated: "2026-09-22"
categories:
  - "Next.js"
tags:
  - "vercel firewall"
  - "vercel waf"
  - "ai bots rate limit"
  - "claudebot 429"
  - "gptbot 429"
  - "nextjs robots.ts"
  - "llms.txt vercel"
llm-intent: "how-to"
audience-level: "intermediate"
llm-purpose: "Configure Vercel Firewall and WAF for AI bots: stop 429 rate limits for Claude, ChatGPT, and Gemini on your Next.js site. Step-by-step CLI setup."
---

**Summary Triples**
- (Configure Vercel Firewall for AI Bots: Stop 429 Rate Limits at the Edge, expresses-intent, how-to)
- (Configure Vercel Firewall for AI Bots: Stop 429 Rate Limits at the Edge, covers-topic, vercel firewall)
- (Configure Vercel Firewall for AI Bots: Stop 429 Rate Limits at the Edge, provides-guidance-for, Configure Vercel Firewall and WAF for AI bots: stop 429 rate limits for Claude, ChatGPT, and Gemini on your Next.js site. Step-by-step CLI setup.)

### {GOAL}
Configure Vercel Firewall and WAF for AI bots: stop 429 rate limits for Claude, ChatGPT, and Gemini on your Next.js site. Step-by-step CLI setup.

### {PREREQS}
- Familiarity with the concepts discussed in this article.

### {STEPS}
1. Follow the detailed walkthrough in the article content below.

<!-- llm:goal="Configure Vercel Firewall and WAF for AI bots: stop 429 rate limits for Claude, ChatGPT, and Gemini on your Next.js site. Step-by-step CLI setup." -->

# Configure Vercel Firewall for AI Bots: Stop 429 Rate Limits at the Edge
> Configure Vercel Firewall and WAF for AI bots: stop 429 rate limits for Claude, ChatGPT, and Gemini on your Next.js site. Step-by-step CLI setup.
Matija Žiberna · 2026-09-16

I was testing how AI models like Claude and ChatGPT index content on my Next.js site when I hit an unexpected wall. I had carefully implemented [Content Signals in a dynamic Next.js robots.ts file](/blog/content-signals-nextjs-robots-txt), created an [llms.txt route for AI search discovery](/blog/implementing-llms-txt-nextjs-15-sanity-cms), and exposed clean markdown mirrors for my articles. Yet, whenever automated scanning tools or live chat browser agents attempted to fetch pages, they were slammed with an HTTP `429 Too Many Requests` error.

When I opened the Vercel dashboard and checked the Firewall Rate Limiting tab to see what was getting blocked, the interface displayed a confusing message: "There's no data available for your selection." 

After diving deep into Vercel's edge architecture and the Vercel CLI, I realized that application-level files like `robots.txt` never even get evaluated if Vercel's automated edge mitigations intercept requests first. In this guide, I will walk you through how Vercel's edge firewall handles automated bots, why standard allowlisting rules fail for live AI browser agents, and how to configure edge bypass rules so AI tools can read your public content cleanly without compromising sensitive routes.

---

## Understanding Why AI Bots Hit HTTP 429 at the Edge

Before writing any configuration rules, it helps to understand the execution hierarchy of the Vercel edge network. When an incoming request reaches your domain, it does not immediately execute your Next.js route handlers. Instead, it passes through three distinct layers in strict sequential order.

The first layer is Platform System Mitigations. This is an always-on, automated DDoS and bot protection layer managed by Vercel. It tracks volumetric request patterns across cloud provider IP pools. If an automated script or bot from AWS or Google Cloud sends rapid bursts of requests, this layer intercepts the traffic and returns an HTTP 429 or a JavaScript challenge before any custom rule is evaluated. This also explains why the Rate Limiting tab in the dashboard displays no data. That dashboard tab only logs custom user-created rate-limiting rules, not platform-level mitigations.

The second layer consists of static IP blocks, and the third layer runs your Custom WAF Rules.

This creates a major problem for AI search engines and browsing assistants. When a user asks Claude or ChatGPT to summarize an article by URL, the request does not come from a traditional web crawler. Background crawlers like `ClaudeBot` or `GPTBot` operate on scheduled indexing runs. Live user browsing tools, however, run on ephemeral cloud instances with generic or rotating headers. Because these requests originate from shared cloud hosting subnets, Vercel's system mitigations treat them as suspicious automation and throttle them immediately.

To solve this, we need to create explicit bypass rules in our custom firewall configuration and place them high enough in the execution chain to exempt trusted traffic from edge scrutiny.

---

## Step 1: Defining Your Application Crawler Baseline in Next.js

Our first implementation step is to ensure our Next.js application explicitly declares which automated agents are welcomed and what content signals we expect them to follow. 

In the Next.js App Router, we handle this by generating our `robots.txt` programmatically using a dedicated TypeScript route file.

```typescript
// File: src/app/robots.ts
import { MetadataRoute } from 'next'

const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://www.buildwithmatija.com'

const adminPaths = ['/studio', '/api/', '/admin', '/dashboard', '/wp-admin']

const llmAgents = [
  'GPTBot',
  'ChatGPT-User',
  'Claude-Web',
  'ClaudeBot',
  'anthropic-ai',
  'PerplexityBot',
  'Applebot-Extended',
  'Google-Extended',
]

export default function robots(): MetadataRoute.Robots {
  return {
    sitemap: `${baseUrl}/sitemap.xml`,
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: adminPaths,
        other: {
          'Content-Signal': 'ai-train=no, search=yes, ai-input=no',
        },
      },
      ...llmAgents.map((userAgent) => ({
        userAgent,
        allow: '/',
        disallow: adminPaths,
        other: {
          'Content-Signal': 'ai-train=no, search=yes, ai-input=no',
        },
      })),
    ],
  }
}
```

This route exports a default `robots` function adhering to Next.js metadata specifications. It explicitly identifies primary AI agents, separates administrative paths from public content, and attaches modern `Content-Signal` headers indicating that search and citation are welcome while raw training is restricted.

While this file establishes proper crawler governance at the application level, it cannot protect requests from being throttled at Vercel's CDN edge. To further protect origin compute and bandwidth as automated scrapers query your content, pairing edge bypass rules with strategies to [reduce Vercel Fast Origin Transfer using ISR](/blog/reduce-vercel-fast-origin-transfer-isr-nextjs) ensures that crawler bursts do not spike infrastructure invoices. Now that our application baseline is defined, we must configure Vercel's edge firewall to mirror these permissions.

---

## Step 2: Inspecting Active Firewall Rules via the Vercel CLI

Rather than clicking through multiple screens in the Vercel dashboard, managing firewall configurations via the Vercel CLI provides complete visibility into rule priority, status, and staged drafts.

Open your terminal in your project root and inspect the live status of your firewall configuration:

```bash
# File: terminal
npx vercel firewall overview --non-interactive
```

When you execute this command, the CLI outputs a comprehensive summary of your edge environment:

```text
Firewall:             Enabled
Custom Rules:         Active
IP Blocks:            0
System Bypass:        0 IPs
Attack Mode:          Off
System Mitigations:   Active
```

Notice the `System Mitigations: Active` line. This confirms that Vercel's automated edge traffic scrubbing is actively filtering requests before they reach your code. Next, list your existing custom rules to inspect their execution order:

```bash
# File: terminal
npx vercel firewall rules list --non-interactive
```

In Vercel WAF, rules execute sequentially from index 1 to N. When an incoming request matches a rule with a `Bypass` action, rule processing halts immediately, and the request is permitted through to the application without being subjected to any subsequent custom rate-limiting rules.

Now that we know how our rules are evaluated, we can begin adding targeted bypass rules.

---

## Step 3: Configuring Broad Substring Bypass Rules for AI User Agents

A common mistake when configuring firewall rules is creating an allowlist rule only for `ClaudeBot` or `GPTBot`. As established earlier, real-time browsing requests from Claude Chat or ChatGPT Search often use identifiers like `Claude-Web`, `anthropic-ai`, or `ChatGPT-User`.

Vercel WAF rule conditions use a `sub` (substring / contains) operator that evaluates strings in a case-insensitive manner. By targeting the root vendor names, a single rule will cover every crawler, search bot, and user-facing tool operated by that provider.

Run the following commands in your terminal to stage bypass rules for Claude, OpenAI, Google, and Bing:

```bash
# File: terminal
# Stage bypass for all Claude and Anthropic agents
npx vercel firewall rules add "Rule for User Agent: Claude" \
  --condition '{"type":"user_agent","op":"sub","value":"Claude"}' \
  --action bypass --yes --non-interactive

npx vercel firewall rules add "Rule for User Agent: Anthropic" \
  --condition '{"type":"user_agent","op":"sub","value":"Anthropic"}' \
  --action bypass --yes --non-interactive

# Stage bypass for all OpenAI and ChatGPT agents
npx vercel firewall rules add "Rule for User Agent: OpenAI" \
  --condition '{"type":"user_agent","op":"sub","value":"OpenAI"}' \
  --action bypass --yes --non-interactive

npx vercel firewall rules add "Rule for User Agent: ChatGPT" \
  --condition '{"type":"user_agent","op":"sub","value":"ChatGPT"}' \
  --action bypass --yes --non-interactive

# Stage bypass for Googlebot and Gemini
npx vercel firewall rules add "Rule for User Agent: Google" \
  --condition '{"type":"user_agent","op":"sub","value":"Google"}' \
  --action bypass --yes --non-interactive

npx vercel firewall rules add "Rule for User Agent: Gemini" \
  --condition '{"type":"user_agent","op":"sub","value":"Gemini"}' \
  --action bypass --yes --non-interactive
```

These commands stage six discrete custom rules. Each rule instructs Vercel's edge to check the incoming `User-Agent` header. If the string contains the target term anywhere in the header, Vercel grants a full bypass. 

However, relying strictly on user-agent strings is only half the battle. Some automated tools and cloud scanners do not transmit recognizable bot headers. To ensure our reading surfaces remain completely accessible, we must also apply path-based bypasses.

---

## Step 4: Whitelisting Content Paths Across All Locales

On a modern content-driven Next.js site, public blog articles, markdown endpoints, and hub pages are static or incrementally regenerated. Serving these cached pages from Vercel's edge cache consumes negligible server compute. Rate limiting these routes provides very little security benefit while frequently penalizing legitimate AI readers.

We will create path-based bypass rules covering our homepage, all blog routes regardless of locale, and our primary content hubs.

```bash
# File: terminal
# Stage bypass for all blog content (HTML and Markdown, all locales)
npx vercel firewall rules add "Rule for Path: All Blog Pages" \
  --condition '{"type":"path","op":"sub","value":"/blog"}' \
  --action bypass --yes --non-interactive

# Stage bypass for the homepage across localized roots
npx vercel firewall rules add "Rule for Path: Homepage" \
  --condition '{"type":"path","op":"eq","value":"/"}' \
  --or --condition '{"type":"path","op":"eq","value":"/de"}' \
  --or --condition '{"type":"path","op":"eq","value":"/sl"}' \
  --or --condition '{"type":"path","op":"eq","value":"/en"}' \
  --action bypass --yes --non-interactive

# Stage bypass for core hubs and discovery routes
npx vercel firewall rules add "Rule for Path: Content Hubs" \
  --condition '{"type":"path","op":"pre","value":"/llms"}' \
  --or --condition '{"type":"path","op":"pre","value":"/cms"}' \
  --or --condition '{"type":"path","op":"pre","value":"/topics"}' \
  --or --condition '{"type":"path","op":"pre","value":"/insights"}' \
  --action bypass --yes --non-interactive
```

The first command evaluates whether the request path contains `/blog`. Because it uses the substring operator, it matches `/blog`, `/blog/my-post`, `/blog/md/my-post`, and localized paths such as `/de/blog/my-post` in a single rule.

The second command utilizes the `--or` argument to group homepage entry points into one logical rule. The third command targets discovery endpoints like `/llms.txt` and core topic hubs using the prefix (`pre`) operator.

With these rules in place, our public content paths are exempt from custom rate limits, allowing our sensitive endpoints (like `/api/auth` or form handlers) to remain strictly protected.

---

## Step 5: Reviewing Staged Changes and Publishing to Production

One crucial characteristic of the Vercel Firewall system is that creating rules only stages them in draft mode. A staged rule has zero impact on live edge traffic until you explicitly publish it.

Before deploying, run the diff command to verify the staged changes:

```bash
# File: terminal
npx vercel firewall diff --non-interactive
```

Review the pending additions in the output:

```text
Pending changes (9):
  + Added rule "Rule for User Agent: Claude"
  + Added rule "Rule for User Agent: Anthropic"
  + Added rule "Rule for User Agent: OpenAI"
  + Added rule "Rule for User Agent: ChatGPT"
  + Added rule "Rule for User Agent: Google"
  + Added rule "Rule for User Agent: Gemini"
  + Added rule "Rule for Path: All Blog Pages"
  + Added rule "Rule for Path: Homepage"
  + Added rule "Rule for Path: Content Hubs"
```

Once confirmed, deploy the staged rules live to production by running the publish command:

```bash
# File: terminal
npx vercel firewall publish --non-interactive --yes
```

The CLI pushes the changes to Vercel's global edge nodes within hundreds of milliseconds. Now, we need to verify our live endpoints against real bot user-agents.

---

## Step 6: Verifying Edge Access with Real Bot Signatures

To verify that your bypass rules are operating correctly and that Vercel is returning clean HTTP 200 responses, send test requests through `curl` using representative user-agent headers.

Run the following commands against your live production domain:

```bash
# File: terminal
# Test using Claude live browser agent signature
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude/1.0" \
  https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation

# Test using Gemini crawler signature
curl -s -o /dev/null -w "%{http_code}\n" -A "Gemini/2.5" \
  https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation

# Test using ChatGPT live user browsing signature
curl -s -o /dev/null -w "%{http_code}\n" -A "ChatGPT-User/1.0" \
  https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation
```

Each command should return an immediate `200` status code. You can also verify that the full HTML or raw markdown content is delivered properly without edge interception:

```bash
# File: terminal
curl -ILs https://www.buildwithmatija.com/blog/multi-tenant-cms-reduce-website-fragmentation | grep -E "HTTP/|content-type|x-vercel-id"
```

The response headers will confirm that the request was processed directly by Vercel without challenge screens or rate-limit restrictions:

```text
HTTP/2 200 
content-type: text/html; charset=utf-8
x-vercel-id: arn1::iad1::xdnvz-1789566490335-b16e8d61a905
```

If an automated scanner ever runs an intensive security audit and triggers platform-level DDoS scrubbing regardless of rules, remember that you can temporarily pause automatic system mitigations for 24 hours using the CLI:

```bash
# File: terminal
vercel firewall system-mitigations pause
```

This provides a clean window for testing before you re-enable protection with `vercel firewall system-mitigations resume`.

---

## Conclusion

In this guide, we resolved the disconnect between application-level crawler directives and edge firewall rate limits on Vercel. We looked at how Vercel's multi-tier firewall processes requests, why live AI browsing agents get blocked by platform system mitigations even when standard crawlers are permitted, and why dashboard analytics often show zero rate-limiting events.

By combining a structured `robots.ts` configuration with targeted substring bypass rules in Vercel WAF, you can give Claude, ChatGPT, Gemini, and search crawlers completely unhindered access to your blog, homepage, and markdown endpoints while preserving strict rate limits on your sensitive API routes.

---

## Related Reading & Edge Infrastructure

For more on edge caching, Next.js metadata routes, and AI discovery:

- **[How to Implement Content Signals in Next.js 16 (robots.ts)](/blog/content-signals-nextjs-robots-txt)** — Declaring AI crawler permissions with Next.js 16's `other` metadata map.
- **[Implementing llms.txt in Next.js 15 & Sanity CMS](/blog/implementing-llms-txt-nextjs-15-sanity-cms)** — Building a structured content manifest for AI search models and LLM scrapers.
- **[How to Reduce Vercel Fast Origin Traffic by 95% Using ISR](/blog/reduce-vercel-fast-origin-transfer-isr-nextjs)** — Shifting compute from dynamic render to edge-cached ISR to slash transfer costs.
- **[Dynamic robots.txt in Multi-Tenant Next.js](/blog/dynamic-robots-txt-nextjs-multi-tenant)** — Handling domain-specific crawler directives across multi-tenant deployments.

Let me know in the comments if you have questions, and subscribe for more practical development guides.

Thanks, Matija