---
title: "How to Implement Content Signals in Next.js 16 (robots.ts)"
slug: "content-signals-nextjs-robots-txt"
published: "2026-09-16"
updated: "2026-09-23"
categories:
  - "Next.js"
tags:
  - "content signals"
  - "nextjs robots.ts"
  - "nextjs 16.3"
  - "ai crawler directives"
  - "robots.txt ai-train"
  - "ai-input"
  - "cloudflare content signals"
  - "metadata route robots"
  - "non-standard robots directives"
  - "next.js app router"
llm-intent: "how-to"
audience-level: "intermediate"
llm-purpose: "Implement Content Signals in Next.js 16: use the new other field in app/robots.ts to declare ai-train, search, and ai-input preferences for automated AI…"
llm-prereqs:
  - "Next.js"
  - "TypeScript"
---

**Summary Triples**
- (How to Implement Content Signals in Next.js 16 (robots.ts), expresses-intent, how-to)
- (How to Implement Content Signals in Next.js 16 (robots.ts), covers-topic, content signals)
- (How to Implement Content Signals in Next.js 16 (robots.ts), provides-guidance-for, Implement Content Signals in Next.js 16: use the new other field in app/robots.ts to declare ai-train, search, and ai-input preferences for automated AI…)

### {GOAL}
Implement Content Signals in Next.js 16: use the new other field in app/robots.ts to declare ai-train, search, and ai-input preferences for automated AI…

### {PREREQS}
- Next.js
- TypeScript

### {STEPS}
1. Follow the detailed walkthrough in the article content below.

<!-- llm:goal="Implement Content Signals in Next.js 16: use the new other field in app/robots.ts to declare ai-train, search, and ai-input preferences for automated AI…" -->
<!-- llm:prereq="Next.js" -->
<!-- llm:prereq="TypeScript" -->

# How to Implement Content Signals in Next.js 16 (robots.ts)
> Implement Content Signals in Next.js 16: use the new other field in app/robots.ts to declare ai-train, search, and ai-input preferences for automated AI…
Matija Žiberna · 2026-09-16

I was recently looking through my Cloudflare dashboard, checking their new AI Overview and scraper analytics, when I noticed something glaring: my production site had no signal preferences configured at all. While bots and AI crawlers were routinely visiting my content, Cloudflare's report showed blank preferences because standard `robots.txt` offers only binary choices—either you allow an automated crawler completely, or you lock them out entirely.

With AI crawlers indexing content for live search answers, RAG systems, and foundation model training, site owners need a way to declare nuanced, granular permissions without blocking traditional search discovery.

When I tried to configure the emerging Content Signals standard in Next.js, I hit an immediate framework limitation: older versions of `robots.ts` stripped non-standard directives from output. With Next.js 16.3, the framework introduced an official escape hatch for non-standard directives. In this guide, I will show you how to implement Content Signals directly inside `app/robots.ts` using the new `other` directive map, declaring clear preferences for search indexing, live AI answers, and model training.

---

## Understanding Content Signals Directives

Before writing configuration code, we need to understand how Content Signals convey publisher preferences to automated crawlers. Originating from an IETF internet draft and popularized by Cloudflare, Content Signals establish three distinct categories of content usage:

1. `search`: Building a search engine index and displaying links and short snippets in search results.
2. `ai-input`: Passing website content into an AI system in real time, such as Retrieval-Augmented Generation (RAG) queries, grounded chat answers, or AI search summaries.
3. `ai-train`: Ingesting content to train, pre-train, or fine-tune artificial intelligence foundation models.

Each category accepts a simple `yes` or `no` value. A directive string combines them in comma-delimited pairs, like this:

```txt
Content-Signal: ai-train=yes, search=yes, ai-input=yes
```

Depending on your business goals, you can disallow model training while remaining discoverable in AI search engines, or you can allow full access across all three categories to maximize your presence across AI assistants and web crawlers.

---

## Step 1: Upgrading Next.js to 16.3 or Newer

Next.js added support for custom directives on crawler rules in version 16.3.0. In older releases, the `MetadataRoute.Robots` TypeScript interface and the underlying string generator only handled `userAgent`, `allow`, `disallow`, and `crawlDelay`. Any additional keys passed to the rule object were quietly dropped during serialization.

Check your current Next.js version in `package.json` and ensure it is updated to at least `16.3.0`:

```json
// File: package.json
{
  "dependencies": {
    "next": "^16.3.5"
  }
}
```

If you need to update your installation, run your package manager update command:

```bash
pnpm add next@latest
```

This ensures your project receives the updated `MetadataRoute.Robots` types, which include the `other` directive record:

```typescript
type RobotsRule = {
  userAgent?: string | string[]
  allow?: string | string[]
  disallow?: string | string[]
  crawlDelay?: number
  other?: Record<string, string | number | Array<string | number>>
}
```

With the package upgraded, Next.js will preserve the exact casing of keys declared inside `other` and render each key-value pair directly beneath the corresponding `User-agent` block.

---

## Step 2: Configuring Content Signals in `app/robots.ts`

Now we can implement our rules inside `src/app/robots.ts`. We want our signals to apply both to generic crawlers via the wildcard `*` agent and to specific, high-profile AI crawlers such as `GPTBot`, `ClaudeBot`, and `PerplexityBot`.

Here is the complete implementation:

```typescript
// File: src/app/robots.ts
import { MetadataRoute } from 'next'
import { siteOrigin } from '@/lib/site-url'

const baseUrl = siteOrigin

const adminPaths = ['/studio', '/api/', '/admin', '/dashboard', '/wp-admin']

const llmAgents = [
  'GPTBot',
  'ChatGPT-User',
  'Claude-Web',
  'ClaudeBot',
  'anthropic-ai',
  'PerplexityBot',
  'CCBot',
  'Omgilibot',
  'FacebookBot',
  'Google-Extended',
  'cohere-ai',
  'Applebot-Extended',
  'Bytespider',
  'Diffbot',
]

export default function robots(): MetadataRoute.Robots {
  const contentSignalPolicy = 'ai-train=yes, search=yes, ai-input=yes'

  return {
    sitemap: `${baseUrl}/sitemap.xml`,
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: adminPaths,
        other: {
          'Content-Signal': contentSignalPolicy,
        },
      },
      ...llmAgents.map((userAgent) => ({
        userAgent,
        allow: '/',
        disallow: adminPaths,
        other: {
          'Content-Signal': contentSignalPolicy,
        },
      })),
    ],
  }
}
```

In this implementation, we define an array of restricted administrative paths and a list of recognized LLM crawler identifiers. We define our `contentSignalPolicy` variable according to our preferred permissions. In this example, we set `ai-train=yes, search=yes, ai-input=yes` to invite search engines, citation systems, and AI tools to index and cite our work freely.

Next, we map our rules array. The first rule targets the global wildcard `*`. The subsequent rules iterate through our list of specific AI user-agents. Inside each rule, we supply the `other` dictionary with our `'Content-Signal'` header.

Because robots.txt parser rules state that crawlers matching a specific user-agent header disregard the generic wildcard block, adding `other` to each explicit AI bot entry ensures the directive is observed regardless of which block the crawler matches.

---

## Step 3: Verifying the Generated `robots.txt` Output

Next.js automatically serves the return value of `app/robots.ts` at the `/robots.txt` path. You can verify the final serialized text by running your development server or requesting the route directly in your browser.

The generated output will look like this:

```txt
User-Agent: *
Allow: /
Disallow: /studio
Disallow: /api/
Disallow: /admin
Disallow: /dashboard
Disallow: /wp-admin
Content-Signal: ai-train=yes, search=yes, ai-input=yes

User-Agent: GPTBot
Allow: /
Disallow: /studio
Disallow: /api/
Disallow: /admin
Disallow: /dashboard
Disallow: /wp-admin
Content-Signal: ai-train=yes, search=yes, ai-input=yes

User-Agent: ClaudeBot
Allow: /
Disallow: /studio
Disallow: /api/
Disallow: /admin
Disallow: /dashboard
Disallow: /wp-admin
Content-Signal: ai-train=yes, search=yes, ai-input=yes

Sitemap: https://www.buildwithmatija.com/sitemap.xml
```

Notice how Next.js preserves the `Content-Signal` casing and groups the directive cleanly inside each stanza. The admin paths remain disallowed, the sitemap URL resolves dynamically against your configured domain, and compliant AI scrapers receive unambiguous guidelines on how your content may be consumed.

---

## Conclusion

Controlling how automated agents interact with your content no longer requires an all-or-nothing approach. In this guide, we explored how the Content Signals standard categorizes automated content usage across search indexing, real-time AI input, and model training. We then leveraged the `other` directive map introduced in Next.js 16.3 to emit type-safe, non-standard directives directly inside `app/robots.ts`, avoiding the need to write custom route handlers or manage raw static text files.

With this setup in place, your application provides clear, machine-readable preferences for search engines and AI operators while keeping your metadata configuration centralized and maintainable.

Keep in mind that application-level rules in `robots.ts` only take effect if the incoming crawler actually reaches Next.js. If you host on Vercel, automated AI agents from cloud hosting IP ranges may be throttled before this file is ever evaluated — see [Configure Vercel Firewall for AI Bots Without 429 Rate Limits](/blog/configure-vercel-firewall-ai-bots-edge-rate-limits) for the companion edge WAF configuration. For multi-site setups, see [Dynamic robots.txt in Multi-Tenant Next.js](/blog/dynamic-robots-txt-nextjs-multi-tenant), and pair this with an [llms.txt route for structured agent discovery](/blog/implementing-llms-txt-nextjs-15-sanity-cms).

Let me know in the comments if you have questions, and subscribe for more practical development guides.

Thanks, Matija