Declare AI content preferences with Next.js 16.3 and the new other directive map in app/robots.ts.
In-depth Next.js guides covering App Router, RSC, ISR, and deployment. Get code examples, optimization checklists, and prompts to accelerate development.
No spam. Unsubscribe anytime.
I was recently looking through my Cloudflare dashboard, checking their new AI Overview and scraper analytics, when I noticed something glaring: my production site had no signal preferences configured at all. While bots and AI crawlers were routinely visiting my content, Cloudflare's report showed blank preferences because standard robots.txt offers only binary choices—either you allow an automated crawler completely, or you lock them out entirely.
With AI crawlers indexing content for live search answers, RAG systems, and foundation model training, site owners need a way to declare nuanced, granular permissions without blocking traditional search discovery.
When I tried to configure the emerging Content Signals standard in Next.js, I hit an immediate framework limitation: older versions of robots.ts stripped non-standard directives from output. With Next.js 16.3, the framework introduced an official escape hatch for non-standard directives. In this guide, I will show you how to implement Content Signals directly inside app/robots.ts using the new other directive map, declaring clear preferences for search indexing, live AI answers, and model training.
Before writing configuration code, we need to understand how Content Signals convey publisher preferences to automated crawlers. Originating from an IETF internet draft and popularized by Cloudflare, Content Signals establish three distinct categories of content usage:
search: Building a search engine index and displaying links and short snippets in search results.ai-input: Passing website content into an AI system in real time, such as Retrieval-Augmented Generation (RAG) queries, grounded chat answers, or AI search summaries.ai-train: Ingesting content to train, pre-train, or fine-tune artificial intelligence foundation models.Each category accepts a simple yes or no value. A directive string combines them in comma-delimited pairs, like this:
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Depending on your business goals, you can disallow model training while remaining discoverable in AI search engines, or you can allow full access across all three categories to maximize your presence across AI assistants and web crawlers.
Next.js added support for custom directives on crawler rules in version 16.3.0. In older releases, the MetadataRoute.Robots TypeScript interface and the underlying string generator only handled userAgent, allow, disallow, and crawlDelay. Any additional keys passed to the rule object were quietly dropped during serialization.
Check your current Next.js version in package.json and ensure it is updated to at least 16.3.0:
// File: package.json
{
"dependencies": {
"next": "^16.3.5"
}
}
If you need to update your installation, run your package manager update command:
pnpm add next@latest
This ensures your project receives the updated MetadataRoute.Robots types, which include the other directive record:
type RobotsRule = {
userAgent?: string | string[]
allow?: string | string[]
disallow?: string | string[]
crawlDelay?: number
other?: Record<string, string | number | Array<string | number>>
}
With the package upgraded, Next.js will preserve the exact casing of keys declared inside other and render each key-value pair directly beneath the corresponding User-agent block.
app/robots.tsNow we can implement our rules inside src/app/robots.ts. We want our signals to apply both to generic crawlers via the wildcard * agent and to specific, high-profile AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot.
Here is the complete implementation:
// File: src/app/robots.ts
import { MetadataRoute } from 'next'
import { siteOrigin } from '@/lib/site-url'
const baseUrl = siteOrigin
const adminPaths = ['/studio', '/api/', '/admin', '/dashboard', '/wp-admin']
const llmAgents = [
'GPTBot',
'ChatGPT-User',
'Claude-Web',
'ClaudeBot',
'anthropic-ai',
'PerplexityBot',
'CCBot',
'Omgilibot',
'FacebookBot',
'Google-Extended',
'cohere-ai',
'Applebot-Extended',
'Bytespider',
'Diffbot',
]
export default function robots(): MetadataRoute.Robots {
const contentSignalPolicy = 'ai-train=yes, search=yes, ai-input=yes'
return {
sitemap: `${baseUrl}/sitemap.xml`,
rules: [
{
userAgent: '*',
allow: '/',
disallow: adminPaths,
other: {
'Content-Signal': contentSignalPolicy,
},
},
...llmAgents.map((userAgent) => ({
userAgent,
allow: '/',
disallow: adminPaths,
other: {
'Content-Signal': contentSignalPolicy,
},
})),
],
}
}
In this implementation, we define an array of restricted administrative paths and a list of recognized LLM crawler identifiers. We define our contentSignalPolicy variable according to our preferred permissions. In this example, we set ai-train=yes, search=yes, ai-input=yes to invite search engines, citation systems, and AI tools to index and cite our work freely.
Next, we map our rules array. The first rule targets the global wildcard *. The subsequent rules iterate through our list of specific AI user-agents. Inside each rule, we supply the other dictionary with our 'Content-Signal' header.
Because robots.txt parser rules state that crawlers matching a specific user-agent header disregard the generic wildcard block, adding other to each explicit AI bot entry ensures the directive is observed regardless of which block the crawler matches.
robots.txt OutputNext.js automatically serves the return value of app/robots.ts at the /robots.txt path. You can verify the final serialized text by running your development server or requesting the route directly in your browser.
The generated output will look like this:
User-Agent: *
Allow: /
Disallow: /studio
Disallow: /api/
Disallow: /admin
Disallow: /dashboard
Disallow: /wp-admin
Content-Signal: ai-train=yes, search=yes, ai-input=yes
User-Agent: GPTBot
Allow: /
Disallow: /studio
Disallow: /api/
Disallow: /admin
Disallow: /dashboard
Disallow: /wp-admin
Content-Signal: ai-train=yes, search=yes, ai-input=yes
User-Agent: ClaudeBot
Allow: /
Disallow: /studio
Disallow: /api/
Disallow: /admin
Disallow: /dashboard
Disallow: /wp-admin
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Sitemap: https://www.buildwithmatija.com/sitemap.xml
Notice how Next.js preserves the Content-Signal casing and groups the directive cleanly inside each stanza. The admin paths remain disallowed, the sitemap URL resolves dynamically against your configured domain, and compliant AI scrapers receive unambiguous guidelines on how your content may be consumed.
Controlling how automated agents interact with your content no longer requires an all-or-nothing approach. In this guide, we explored how the Content Signals standard categorizes automated content usage across search indexing, real-time AI input, and model training. We then leveraged the other directive map introduced in Next.js 16.3 to emit type-safe, non-standard directives directly inside app/robots.ts, avoiding the need to write custom route handlers or manage raw static text files.
With this setup in place, your application provides clear, machine-readable preferences for search engines and AI operators while keeping your metadata configuration centralized and maintainable.
Keep in mind that application-level rules in robots.ts only take effect if the incoming crawler actually reaches Next.js. If you host on Vercel, automated AI agents from cloud hosting IP ranges may be throttled before this file is ever evaluated — see Configure Vercel Firewall for AI Bots Without 429 Rate Limits for the companion edge WAF configuration. For multi-site setups, see Dynamic robots.txt in Multi-Tenant Next.js, and pair this with an llms.txt route for structured agent discovery.
Let me know in the comments if you have questions, and subscribe for more practical development guides.
Thanks, Matija