Build a CMS webhook pipeline that keeps OpenAI File Search results fresh
In-depth Next.js guides covering App Router, RSC, ISR, and deployment. Get code examples, optimization checklists, and prompts to accelerate development.
No spam. Unsubscribe anytime.
If your OpenAI File Search results go stale every time your CMS content changes, you need a sync pipeline between your CMS and OpenAI Vector Stores.
I built a webhook-based system that syncs posts when I publish. This guide shows the full implementation using Sanity CMS as the example, but the same pattern works with any headless CMS that supports webhooks: Payload, Strapi, Contentful, or a custom admin.
Updated June 2026: OpenAI now positions File Search as a tool in the Responses API. This guide keeps the CMS webhook architecture, but the OpenAI integration now uses the current Vector Store SDK methods. If you still have an Assistants API implementation, the same sync layer still applies, but new projects should treat Responses API as the default path.
By the end, you'll have a production-ready sync system with author control, status tracking, and visible error handling. Your content will be ready for semantic search through File Search, ChatKit, or any OpenAI workflow that can query a vector store.
When you build an AI-powered search feature, OpenAI needs your content in its Vector Store. The typical workflow looks like this:
This breaks down immediately. You forget to upload posts. Your search results become stale. The AI doesn't know about your latest content.
What we need is automation: publish in the CMS, content automatically appears in the vector store. That's what webhooks enable.
The architecture is straightforward. Your CMS fires a webhook when content is published. Your Next.js API route catches it, formats the content, uploads to OpenAI, and updates status fields back in the CMS.
Here's the flow:
The webhook response can stay fast because the expensive work should happen outside the request path. In a long-running Node server you can start the sync in-process; in serverless environments, use a queue, background job, or platform background task so the upload is not killed when the HTTP response finishes.
This guide uses Sanity CMS, but the core pattern is CMS-agnostic. The vector store service works with any content source. Only the webhook handler and status tracking need CMS-specific code.
By the end of this guide, you'll have:
The example uses Sanity, but I'll point out which parts are CMS-specific so you can adapt it to your setup.
Before starting, make sure you have:
You'll also need the openai npm package installed:
npm install openai
# or
pnpm add openai
First, create a vector store that will hold your CMS content. You can do this once in a setup script, then store the returned ID in your environment variables.
// File: scripts/create-vector-store.ts
import OpenAI from 'openai'
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
})
async function main() {
const vectorStore = await openai.vectorStores.create({
name: 'Blog Content Store',
})
console.log('Vector store ID:', vectorStore.id)
}
main().catch(error => {
console.error(error)
process.exit(1)
})
Save the returned ID in OPENAI_VECTOR_STORE_ID. It looks like vs_abc123xyz.
The vector store is where OpenAI indexes your content for File Search. When users ask questions through the Responses API, ChatKit, or an assistant-style interface, OpenAI searches this store and retrieves relevant chunks from your CMS content.
Add these to your .env.local file:
# OpenAI Configuration
OPENAI_API_KEY=your-openai-api-key
OPENAI_VECTOR_STORE_ID=vs_abc123xyz
# Your CMS API credentials (example with Sanity)
SANITY_API_TOKEN=your-sanity-editor-token
NEXT_PUBLIC_SANITY_PROJECT_ID=your-project-id
NEXT_PUBLIC_SANITY_DATASET=production
SANITY_WEBHOOK_SECRET=your-webhook-secret
# Application
NEXT_PUBLIC_BASE_URL=https://yourdomain.com
NODE_ENV=development
The OpenAI key needs access to Files and Vector Stores. For your CMS token, you'll need write permissions—the webhook updates status fields after syncing.
The NODE_ENV=development setting lets you test webhooks locally without signature verification. In production, this automatically enables security checks.
This is the CMS-agnostic core. It handles formatting CMS content, uploading it to OpenAI Files, and attaching the uploaded file to your vector store. Create this file:
// File: src/lib/openai/vector-store-service.ts
import OpenAI from 'openai'
import { toFile } from 'openai/uploads'
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
})
const VECTOR_STORE_ID = process.env.OPENAI_VECTOR_STORE_ID!
export interface BlogPostContent {
title: string
slug: string
subtitle?: string
excerpt?: string
author: string
publishedAt: string
categories: string[]
markdownContent: string
url: string
}
export interface VectorStoreSyncResult {
success: boolean
fileId: string
vectorStoreFileId?: string
error?: string
}
function formatBlogPostForVectorStore(post: BlogPostContent): string {
const parts: string[] = []
parts.push('---')
parts.push(`Title: ${post.title}`)
if (post.subtitle) parts.push(`Subtitle: ${post.subtitle}`)
if (post.excerpt) parts.push(`Excerpt: ${post.excerpt}`)
parts.push(`Author: ${post.author}`)
parts.push(`Published: ${new Date(post.publishedAt).toLocaleDateString('en-US', {
year: 'numeric',
month: 'long',
day: 'numeric',
})}`)
if (post.categories.length > 0) {
parts.push(`Categories: ${post.categories.join(', ')}`)
}
parts.push(`URL: ${post.url}`)
parts.push('---')
parts.push('')
parts.push(`# ${post.title}`)
if (post.subtitle) {
parts.push('')
parts.push(`*${post.subtitle}*`)
}
parts.push('')
parts.push(post.markdownContent)
return parts.join('\n')
}
export async function uploadBlogPostToVectorStore(
post: BlogPostContent
): Promise<VectorStoreSyncResult> {
try {
console.log(`[VectorStore] Starting upload for post: ${post.slug}`)
const formattedContent = formatBlogPostForVectorStore(post)
const fileBuffer = Buffer.from(formattedContent, 'utf-8')
const fileName = `${post.slug}.md`
const file = await toFile(fileBuffer, fileName, {
type: 'text/markdown',
})
const uploadedFile = await openai.files.create({
file,
purpose: 'assistants',
})
const vectorStoreFile = await openai.vectorStores.files.create(
VECTOR_STORE_ID,
{
file_id: uploadedFile.id,
}
)
console.log(`[VectorStore] File attached to vector store: ${VECTOR_STORE_ID}`)
return {
success: true,
fileId: uploadedFile.id,
vectorStoreFileId: vectorStoreFile.id,
}
} catch (error) {
console.error(`[VectorStore] Error uploading post ${post.slug}:`, error)
return {
success: false,
fileId: '',
error: error instanceof Error ? error.message : 'Unknown error occurred',
}
}
}
This service does three things: formats your CMS content as markdown with metadata, uploads it to OpenAI Files, then attaches the file to your vector store.
The formatting matters. File Search works better when content has structure. The metadata header lets you include context like title, author, publish date, categories, and URL without mixing that metadata into the article body. When the model retrieves a chunk, it has enough surrounding context to cite the right article or explain where the information came from.
The BlogPostContent interface is generic: title, content, URL, and metadata. Any CMS can populate this shape. Sanity, Payload, Strapi, Contentful, or your own database all feed the same vector store service.
The important update from older implementations is that you no longer need a manual REST call just to attach a file to the vector store. The Node SDK exposes vector store helpers directly, so the upload path can stay inside the OpenAI client.
Now we add fields to your CMS so authors can control syncing and see status. This part is CMS-specific, but the pattern applies everywhere: checkbox to enable sync, fields to track status.
Here's the Sanity implementation:
// File: src/lib/sanity/schemaTypes/postType.ts
import { defineField } from 'sanity'
// Add these fields to your existing post schema
defineField({
name: 'includeInVectorStore',
title: 'Include in Vector Store',
type: 'boolean',
description: 'Enable this to sync this post to OpenAI vector store for AI search',
initialValue: false,
}),
defineField({
name: 'vectorStoreFileId',
title: 'Vector Store File ID',
type: 'string',
description: 'OpenAI file ID (auto-populated)',
readOnly: true,
hidden: true,
}),
defineField({
name: 'vectorStoreSyncedAt',
title: 'Last Synced',
type: 'datetime',
description: 'When this post was last synced',
readOnly: true,
hidden: ({ document }) => !document?.vectorStoreFileId,
}),
defineField({
name: 'vectorStoreSyncStatus',
title: 'Sync Status',
type: 'string',
options: {
list: [
{ title: 'Not Synced', value: 'not_synced' },
{ title: 'Pending', value: 'pending' },
{ title: 'Synced', value: 'synced' },
{ title: 'Failed', value: 'failed' },
],
},
readOnly: true,
hidden: ({ document }) => !document?.includeInVectorStore,
initialValue: 'not_synced',
}),
defineField({
name: 'vectorStoreSyncError',
title: 'Sync Error',
type: 'text',
description: 'Error message if sync failed',
readOnly: true,
hidden: ({ document }) => document?.vectorStoreSyncStatus !== 'failed',
}),
The checkbox (includeInVectorStore) is always visible. Authors decide which posts to sync. The status fields appear only when relevant—no clutter.
When an author publishes with the checkbox enabled, the webhook sees it and triggers the sync. The status goes from not_synced to pending to synced (or failed if something breaks). Authors can refresh and see progress.
For other CMS platforms:
admin.readOnly and admin.condition for visibilityThe pattern is universal: one writable boolean, several read-only status fields.
The webhook handler is where CMS events trigger actions. Your CMS calls this endpoint when a post is published. The handler validates the request, checks if syncing is enabled, and kicks off the background upload.
Here's the Next.js App Router implementation:
// File: src/app/api/revalidate/route.ts
import { NextRequest, NextResponse } from 'next/server'
import { createClient } from '@sanity/client'
import { uploadBlogPostToVectorStore } from '@/lib/openai/vector-store-service'
// Sanity client with write permissions
const sanityClient = createClient({
projectId: process.env.NEXT_PUBLIC_SANITY_PROJECT_ID!,
dataset: process.env.NEXT_PUBLIC_SANITY_DATASET!,
apiVersion: '2024-01-01',
token: process.env.SANITY_API_TOKEN,
useCdn: false,
})
async function syncToVectorStore(postId: string, slug: string | undefined) {
try {
console.log(`[VectorStore] Syncing post: ${postId}`)
// Update status to pending
await sanityClient
.patch(postId)
.set({
vectorStoreSyncStatus: 'pending',
vectorStoreSyncError: null,
})
.commit()
// Fetch full post data from CMS
const post = await sanityClient.fetch(
`*[_id == $postId][0]{
title,
"slug": slug.current,
subtitle,
excerpt,
"author": author->name,
publishedAt,
"categories": categories[]->title,
markdownContent,
}`,
{ postId }
)
if (!post) {
throw new Error('Post not found')
}
if (!post.markdownContent) {
throw new Error('Post has no markdown content')
}
// Prepare content for vector store
const blogPostContent = {
title: post.title,
slug: post.slug,
subtitle: post.subtitle,
excerpt: post.excerpt,
author: post.author || 'Unknown Author',
publishedAt: post.publishedAt,
categories: post.categories || [],
markdownContent: post.markdownContent,
url: `${process.env.NEXT_PUBLIC_BASE_URL}/blog/${post.slug}`,
}
console.log(`[VectorStore] Uploading to OpenAI...`)
// Upload to vector store
const result = await uploadBlogPostToVectorStore(blogPostContent)
if (result.success) {
// Update CMS with success
await sanityClient
.patch(postId)
.set({
vectorStoreFileId: result.fileId,
vectorStoreSyncStatus: 'synced',
vectorStoreSyncedAt: new Date().toISOString(),
vectorStoreSyncError: null,
})
.commit()
console.log(`[VectorStore] Successfully synced: ${post.slug}`)
} else {
// Update CMS with failure
await sanityClient
.patch(postId)
.set({
vectorStoreSyncStatus: 'failed',
vectorStoreSyncError: result.error || 'Unknown error',
})
.commit()
console.error(`[VectorStore] Failed to sync: ${post.slug}`)
}
} catch (error) {
console.error(`[VectorStore] Sync error:`, error)
// Update CMS with error
try {
await sanityClient
.patch(postId)
.set({
vectorStoreSyncStatus: 'failed',
vectorStoreSyncError: error instanceof Error ? error.message : 'Unknown error',
})
.commit()
} catch (updateError) {
console.error('[VectorStore] Failed to update error status:', updateError)
}
}
}
export async function POST(request: NextRequest) {
try {
const body = await request.json()
const { _type, slug, _id, includeInVectorStore } = body
console.log('Webhook received:', { _type, slug: slug?.current, _id })
if (_type === 'post') {
// Handle vector store sync (non-blocking)
if (includeInVectorStore === true) {
console.log(`[VectorStore] Post marked for sync: ${slug?.current}`)
// Trigger async sync (don't await - keep webhook fast)
syncToVectorStore(_id, slug?.current)
.catch(error => {
console.error(`[VectorStore] Background sync failed:`, error)
})
}
return NextResponse.json({
revalidated: true,
vectorStoreSync: includeInVectorStore ? 'initiated' : 'skipped',
timestamp: new Date().toISOString()
})
}
return NextResponse.json({
message: 'Webhook processed',
timestamp: new Date().toISOString()
})
} catch (error) {
console.error('Webhook error:', error)
return NextResponse.json(
{ error: 'Failed to process webhook' },
{ status: 500 }
)
}
}
The key design choice here is separating the webhook response from the sync work. The CMS should get a quick response, while the OpenAI upload runs through a background path.
For a self-hosted Node server, the fire-and-forget pattern above can be enough. For Vercel, Netlify, or other serverless deployments, use a queue, scheduled worker, or platform-supported background task instead. Otherwise the runtime may stop work after the HTTP response is returned.
The sync function updates status three times: first to pending when it starts, then to synced or failed when it finishes. Authors can refresh the post in the CMS and see progress.
For CMS integration, the Sanity-specific parts are the sanityClient calls. Here's how you'd adapt this to other platforms:
Payload CMS:
// Instead of sanityClient.patch()
await payload.update({
collection: 'posts',
id: postId,
data: {
vectorStoreSyncStatus: 'pending',
vectorStoreSyncError: null,
},
})
Strapi:
// Using Strapi's SDK
await strapi.entityService.update('api::post.post', postId, {
data: {
vectorStoreSyncStatus: 'pending',
vectorStoreSyncError: null,
},
})
The pattern is identical: receive webhook, validate, fetch content, upload to OpenAI, update status. Only the CMS client calls change.
Your CMS needs to know where to send publish events. The setup varies by platform, but the concept is universal: tell your CMS to POST to your webhook URL when content is published.
For Sanity, go to your project settings in the Sanity dashboard:
https://yourdomain.com/api/revalidateproduction)SANITY_WEBHOOK_SECRET)The secret is important for production. It lets you verify webhooks are actually from your CMS, not random POST requests. In development, we skip verification with NODE_ENV=development for easier testing.
For other platforms:
Payload CMS: Use hooks in your collection config:
hooks: {
afterChange: [
async ({ doc, req, operation }) => {
if (operation === 'create' || operation === 'update') {
// Trigger your webhook endpoint
await fetch(`${process.env.APP_URL}/api/revalidate`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(doc)
})
}
}
]
}
Strapi: Create a lifecycle hook in src/api/post/content-types/post/lifecycles.js:
module.exports = {
async afterCreate(event) {
await fetch(`${process.env.APP_URL}/api/revalidate`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(event.result)
})
}
}
The webhook URL stays the same. Your Next.js handler doesn't care which CMS sent the data—it just needs the post ID and the includeInVectorStore flag.
Before deploying, test the entire flow locally. Start your development server:
pnpm dev
Use cURL to simulate a webhook from your CMS:
curl -X POST http://localhost:3000/api/revalidate \
-H "Content-Type: application/json" \
-d '{
"_type": "post",
"_id": "test-post-id",
"slug": { "current": "test-post" },
"includeInVectorStore": true
}'
Watch your terminal. You should see logs like:
Webhook received: { _type: 'post', slug: 'test-post', _id: 'test-post-id' }
[VectorStore] Post marked for sync: test-post
[VectorStore] Syncing post: test-post-id
[VectorStore] Starting upload for post: test-post
[VectorStore] Uploading file: test-post.md
[VectorStore] File uploaded with ID: file-abc123
[VectorStore] File added to vector store: vs_abc123
[VectorStore] Successfully synced: test-post
Check your OpenAI dashboard at https://platform.openai.com/storage/vector_stores. You should see the new file in your vector store with a .md extension.
If you get errors, the most common issues are:
OPENAI_VECTOR_STORE_ID not set in .env.localThe local test proves the entire pipeline works: webhook → fetch content → format → upload → update status. Once this works, deployment is straightforward.
Deploy your Next.js app to your hosting platform (Vercel, Railway, etc.). Make sure all environment variables are set in your production environment—especially OPENAI_API_KEY and OPENAI_VECTOR_STORE_ID.
One critical difference in production: webhook signature verification. You'll want to validate that webhooks actually come from your CMS. Here's how to add that to your webhook handler:
// File: src/app/api/revalidate/route.ts
import { isValidSignature, SIGNATURE_HEADER_NAME } from '@sanity/webhook'
const WEBHOOK_SECRET = process.env.SANITY_WEBHOOK_SECRET
export async function POST(request: NextRequest) {
try {
const body = await request.text()
// Verify signature in production
if (WEBHOOK_SECRET && process.env.NODE_ENV !== 'development') {
const signature = request.headers.get(SIGNATURE_HEADER_NAME)
if (!signature) {
return NextResponse.json(
{ error: 'Missing signature' },
{ status: 401 }
)
}
const isValid = await isValidSignature(body, signature, WEBHOOK_SECRET)
if (!isValid) {
return NextResponse.json(
{ error: 'Invalid signature' },
{ status: 401 }
)
}
}
const data = JSON.parse(body)
// ... rest of your handler
} catch (error) {
// ... error handling
}
}
The process.env.NODE_ENV !== 'development' check lets you skip verification locally while requiring it in production. This protects your webhook from unauthorized requests.
For non-Sanity platforms, signature verification works similarly. Most CMS platforms sign webhooks with HMAC. Check your CMS documentation for the specific header name and verification method.
Once deployed, test with a real post in your CMS:
If the status shows "Failed," check your deployment logs for error messages. The vectorStoreSyncError field in your CMS will show what went wrong.
Verify the file uploaded by checking your OpenAI dashboard. The file should appear in your vector store with the post's slug as the filename (e.g., test-post.md).
Now that your CMS content is in the vector store, you can use it with File Search. For new OpenAI projects, the cleanest path is the Responses API:
// File: src/lib/openai/search-cms-content.ts
import OpenAI from 'openai'
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
})
export async function searchCmsContent(question: string) {
const response = await openai.responses.create({
model: 'gpt-5.5',
input: question,
tools: [
{
type: 'file_search',
vector_store_ids: [process.env.OPENAI_VECTOR_STORE_ID!],
},
],
})
return response
}
The vector store is the reusable layer. You can query it from the Responses API, wire it into ChatKit, or keep using it from an existing assistant-style setup while you migrate. The CMS sync logic does not need to care which interface eventually searches the files.
The metadata we added at upload time helps the model answer with context. Instead of only retrieving a paragraph, it can see the article title, author, publish date, categories, and URL near the content.
You now have a production-ready system that automatically syncs CMS content to OpenAI's Vector Store. Authors control which posts to sync via checkbox. Status updates happen in real-time. Error handling ensures failures are visible and debuggable.
The core pattern—webhook triggers upload, background processing, status tracking—works with any headless CMS that supports webhooks. I showed Sanity as the example, but the vector store service is completely CMS-agnostic. The webhook handler needs CMS-specific client code, but the structure stays the same.
Your content is now searchable through AI. As you publish new posts, they automatically appear in the vector store within seconds. No manual exports, no batch uploads, no stale search results.
This setup scales. Whether you publish once a week or ten times a day, the automation handles it. Authors don't think about syncing—they just write and publish.
If you're building AI-powered search or chat interfaces for content, this webhook pattern is the foundation. Combine it with ChatKit (see my previous guide) and you have a complete AI-powered knowledge base.
This implementation handles first-time syncs. For production, I would extend it with four more operations:
When a CMS post changes, upload a fresh markdown file and replace the old vector store attachment. Store the latest fileId in your CMS so the old file can be removed or ignored.
When a post is unpublished or deleted, remove the old file from the vector store. This prevents File Search from returning content that no longer exists publicly.
Add an admin-only script that syncs every existing post. Webhooks handle future changes, but batch sync gives you a clean starting point.
Run a scheduled check that compares CMS posts marked includeInVectorStore against files currently attached to the vector store. If a post is missing, stale, or failed, surface it in your admin UI.
Not necessarily. For a blog, one vector store is usually enough. Use metadata in the markdown to separate posts, docs, landing pages, or categories. Create separate vector stores only when access control, product boundaries, or retrieval quality require hard separation.
Markdown is usually the best default for CMS content because headings, lists, code blocks, links, and metadata remain readable. JSON can work for structured records, but long-form content is easier to retrieve when formatted as markdown.
Avoid blindly adding a new file every time a post changes. Save the old OpenAI file ID in your CMS, then remove or replace the old attachment when you upload the new version.
Yes. Payload's collection hooks can call the same vector store service. The CMS-specific part is fetching the post and writing the sync status back to the document.
Thanks for reading! Subscribe for more practical guides on building with AI APIs and modern content workflows.
Thanks, Matija