How to Programmatically Generate robots.txt and sitemap.xml in Next.js 14

Export MetadataRoute typed functions from app/robots.ts and app/sitemap.ts to automatically generate dynamic SEO files at build time.

In Next.js 14 with the App Router, you can programmatically generate robots.txt and sitemap.xml without plugins or manual file creation. The woosal1337/blog repository demonstrates this pattern using Next.js's built-in metadata route conventions. These files become static assets during next build, eliminating maintenance overhead for dynamic content.

The MetadataRoute API Pattern

Next.js recognizes two special filenames in the app directory: robots.ts and sitemap.ts. When you export a default function from these files with the correct return type, Next.js automatically serves them at /robots.txt and /sitemap.xml.

This approach replaces older plugin-based solutions and integrates seamlessly with React Server Components and async data fetching.

Generating robots.txt with MetadataRoute.Robots

Create app/robots.ts and export a function returning MetadataRoute.Robots. This object defines crawl rules, sitemap location, and host preference.

// app/robots.ts
import { SITE_URL, absoluteUrl } from "@/lib/seo";
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: "*" }],
    sitemap: absoluteUrl("/sitemap.xml"),
    host: SITE_URL,
  };
}

The rules array accepts multiple directives per user agent. You can restrict specific paths with disallow: ["/admin", "/drafts"] or allow selective crawling. The sitemap field hints crawlers to your sitemap location—critical for search engine discovery.

Generating sitemap.xml with MetadataRoute.Sitemap

The MetadataRoute.Sitemap type expects an array of URL records. Each record supports url, lastModified, changeFrequency, and priority properties.

// app/sitemap.ts
import { getAllPosts } from "@/lib/blog";
import { absoluteUrl } from "@/lib/seo";
import { getAllEpisodes, kitFileHref, listKitFiles } from "@/lib/videos";
import type { MetadataRoute } from "next";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  // Static routes
  const staticRoutes = [
    "",
    "/blog",
    "/projects",
    "/videos",
    "/about",
    "/lab",
  ].map((route) => ({
    url: absoluteUrl(route || "/"),
  }));

  // Dynamic blog posts
  const posts = await getAllPosts();
  const postRoutes = posts.map((post) => ({
    url: absoluteUrl(`/blog/${post.slug}`),
    lastModified: (post.updated ?? post.date).split("T")[0],
  }));

  // Dynamic video episodes
  const episodes = await getAllEpisodes();
  const episodeRoutes = episodes.map((episode) => ({
    url: absoluteUrl(`/videos/${episode.slug}`),
    lastModified: episode.date.split("T")[0],
  }));

  // Nested assets (episode kits)
  const kitRoutes = episodes.flatMap((episode) =>
    episode.kit
      ? listKitFiles(episode.kit.dir).map((file) => ({
          url: absoluteUrl(kitFileHref(episode.slug, file.path)),
          lastModified: episode.date.split("T")[0],
        }))
      : []
  );

  return [...staticRoutes, ...postRoutes, ...episodeRoutes, ...kitRoutes];
}

The function can be async, enabling database queries or filesystem reads during build. In woosal1337/blog, getAllPosts() reads MDX files from content/blog/ and getAllEpisodes() parses video metadata.

URL Utilities for Consistent Domains

Absolute URLs are required for both files. The repository centralizes this logic in lib/seo.ts:

// lib/seo.ts
export const SITE_URL = "https://www.chele.bi";

export function absoluteUrl(pathname: string): string {
  return new URL(pathname, SITE_URL).toString();
}

This utility prevents protocol mismatches and ensures trailing slash consistency. Environment variables can inject SITE_URL for multi-environment deployments:

export const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL || "http://localhost:3000";

File Structure Reference

File	Purpose
[app/robots.ts](https://github.com/woosal1337/blog/blob/main/app/robots.ts)	Exports robots() → generates /robots.txt
[app/sitemap.ts](https://github.com/woosal1337/blog/blob/main/app/sitemap.ts)	Exports sitemap() → generates /sitemap.xml
[lib/seo.ts](https://github.com/woosal1337/blog/blob/main/lib/seo.ts)	absoluteUrl() helper and SITE_URL constant
[lib/blog.ts](https://github.com/woosal1337/blog/blob/main/lib/blog.ts)	getAllPosts() for blog content
[lib/videos.ts](https://github.com/woosal1337/blog/blob/main/lib/videos.ts)	getAllEpisodes(), listKitFiles() for video content

Extending the Pattern

To add new content types to your sitemap, follow this template:

1. Create a data fetcher in lib/ (e.g., getAllProducts())
2. Map results to MetadataRoute.Sitemap entries
3. Merge into the returned array in app/sitemap.ts

The build process automatically picks up changes—no cache invalidation or manual regeneration required.

Summary

• Use MetadataRoute.Robots in app/robots.ts for crawl directives and sitemap hints
• Use MetadataRoute.Sitemap in app/sitemap.ts for URL listings with optional change frequency and priority
• Export default functions—Next.js handles routing and content-type headers automatically
• Centralize URL construction via a utility like absoluteUrl() to avoid domain mismatches
• Leverage async functions to pull from CMS, database, or filesystem at build time

Frequently Asked Questions

Does this work with Next.js Pages Router?

No. The MetadataRoute convention requires App Router (Next.js 13.3+). For Pages Router, use getServerSideProps or external packages like next-sitemap.

When are these files generated?

At next build time. The functions execute during static site generation, producing static robots.txt and sitemap.xml files in the output directory. They do not execute per-request in production.

Can I split large sitemaps into multiple files?

Yes. Return an array of sitemap objects from multiple .ts files in subdirectories—Next.js supports app/sitemap.xml/[id]/route.ts for dynamic sitemap indexes when you exceed 50,000 URLs.

Why use .ts instead of .js?

TypeScript provides IntelliSense for MetadataRoute types and catches property errors before build. JavaScript works identically if you prefer, but you lose compile-time validation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}