# How to Programmatically Generate robots.txt and sitemap.xml in Next.js 14

> Learn to programmatically generate robots.txt and sitemap.xml in Next.js 14. Export MetadataRoute typed functions from app/robots.ts and app/sitemap.ts for dynamic SEO file generation at build time.

- Repository: [Ege Chelebi/blog](https://github.com/woosal1337/blog)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Export `MetadataRoute` typed functions from [`app/robots.ts`](https://github.com/woosal1337/blog/blob/main/app/robots.ts) and [`app/sitemap.ts`](https://github.com/woosal1337/blog/blob/main/app/sitemap.ts) to automatically generate dynamic SEO files at build time.**

In Next.js 14 with the App Router, you can programmatically generate [`robots.txt`](https://github.com/woosal1337/blog/blob/main/robots.txt) and [`sitemap.xml`](https://github.com/woosal1337/blog/blob/main/sitemap.xml) without plugins or manual file creation. The **woosal1337/blog** repository demonstrates this pattern using Next.js's built-in metadata route conventions. These files become static assets during `next build`, eliminating maintenance overhead for dynamic content.

## The MetadataRoute API Pattern

Next.js recognizes two special filenames in the `app` directory: **[`robots.ts`](https://github.com/woosal1337/blog/blob/main/robots.ts)** and **[`sitemap.ts`](https://github.com/woosal1337/blog/blob/main/sitemap.ts)**. When you export a default function from these files with the correct return type, Next.js automatically serves them at [`/robots.txt`](https://github.com/woosal1337/blog/blob/main//robots.txt) and [`/sitemap.xml`](https://github.com/woosal1337/blog/blob/main//sitemap.xml).

This approach replaces older plugin-based solutions and integrates seamlessly with React Server Components and async data fetching.

## Generating robots.txt with MetadataRoute.Robots

Create [`app/robots.ts`](https://github.com/woosal1337/blog/blob/main/app/robots.ts) and export a function returning **`MetadataRoute.Robots`**. This object defines crawl rules, sitemap location, and host preference.

```tsx
// app/robots.ts
import { SITE_URL, absoluteUrl } from "@/lib/seo";
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: "*" }],
    sitemap: absoluteUrl("/sitemap.xml"),
    host: SITE_URL,
  };
}

```

The `rules` array accepts multiple directives per user agent. You can restrict specific paths with `disallow: ["/admin", "/drafts"]` or allow selective crawling. The `sitemap` field hints crawlers to your sitemap location—critical for search engine discovery.

## Generating sitemap.xml with MetadataRoute.Sitemap

The **`MetadataRoute.Sitemap`** type expects an array of URL records. Each record supports `url`, `lastModified`, `changeFrequency`, and `priority` properties.

```tsx
// app/sitemap.ts
import { getAllPosts } from "@/lib/blog";
import { absoluteUrl } from "@/lib/seo";
import { getAllEpisodes, kitFileHref, listKitFiles } from "@/lib/videos";
import type { MetadataRoute } from "next";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  // Static routes
  const staticRoutes = [
    "",
    "/blog",
    "/projects",
    "/videos",
    "/about",
    "/lab",
  ].map((route) => ({
    url: absoluteUrl(route || "/"),
  }));

  // Dynamic blog posts
  const posts = await getAllPosts();
  const postRoutes = posts.map((post) => ({
    url: absoluteUrl(`/blog/${post.slug}`),
    lastModified: (post.updated ?? post.date).split("T")[0],
  }));

  // Dynamic video episodes
  const episodes = await getAllEpisodes();
  const episodeRoutes = episodes.map((episode) => ({
    url: absoluteUrl(`/videos/${episode.slug}`),
    lastModified: episode.date.split("T")[0],
  }));

  // Nested assets (episode kits)
  const kitRoutes = episodes.flatMap((episode) =>
    episode.kit
      ? listKitFiles(episode.kit.dir).map((file) => ({
          url: absoluteUrl(kitFileHref(episode.slug, file.path)),
          lastModified: episode.date.split("T")[0],
        }))
      : []
  );

  return [...staticRoutes, ...postRoutes, ...episodeRoutes, ...kitRoutes];
}

```

The function can be **async**, enabling database queries or filesystem reads during build. In **woosal1337/blog**, `getAllPosts()` reads MDX files from `content/blog/` and `getAllEpisodes()` parses video metadata.

## URL Utilities for Consistent Domains

Absolute URLs are required for both files. The repository centralizes this logic in [`lib/seo.ts`](https://github.com/woosal1337/blog/blob/main/lib/seo.ts):

```tsx
// lib/seo.ts
export const SITE_URL = "https://www.chele.bi";

export function absoluteUrl(pathname: string): string {
  return new URL(pathname, SITE_URL).toString();
}

```

This utility prevents protocol mismatches and ensures trailing slash consistency. Environment variables can inject `SITE_URL` for multi-environment deployments:

```tsx
export const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL || "http://localhost:3000";

```

## File Structure Reference

| File | Purpose |
|------|---------|
| [[`app/robots.ts`](https://github.com/woosal1337/blog/blob/main/app/robots.ts)](https://github.com/woosal1337/blog/blob/main/app/robots.ts) | Exports `robots()` → generates [`/robots.txt`](https://github.com/woosal1337/blog/blob/main//robots.txt) |
| [[`app/sitemap.ts`](https://github.com/woosal1337/blog/blob/main/app/sitemap.ts)](https://github.com/woosal1337/blog/blob/main/app/sitemap.ts) | Exports `sitemap()` → generates [`/sitemap.xml`](https://github.com/woosal1337/blog/blob/main//sitemap.xml) |
| [[`lib/seo.ts`](https://github.com/woosal1337/blog/blob/main/lib/seo.ts)](https://github.com/woosal1337/blog/blob/main/lib/seo.ts) | `absoluteUrl()` helper and `SITE_URL` constant |
| [[`lib/blog.ts`](https://github.com/woosal1337/blog/blob/main/lib/blog.ts)](https://github.com/woosal1337/blog/blob/main/lib/blog.ts) | `getAllPosts()` for blog content |
| [[`lib/videos.ts`](https://github.com/woosal1337/blog/blob/main/lib/videos.ts)](https://github.com/woosal1337/blog/blob/main/lib/videos.ts) | `getAllEpisodes()`, `listKitFiles()` for video content |

## Extending the Pattern

To add new content types to your sitemap, follow this template:

1. Create a data fetcher in `lib/` (e.g., `getAllProducts()`)
2. Map results to `MetadataRoute.Sitemap` entries
3. Merge into the returned array in [`app/sitemap.ts`](https://github.com/woosal1337/blog/blob/main/app/sitemap.ts)

The build process automatically picks up changes—no cache invalidation or manual regeneration required.

## Summary

- **Use `MetadataRoute.Robots`** in [`app/robots.ts`](https://github.com/woosal1337/blog/blob/main/app/robots.ts) for crawl directives and sitemap hints
- **Use `MetadataRoute.Sitemap`** in [`app/sitemap.ts`](https://github.com/woosal1337/blog/blob/main/app/sitemap.ts) for URL listings with optional change frequency and priority
- **Export default functions**—Next.js handles routing and content-type headers automatically
- **Centralize URL construction** via a utility like `absoluteUrl()` to avoid domain mismatches
- **Leverage async functions** to pull from CMS, database, or filesystem at build time

## Frequently Asked Questions

### Does this work with Next.js Pages Router?

No. The `MetadataRoute` convention requires **App Router** (Next.js 13.3+). For Pages Router, use `getServerSideProps` or external packages like `next-sitemap`.

### When are these files generated?

At **`next build` time**. The functions execute during static site generation, producing static [`robots.txt`](https://github.com/woosal1337/blog/blob/main/robots.txt) and [`sitemap.xml`](https://github.com/woosal1337/blog/blob/main/sitemap.xml) files in the output directory. They do not execute per-request in production.

### Can I split large sitemaps into multiple files?

Yes. Return an array of sitemap objects from multiple `.ts` files in subdirectories—Next.js supports `app/sitemap.xml/[id]/route.ts` for dynamic sitemap indexes when you exceed 50,000 URLs.

### Why use `.ts` instead of `.js`?

TypeScript provides IntelliSense for `MetadataRoute` types and catches property errors before build. JavaScript works identically if you prefer, but you lose compile-time validation.