How to Audit Next.js Rendering for SEO

Auditing Next.js rendering for SEO requires verifying what the server actually delivers in the initial HTTP response versus what relies on browser execution after hydration. Although Next.js provides built-in primitives for Static Site Generation (SSG) and Server-Side Rendering (SSR), deploying the framework does not automatically ensure complete initial HTML. Implementation choices-such as fetching primary content inside useEffect hooks, wrapping indexable text in client-only boundaries, or catching API errors on the client-can still leave the initial payload empty, introduce conflicting metadata, or return a 200 OK status code for missing pages.

A technical review must begin by identifying whether the application uses the App Router, the Pages Router, or a hybrid migration of both, as diagnostic checks and code-level fixes differ substantially between the two architectures. The App Router defaults to React Server Components, uses a centralized Metadata API, and handles streaming via Suspense boundaries, whereas the Pages Router relies on route-level data fetching functions like getServerSideProps and manages document head tags through next/head . Understanding which routing paradigm controls a given URL template determines how to inspect and remediate rendering anomalies.

Rather than treating JavaScript execution as a binary pass-or-fail test, a practical Next.js audit evaluates the exact contract between the server and search engine crawlers across four technical layers: initial HTML parity with the hydrated DOM, server-rendered metadata and canonical tags, accurate HTTP status headers for redirects and missing routes, and unrestricted crawler access to framework assets under the /_next/ directory.

Evaluating rendering modes against crawler requirements

Next.js allows developers to configure rendering strategies per route, which means a single application often serves a mix of pre-rendered static files, dynamically generated server responses, and client-rendered shells. Auditing these routes requires matching each template's technical configuration against three operational constraints: how frequently the underlying data changes, the total number of URLs in the route pattern, and whether search engines need complete HTML in the initial server response.

Documented rendering modes across next.js architectures

While the conceptual rendering modes remain consistent, the code primitives that trigger them differ between the Pages Router and the App Router. Identifying the active mode during a code review or header inspection is the first step in diagnosing why a route responds quickly with cached markup or slowly with an uncached database query.

A common misconception in App Router audits is assuming that adding the 'use client' directive disables server rendering completely. In Next.js, Client Components are still pre-rendered on the server during the initial request to generate static HTML markup for their initial state. A route only degrades into pure Client-Side Rendering for its primary content when data fetching is deferred to browser-lifecycle hooks like useEffect , or when a component explicitly checks for the window object before rendering indexable text.

Decision criteria and operational Trade-Offs

Selecting or recommending a rendering mode during an audit involves balancing server response latency against build infrastructure limits and content freshness requirements.

• High-volume catalogs and editorial archives (ISR): When a site contains tens of thousands of product or article URLs, pre-rendering every page via pure SSG during CI/CD deployments causes excessive build times and deployment bottlenecks. ISR resolves this by generating a subset of high-priority routes at build time (or generating routes on demand upon the first request) and updating cached HTML in the background via time-based or on-demand revalidation ( revalidatePath or revalidateTag ). This preserves fast TTFB and complete initial HTML for crawlers without stalling deployments.
• Low-volume, rarely updated pages (SSG): Core marketing pages, documentation, and policy routes with low URL counts and infrequent edits are well-suited for pure SSG. Because the HTML is compiled ahead of time, server response time is decoupled from database performance during crawl spikes.
• Real-time inventory or request-dependent routes (SSR): Routes whose indexable content changes continuously or depends on request-level context require SSR. However, because uncached SSR executes server logic on every bot and user request, slow backend APIs directly inflate TTFB. If an SSR route shows high response latency during a crawl audit, adding an edge caching layer with short-lived s-maxage headers can reduce server load while keeping content reasonably current.
• Authenticated dashboards and non-indexable UI widgets (CSR): Pure client-side data fetching should be restricted to private user account areas, interactive calculators, or supplementary page elements (such as personalized recommendations or live pricing widgets) that do not need to be indexed as primary page content.

Auditing streaming and suspense boundaries in the app router

The App Router supports HTTP streaming via React Suspense boundaries and route-level loading.js files. When streaming is active, Next.js immediately sends the static layout shell and fallback UI (such as skeleton loaders) in the initial HTML chunks, keeping the HTTP connection open while slower asynchronous Server Components resolve on the server. Once the data fetches complete, the server streams the final HTML chunks along with inline <script> tags that swap the fallback markup with the resolved content in the DOM.

During an SEO audit, streaming requires specific verification to distinguish between server-streamed HTML and client-gated rendering:

1. Inspect the raw HTTP response stream: Fetch the URL using a command-line tool like curl -s <URL> , which waits for the HTTP response stream to close before outputting the full payload. Search the raw output for the page's primary heading and body text.
2. Locate fallback and resolved markup: If the route uses loading.js or <Suspense> , the top of the raw HTML document will contain the fallback loading state inside the main document container, while the actual indexable content will appear near the bottom of the payload inside hidden container elements (such as <div hidden id="S:..."> ) paired with inline replacement scripts.
3. Verify crawler visibility vs. client-only gating: Because streamed Server Components resolve within the initial HTTP response before the connection closes, crawlers receive the text in the raw payload, though replacing the fallback state inside the document tree still relies on executing the inline swap scripts. Conversely, if the primary text is absent from the completed curl output entirely, the component inside the boundary is not streaming from the server-it is fetching data on the client after hydration, which unnecessarily delays content visibility until secondary JavaScript rendering occurs.
4. Evaluate critical content placement: Even when server streaming functions properly, core SEO signals and primary body copy should ideally resolve before or outside non-essential Suspense boundaries so that slow third-party API calls do not hold the HTTP connection open or leave the initial document structure populated primarily with loading skeletons.

Auditing HTML parity and hydration failure modes

HTML parity means that the primary indexable text and internal links exist in both the raw server HTTP response and the final client-rendered DOM. While search engines such as Google can execute JavaScript to render client-side content, deferring primary copy or navigation links to browser execution introduces rendering dependencies, additional network requests, and potential processing delays. Furthermore, crawlers that do not execute JavaScript-or rendering pipelines interrupted by script errors or timeouts-rely entirely on the initial server markup. Delivering complete markup in the initial HTML response removes these client-side dependencies.

In Next.js, parity breaks when components render one state on the server and a materially different state in the browser after React hydration completes. Auditing this requires a structured comparison between the unexecuted response payload and the post-hydration DOM tree.

Step-by-Step HTML parity audit methodology

1. Compare the raw network response against the live DOM in browser developer tools. Open the Network panel, reload the target URL with the document filter active, and inspect the response body of the main document request. Alternatively, use the View Page Source browser command. Compare that unexecuted markup against the Elements panel, which displays the live DOM after React hydration and client-side scripts have run. Search both views for primary heading tags, main body copy, and navigation links to verify whether they arrive in the initial document or appear only after script execution.

2. Monitor the Console panel during initial page load for React hydration warnings. Messages indicating that text content does not match server-rendered HTML, or that hydration failed because the initial UI does not match what was rendered on the server, are direct diagnostic signals that the component tree diverged between the Node.js or Edge runtime and the browser environment.

3. Scale the parity check across route templates using a JavaScript-capable SEO crawler such as Screaming Frog or Sitebulb. Configure the crawler to execute JavaScript using a headless Chromium instance and set it to store both raw HTML and rendered HTML. After crawling representative templates, inspect the word count diffs, link diffs, and side-by-side HTML comparison views. Flag any route template where primary body text or crawlable links appear exclusively in the rendered HTML column.

4. Verify how Google processes the live route using the Google Search Console URL Inspection tool. Run a live test on the URL, open the View Tested Page panel, and examine the HTML tab alongside the More Info tab. The HTML tab displays the rendered DOM after Google's Web Rendering Service (WRS) executes page scripts. Check the Page Resources list and Console Messages under More Info to determine whether failed API fetches, blocked script bundles, or unhandled JavaScript exceptions prevented client-dependent components from populating the DOM during Google's rendering pass.

Common next.js parity and hydration failure patterns

When an audit reveals discrepancies between the raw HTML and the hydrated DOM, the cause typically traces back to how components manage lifecycle hooks, browser globals, or state initialization.

Client-Side fetching inside Server-Rendered routes

A frequent misconception during Next.js migrations is assuming that placing a component inside an App Router page or a Pages Router SSR route guarantees server-rendered HTML. If a child component declares 'use client' (or sits in the Pages Router) and triggers its primary API request inside a useEffect hook or a client-side data-fetching library without server-populated initial data, the server outputs only the initial empty or loading state. To restore parity when using client-side caching libraries such as SWR or React Query, fetch the payload on the server first and pass it into the provider or hook configuration as initial fallback data so the first HTML paint includes the resolved content.

Hydration mismatches and client recovery

Hydration requires the virtual DOM tree generated by React in the browser during its first pass to match the HTML tree produced by the server. When a component branches its return statement using browser-only objects (such as checking window.innerWidth to swap mobile and desktop navigation menus) or non-deterministic values (such as Math.random() or unseeded Date.now() formatting), the two trees diverge.

// Anti-pattern: Causes hydration mismatch and inconsistent link output
'use client';
export default function Navigation({ links }) {
  const isMobile = typeof window !== 'undefined' && window.innerWidth < 768;
  return isMobile ? <MobileDrawer /> : <DesktopNav links={links} />;
}

In this scenario, the server renders DesktopNav because window is undefined, while a mobile browser attempts to hydrate MobileDrawer . When a mismatch occurs, React discards the server-rendered subtree and re-renders the component entirely on the client. Beyond degrading main-thread performance, this pattern risks inconsistent content delivery: if the client-side recovery render fails due to an uncaught runtime error or a blocked script asset, the page may be left in a broken state, or the rendered DOM inspected by a search engine may omit links that were present in the desktop branch.

Server-to-Client state divergence

Parity issues also work in reverse when the server outputs complete content that client hydration subsequently strips out. This occurs when a server-rendered component renders a full article or product list, but a wrapping client component checks a client-side condition-such as an uninitialized state variable, a geo-location check, or a client-side A/B testing script-and replaces the server markup with an empty container or an interstitial component while waiting for a browser API response. If a search engine evaluates the final rendered DOM after executing JavaScript, the client-side state overrides the server HTML. Auditing must therefore confirm not only that indexable content exists in the initial HTTP payload, but also that hydration preserves that content in the final DOM tree.

Metadata output, head tag management, and canonicalization

Auditing document head elements-including <title> , meta descriptions, <meta name="robots"> directives, canonical tags, Open Graph tags, and JSON-LD structured data-requires verifying that these elements are present, deduplicated, and accurately formatted in the initial server HTML response. While search engines that execute JavaScript can process head tags injected on the client, relying on client-side execution introduces unnecessary points of failure. Furthermore, many social media scrapers, messaging link previewers, and AI crawlers do not execute JavaScript at all, reading only the raw initial HTTP payload. A technical audit must confirm how head tags are constructed across the specific Next.js router in use and verify that nested layouts or components do not output conflicting directives.

App router metadata API vs. pages router head management

Next.js handles head tag generation differently depending on the routing architecture. Understanding these mechanics helps isolate the root cause when an audit reveals missing, duplicated, or malformed tags.

Auditing the app router metadata API

In the App Router, Next.js evaluates metadata sequentially from the root segment down to the target leaf page.tsx . When auditing an App Router application, check for three common configuration issues: missing base URLs, shallow object overwriting, and streaming metadata behavior.

Validating metadatabase and absolute URLs

Canonical tags and Open Graph image tags should use absolute URLs to avoid ambiguity when parsers resolve paths. In the App Router, developers often define relative paths in route-level metadata (for example, alternates: { canonical: '/products/shoes' } ) and rely on the root layout.tsx to provide the domain via the metadataBase property.

// app/layout.tsx
import type { Metadata } from 'next';

export const metadata: Metadata = {
  metadataBase: new URL('https://example.com'),
  title: {
    template: '%s | Brand Name',
    default: 'Brand Name',
  },
  openGraph: {
    siteName: 'Brand Name',
    type: 'website',
  },
};

During an audit, inspect the raw HTML for <link rel="canonical"> and <meta property="og:url"> . If they output relative paths or http://localhost:3000 in production, the metadataBase property is either missing or reading from an unconfigured environment variable during the build or server execution process.

Shallow merging and accidental omission

The App Router merges metadata objects shallowly. If a root layout defines a complete openGraph object (containing siteName , locale , and default images ) and a child page exports its own openGraph object containing only a dynamic title and description , the child object completely overwrites the parent openGraph configuration. The resulting HTML will lack the og:site_name and og:image tags unless the child page explicitly re-declares them or spreads a shared configuration object. Crawling the site and filtering for pages missing specific Open Graph or robots properties across distinct route templates quickly exposes shallow-merge overrides.

Streaming metadata and initial head placement

In newer versions of Next.js, when generateMetadata awaits a slow data fetch, the framework can begin streaming the initial UI shell before the metadata resolves. For most user browsers, Next.js streams the resolved metadata tags into the document body and uses client-side DOM manipulation to hoist them into the <head> . For known search engine and social media bots (identified via the User-Agent header), Next.js blocks streaming until generateMetadata resolves so the tags appear directly inside the initial <head> markup.

When auditing with a custom script, curl, or an SEO crawler using a generic user agent string, you may observe metadata tags rendered outside the <head> near the bottom of the raw HTML payload. To validate how search engines receive the response, test the URL using a recognized crawler user agent (such as Googlebot ) and verify that the <title> , canonical, and robots tags sit cleanly inside the <head> element without requiring JavaScript execution.

Auditing the pages router and next/head failure modes

Applications built on the Pages Router (or hybrid setups migrating to the App Router) construct head elements by placing JSX tags inside <Head> components. Because multiple components in the render tree can mount their own <Head> instances simultaneously, Pages Router architectures are prone to tag duplication and parameter pollution.

Duplicate directives from omitted key props

If a global layout component renders a default <meta name="robots" content="index,follow" /> and a specific page template renders <meta name="robots" content="noindex,follow" /> inside its own <Head> block, Next.js will output both tags in the HTML unless they share an identical key attribute. While search engines generally follow the most restrictive directive ( noindex ) when conflicting robots tags are present, duplicate <meta name="description"> , canonical, or Open Graph tags create ambiguous signals.

// pages/account/settings.tsx (Pages Router)
import Head from 'next/head';

export default function SettingsPage() {
  return (
    <>
      <Head>
        {/* Matching the 'key' attribute from the layout prevents duplicate tags */}
        <meta name="robots" content="noindex,follow" key="robots" />
        <link rel="canonical" href="https://example.com/account/settings" key="canonical" />
      </Head>
      <main>...</main>
    </>
  );
}

To audit this, configure your SEO crawler to extract all instances of meta[name="robots"] , meta[name="description"] , and link[rel="canonical"] rather than just the first occurrence, or filter the crawler's canonical and meta reports for "Multiple" or "Duplicate" tag occurrences on a single URL.

Canonical tag pollution via router.aspath

A frequent canonicalization defect in the Pages Router occurs when developers construct self-referencing canonical URLs dynamically using the Next.js router hook's asPath property:

// Flawed implementation in Pages Router
import { useRouter } from 'next/router';
import Head from 'next/head';

export function SeoCanonical() {
  const router = useRouter();
  // DANGER: router.asPath includes all query strings and URL fragments
  const canonicalUrl = `https://example.com${router.asPath}`;
  
  return (
    <Head>
      <link rel="canonical" href={canonicalUrl} key="canonical" />
    </Head>
  );
}

Because router.asPath returns the exact path requested by the client-including query parameters-visiting https://example.com/shoes?utm_source=newsletter&sort=price_asc causes the server (or client) to output that exact parameterized URL as the canonical target. This defeats the purpose of canonicalization by endorsing every tracking parameter or filter combination as a distinct canonical URL.

Furthermore, if the page uses Static Site Generation ( getStaticProps ) without dynamic SSR data, router.isReady is false during the initial server render. Accessing query-dependent router properties prematurely can cause the initial HTML to output a fallback path (or an unpopulated dynamic route string like /products/[slug] ) before hydrating the actual URL on the client.

Verify canonical robustness during an audit by requesting sample URLs across each route template with arbitrary query parameters appended (for example, ?audit_test=123 ). Inspect the raw HTTP response to confirm two things:

1. The <link rel="canonical"> tag is present in the raw HTML source, not injected after client hydration.
2. The canonical href strips the ?audit_test=123 parameter and resolves to the clean, absolute target URL. In code, this requires either passing the clean canonical path directly from getStaticProps / getServerSideProps or stripping query strings from router.asPath.split('?')[0] when dynamic parameters are not meant to be indexed.

Late Client-Side metadata injection

In both routers, if a developer fetches page data inside a client-side useEffect hook and uses that state to populate <Head> (in the Pages Router) or directly mutates document.title , the initial server HTML will contain either empty head tags or generic site-wide defaults. Compare the raw HTML head against the rendered DOM head across dynamic routes (such as product detail or article pages). If the specific product title, meta description, or canonical URL appears only in the rendered DOM, the data fetching supporting that metadata must be moved to the server-via generateMetadata in the App Router or getServerSideProps / getStaticProps in the Pages Router.

Structured data (JSON-LD) implementation and validation

Neither the App Router Metadata API nor the Pages Router provides a dedicated schema property for JSON-LD structured data. Instead, JSON-LD is rendered as a standard <script type="application/ld+json"> tag inside the component tree or layout.

// Rendering JSON-LD in an App Router Server Component
export default async function ProductPage({ params }: { params: { id: string } }) {
  const product = await getProduct(params.id);
  
  const jsonLd = {
    '@context': 'https://schema.org',
    '@type': 'Product',
    name: product.name,
    image: product.imageUrl,
    description: product.description,
  };

  return (
    <section>
      <script
        type="application/ld+json"
        dangerouslySetInnerHTML={{ __html: JSON.stringify(jsonLd) }}
      />
      <h1>{product.name}</h1>
    </section>
  );
}

When auditing JSON-LD in Next.js, verify the implementation method and placement:

• Avoid using the Next.js <Script> component for JSON-LD: The next/script component is designed to optimize the loading of executable JavaScript bundles (using strategies like afterInteractive or lazyOnload ). Wrapping JSON-LD inside next/script defers its injection until after client-side hydration, keeping it out of the initial server HTML. Structured data should always be rendered using a standard lowercase <script> tag with dangerouslySetInnerHTML inside a Server Component or a server-rendered Pages template so it arrives in the initial payload.
• Check for duplicate schema payloads across nested layouts: In the App Router, placing an Organization or BreadcrumbList JSON-LD script in a parent layout and another inside a child page is valid, but auditors should verify that shared components do not accidentally mount duplicate Product or Article schema blocks with conflicting properties during route transitions.

Route output, HTTP status codes, and soft 404 prevention

Search engines rely on HTTP response status codes to determine whether a requested URL should be indexed, dropped from the index, or consolidated with another address. Because Next.js supports both server-side and client-side routing mechanics, an application can visually display an error screen or navigate a user to a new URL while still returning an HTTP 200 OK status header to the crawler. Auditing route responses requires verifying that missing or moved resources trigger true protocol-level status codes before the server flushes the response headers.

Returning true HTTP 404 and 3xx status codes

The mechanism for setting HTTP status codes on page routes depends on whether the application uses the App Router or the Pages Router.

In the App Router, server components and route handlers control response status codes by invoking helper functions imported from next/navigation :

• notFound() : Halts rendering of the current segment, renders the nearest not-found.js UI file, and sets the HTTP status code to 404 Not Found (provided the response headers have not already been sent).
• redirect(url) : Halts rendering and issues a temporary redirect. In Server Components, this returns a 307 Temporary Redirect status header. (When invoked inside a Server Action, it returns a 303 See Other .)
• permanentRedirect(url) : Halts rendering and issues a 308 Permanent Redirect status header, instructing crawlers that the resource has permanently moved and canonical signals should consolidate at the destination URL.

// App Router: app/products/[id]/page.tsx
import { notFound, permanentRedirect } from 'next/navigation';

export default async function ProductPage({ params }: { params: { id: string } }) {
  const product = await fetchProduct(params.id);

  if (!product) {
    notFound();
  }

  if (product.mergedIntoSlug) {
    permanentRedirect(`/products/${product.mergedIntoSlug}`);
  }

  return <h1>{product.name}</h1>;
}

In the Pages Router, HTTP status codes for page routes are controlled by the return object of server-side data-fetching functions ( getServerSideProps or getStaticProps ):

• Returning { notFound: true } instructs Next.js to render the pages/404.js template and serve an HTTP 404 status code. In static routes using getStaticPaths , setting fallback: false also guarantees that any unbuilt path immediately returns a 404 response.
• Returning a redirect object ( { redirect: { destination: '/new-url', permanent: true } } ) issues an HTTP 308 status when permanent is set to true , or a 307 status when set to false . (If getStaticProps returns a redirect during the build phase, Next.js cannot set HTTP headers dynamically at the component level and instead renders a client-side meta-refresh fallback unless the hosting platform intercepts it via route configuration.)

// Pages Router: pages/products/[id].tsx
export async function getServerSideProps({ params }) {
  const product = await fetchProduct(params.id);

  if (!product) {
    return { notFound: true };
  }

  if (product.mergedIntoSlug) {
    return {
      redirect: {
        destination: `/products/${product.mergedIntoSlug}`,
        permanent: true,
      },
    };
  }

  return { props: { product } };
}

For route migrations that do not depend on database lookups-such as changing a directory structure-defining redirects in next.config.js via the redirects array or handling them in middleware.ts executes the redirect at the edge or routing layer before page rendering begins.

Diagnosing soft 404s and Client-Side redirect failure modes

A soft 404 occurs when a server returns an HTTP 200 OK status code for a page that does not exist or contains only an error message. Search engines may either index these error pages or flag them as soft 404s in Google Search Console's Page Indexing report, leaving the URL's indexing state ambiguous.

When auditing a Next.js site, check for three implementation patterns that generate false 200 OK responses or delayed redirects:

1. Client-side error state rendering: When a route fetches its primary entity inside a Client Component (using useEffect , SWR, or React Query), the server immediately responds with a 200 OK status and an empty layout shell. If the subsequent browser-side API request returns a 404, the React component typically updates its state to render a visual "Product Not Found" message. To a crawler inspecting the initial response headers, the URL appears valid.
2. Conditional UI returns in Server Components without notFound() : Developers sometimes handle missing data in a Server Component by returning an inline JSX error message (for example, if (!data) return <div>Item not found</div>; ) instead of calling notFound() . Because a valid React element is returned without throwing the Next.js not-found signal, the server sends a 200 OK header.
3. Client-side redirects via router.push() : Triggering navigation inside a useEffect hook using useRouter().push('/new-path') or useRouter().replace('/new-path') does not send a 3xx HTTP status code. Instead, the server responds with a 200 OK for the original URL, requires the crawler to execute JavaScript to discover the destination URL, and prevents direct server-level consolidation of canonical signals.

Streaming edge cases and Late-Thrown errors

In the App Router, React Suspense boundaries and loading.js files enable HTTP streaming. When a route uses streaming, Next.js flushes the initial HTML shell along with an HTTP 200 OK status header to the client before asynchronous child components finish resolving their data.

An architectural edge case occurs when the primary data fetch for a page is placed inside a child component wrapped in a <Suspense> boundary (or when a route relies on a root loading.js file) and that component subsequently calls notFound() or redirect() . Because the HTTP 200 OK header has already been transmitted over the network with the initial chunk, the server cannot retroactively modify the HTTP status code to a 404 or 308 .

When notFound() is triggered after headers are flushed, Next.js injects a <meta name="robots" content="noindex"> tag into the streamed payload and swaps the UI to the not-found.js fallback on the client. Similarly, if redirect() is called inside a streamed boundary after headers are sent, Next.js falls back to injecting a client-side meta-refresh or script redirect rather than an HTTP 3xx header.

To verify and prevent this during an audit:

• Test non-existent dynamic URLs (for example, /products/invalid-slug-99999 ) using a command-line request like curl -I https://example.com/products/invalid-slug-99999 or the Network tab in browser developer tools with JavaScript disabled. Confirm the initial response status is 404 , not 200 .
• If a route returns 200 OK while still outputting a noindex tag or rendering the 404 UI, inspect the component tree. Ensure the data fetch that validates the existence of the primary route entity occurs in the top-level page.tsx or generateMetadata function outside of any <Suspense> wrapper, or remove the enveloping loading.js file if it forces premature header flushing before route validation completes.

Internal linking, discovery files, and system path directives

Search engine crawlers discover routes and evaluate site architecture primarily by extracting URLs from standard HTML anchor tags and XML sitemaps, while relying on CSS and JavaScript bundles to render page layouts. In a Next.js application, routing primitives and build asset directories require specific verification to ensure internal links are exposed in the raw HTML and system paths are not inadvertently blocked by robots.txt directives.

Auditing internal link output and the link component

Automated crawlers extract links by parsing <a> elements containing a resolvable href attribute. They generally do not trigger interactive DOM events to discover routes. A common audit failure in React and Next.js applications occurs when developers attach programmatic navigation-such as router.push('/target-page') -to an onClick handler on a <button> , <div> , or <span> element, or render an <a> tag without an href attribute. While these elements function for human users clicking in a browser, they do not expose a crawlable URL in the HTML source.

To ensure internal links are discoverable, navigation should be implemented using the next/link component. By default, next/link renders a standard <a href="..."> tag in the initial server HTML while intercepting clicks on the client to perform fast, client-side route transitions without a full page reload.

During an audit, verify internal link markup across primary navigation menus, pagination controls, product grids, and faceted filters:

• Inspect the raw HTML response (using view-source: or curl ) and confirm that every navigational link outputs an <a> element with a valid href path.
• Check codebases running Next.js 12.2 or earlier, or projects utilizing the legacyBehavior prop on <Link> . In older versions, wrapping a custom React component inside <Link> required adding the passHref prop so the underlying <a> tag received the href attribute. In Next.js 13 and later, <Link> renders an <a> tag directly without requiring a nested child anchor or passHref , unless a custom component is used alongside legacyBehavior .
• Reserve router.push() for non-navigational state transitions, such as redirecting a user after a form submission or login event, where crawler discovery is neither necessary nor desired.

Validating sitemaps and robots metadata files

How discovery files like robots.txt and sitemap.xml are generated and maintained depends on whether the project uses the App Router or the Pages Router.

App router native file conventions

The App Router provides built-in file conventions at the root of the app directory. You can provide static robots.txt and sitemap.xml files, or generate them programmatically using app/robots.ts and app/sitemap.ts . These TypeScript or JavaScript files export a default function returning typed objects ( MetadataRoute.Robots and MetadataRoute.Sitemap ) that Next.js compiles into standard text and XML responses.

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin/', '/api/'],
    },
    sitemap: 'https://example.com/sitemap.xml',
  }
}

For applications with tens of thousands of URLs, the App Router supports splitting sitemaps via the generateSitemaps function inside app/sitemap.ts , which outputs segmented sitemap endpoints (for example, /sitemap/0.xml , /sitemap/1.xml ). When auditing a site using generateSitemaps , verify that all generated child sitemap URLs are referenced either in a sitemap index file or explicitly listed in the sitemap array within robots.ts so crawlers can locate every segment.

Pages router dynamic XML routes

The Pages Router does not include native robots.ts or sitemap.ts conventions. Static files can be placed in the public/ directory, or generated at build time using third-party packages like next-sitemap . For dynamic sitemaps driven by a CMS or database, the standard pattern is creating a page file at pages/sitemap.xml.js that uses getServerSideProps to fetch URLs, set the Content-Type header to text/xml , write the raw XML string directly to the response stream ( res.write() ), and close the response ( res.end() ) while returning an empty props object.

When auditing discovery files across either router, check for three common configuration errors:

• Confirm the HTTP response headers for the sitemap include Content-Type: application/xml or text/xml .
• Verify that dynamic sitemaps in the App Router are not serving stale route inventories. By default, sitemap.ts is cached at build time unless it uses a dynamic function or exports a route segment config such as export const revalidate = 3600 to refresh the URL list periodically.
• Ensure the base URL used to construct sitemap entries and the Sitemap: directive in robots.txt pulls from the production domain environment variable rather than outputting a deployment preview URL (such as a *.vercel.app address) or http://localhost:3000 .

Auditing system paths and the /_next/ directory

Next.js serves compiled application assets, scripts, and route data from the reserved /_next/ path prefix. Misconfiguring robots.txt rules around this directory is a frequent cause of rendering failures during search engine crawls.

During an audit, inspect robots.txt immediately for a blanket Disallow: /_next/ directive. Technical teams sometimes add this rule after noticing /_next/data/ JSON URLs or JS chunk paths appearing in server logs or Google Search Console "Crawled - currently not indexed" reports. However, a blanket Disallow: /_next/ rule also blocks /_next/static/ and /_next/image . When search engines cannot fetch CSS and JavaScript bundles from /_next/static/ , their rendering engine sees an unstyled document and cannot execute any client-side JavaScript.

If an audit reveals that search engines are spending significant crawl activity on Pages Router JSON payloads-which often results in 404 errors in server logs whenever a new deployment changes the internal <build-id> hash-restrict the disallow rule specifically to the data directory while leaving static assets accessible:

User-agent: *
Allow: /_next/static/
Allow: /_next/image
Disallow: /_next/data/

To verify that system assets are accessible after updating robots.txt , run a live test in the Google Search Console URL Inspection tool on a representative page, open the "More info" panel under Page resources, and confirm that no CSS stylesheets or JavaScript bundles served from /_next/static/ are listed as blocked by robots.txt .