Separating search observation from search mutation in headless Sitecore

We thought we had a crawler performance problem. On a Sitecore Search deployment with more than 10,000 pages, content updates were taking hours to appear in search results: a full crawl and rebuild ran about 1.5 hours, and another hour or more of processing could pass before users saw the update. That was not acceptable for news and high-visibility content.

We looked at crawl scope, segmented indexes, exclusion rules, scheduling, and ways to reduce the amount of content passing through the pipeline, but none of it changed the basic shape of the problem. A crawler run still meant observing a large surface area of the site, extracting documents from that surface area, and pushing those documents through the search pipeline, so even when only a handful of pages had changed, the crawl model still behaved like a large batch operation.

The recent update for delta crawling seemed promising, but it did not change the ceiling much. The fastest recrawl interval we could get was still one hour, and because lastmod carries a date without a time, multiple publishes to the same page in one day could collapse into a single observed change.

Eventually we started asking different questions: if Next.js is serving SSG pages from a cache, can we read that cache? How is that cache built? Can we get the same write but to a different service?

A normal crawler observes, extracts, and indexes as one coupled workflow. The pattern we are exploring keeps observation, but moves document construction and index mutation behind a canonical hash boundary. The crawler can still hit rendered URLs without pushing every observed page through indexing.

Search freshness should scale with change rate, not crawl rate.

The crawler today

URL → rendered page → CEC extractor → search document → index

A web crawler requests a rendered URL and builds a search document from the result. Google does not index your CMS database; it indexes what the page returns.

Observation, extraction, and indexing are coupled together:

Crawler observes → crawler extracts → crawler indexes

At small scale that coupling is convenient. At larger scale it is expensive, because a 10,000-page crawl still behaves like a 10,000-page indexing operation even when only a small number of rendered experiences actually changed.

The webhook path has the same shape in a different order. A publish alert bundles discovery ("something changed") with ingestion ("push a document"), often from CMS intent rather than a resolved page. Rendering may happen later, in parallel, or not at all.

Discovery, render, ingestion

A publish alert can enqueue ingestion from CMS intent before the rendered page is known. The crawler path at least renders first, but still indexes on every observation:

Alert path:   publish alert → ingest          (render optional or raced)
Crawler path: crawl hit → render → ingest     (every observation indexes)

In both models, ingestion rate tracks alert rate or crawl rate. That is what breaks down at 10,000 pages.

The question is not whether to decouple discovery from ingestion. It is what ingestion should wait for. A publish signals intent; after caches, composition, and datasource resolution settle, the resolved page is the only trustworthy source for a search document.

The target model treats them as three phases:

Discovery → Render resolution → Ingestion

Each phase has a narrow responsibility and an explicit boundary:

Discovery can be frequent. Ingestion should be rare.

Observation is not indexing

Crawl rate and indexing workload do not have to track together. A crawler may observe 10,000 pages while only 40 rendered search documents actually changed:

10,000 observed pages → 40 changed search documents → 40 ingestion writes

The rendering app as document owner

Today the crawler's extractor owns document construction:

Crawler visits rendered page → CEC extractor builds document → search pipeline indexes

We are exploring a split where the rendering application owns the canonical search document contract:

Crawler / ISR / publish / manual action → observe resolved experience
Rendering app builds canonical search document → hash compare
Only changed documents → ingestion queue

The rendered experience is often more than a CMS item. In headless, the page a user receives can depend on component composition, datasource resolution, aggregated content, feature flags, external APIs, personalization, and rendering logic. A publish event tells us intent changed, but it does not necessarily tell us the final experience has changed, and the rendering layer is where that experience is resolved. That makes it the natural place to own a versioned, testable buildSearchDocument contract.

Hashing is the boundary

Without a canonical hash, tying search updates to crawls or renders sounds dangerous, because it implies every request, crawler hit, or revalidation might create an indexing write. With a canonical search document hash, the flow is:

Generate → Compare → Skip     (most of the time)
Generate → Compare → Enqueue → Ingest   (when changed)

Push rate becomes bounded by change rate, not request rate. A page can render hundreds or thousands of times, and if the canonical search document is identical, nothing is pushed.

Experience-validated indexing

We looked at a few patterns that did not work out. A dirty-route registry (mark a route stale, fetch it later for search) means rendering the page twice, once for users and once for discovery, at which point you are already doing the work a single resolved render could do.

The cleanest model we found is to use the page resolution that is already happening. In a headless Next.js implementation, many Sitecore pages flow through the same dynamic route, and that resolution already has the data needed to produce both the page and the search document.

Request or crawler hit
→ dynamic route resolves Sitecore data
→ page props / rendered output are produced
→ canonical search document is produced from the same resolved data
→ document hash is compared against the last indexed hash
→ unchanged documents are skipped
→ changed documents are queued for ingestion

A publish webhook may still trigger revalidation or encourage a route to be observed, but the webhook is not the moment where the final search document is trusted. The trusted moment is when the rendering host resolves the page, and the goal is for that same route resolution to produce a canonical search document rather than rendering once for users and again for search.

Most route resolutions produce no ingestion call. Observations can be frequent; pushes should be bounded by changed search documents.

Ingestion stays off the render path: build the canonical document from resolved route data, hash-compare, enqueue if changed, and let a worker handle queue, rate limiting, retry, and the Search Ingestion API so rendering does not wait on Search availability. The data is already there from route resolution; the rendering pipeline should not pay for a second pass just for search.

What the crawler becomes

The crawler does not disappear. It changes jobs.

Today:  Crawler observes → crawler extracts → crawler indexes
 
Target: Crawler observes → app validates changed experience → ingestion indexes only changed documents

The crawler still has value for discovery, coverage, validation, recovery, and low-traffic pages, but it is no longer the component that must own the final document shape or push every observed page through indexing.

What would make this more principled

Neither gap blocks production since both have workarounds today, but cleaner platform support would reduce implementation complexity.

On Pages Router, getStaticProps already resolves route data at revalidation time. Search work can run from those props; the awkward part is scheduling ingestion without blocking the static generation response.

On App Router, after() schedules work after the response finishes. It does not surface rendered content on its own. You build the search document from the same route resolution as the page and pass it into after() via closure for hash compare and enqueue. On static or ISR routes, that callback runs at build time or revalidation, not on every cache hit. A crawler observation only reaches that path if it triggers revalidation first.

On Sitecore Search, indexing sources split into two models. A crawler source visits rendered URLs and runs the CEC extractor to build search documents. A push source expects documents pushed through the Ingestion API. There is no ping-only discovery mode on a push source: it cannot observe a route without receiving a document. Workarounds today include a separate crawler source to trigger ISR, or an external sitemap walker on a schedule.

This is not proven

This is still a theory, not production-proven. We are exploring this approach as an alternative indexing method to get near-immediate updates with fully populated pages.