Node.js Integration via NPM

May 13, 2026 · View on GitHub

spiderable-middleware is the Node.js package that routes bot traffic from your app to the ostr.io pre-rendering engine.

What it does

spiderable-middleware inspects incoming requests in your Node.js app. When the request comes from a known crawler, social previewer, or AI fetcher, it is transparently proxied to the ostr.io rendering endpoint with your Basic auth credentials. Regular browser traffic continues straight to your application, untouched.

When to use

  • You run a Node.js app and prefer an in-process integration over a reverse-proxy or edge solution.
  • You want pre-rendering decisions to live in your app code alongside other request handling.
  • You need compatibility with any of the Node middleware signatures — Express, Connect, Koa, Fastify (via adapter), NestJS, h3, or a vanilla http server.

For infrastructure-level alternatives that do not require editing application code, see Related below.

Legacy <meta name="fragment"> markup (optional)

The legacy _escaped_fragment_ crawling scheme was deprecated by Google in 2015 but is still honored by some non-Google crawlers. If you care about those edge cases, add the meta tag to your HTML shell:

<html>
  <head>
    <meta name="fragment" content="!">
  </head>
</html>

Modern crawlers rely on the User-Agent check that spiderable-middleware performs automatically, so this markup is not required for Googlebot, Bingbot, facebookexternalhit, meta-externalagent, AI fetchers, or any crawler that advertises itself via User-Agent.

For Node.js apps you can also route bot traffic outside the app layer, without adding any dependency: