@secondlayer/subgraphs

June 9, 2026 · View on GitHub

Typed on-chain indexing for Stacks. Declare event filters + column schema with defineSubgraph(); the runtime decodes blocks, matches filters, runs your handlers inside a transactional context, and exposes the result as a Postgres schema you query over REST or SQL.

Subgraphs have a visibility: managed deploys default visibility: public — anon-readable at /v1/subgraphs/<name>/<table>, in a single global public namespace claimed on publish (409 PUBLIC_NAME_TAKEN if the name is taken). BYO-database deploys default private. Override with --visibility public|private at deploy, or flip later with sl subgraphs publish|unpublish <name>.

Subgraph rows fan out to HTTP subscribers through a post-flush outbox emitter — signed Standard Webhooks POSTs with retries, circuit breaker, and replay.

Install

bun add @secondlayer/subgraphs

Quick Start

For the full hosted beta loop, including project setup, fast deploys with --start-block, querying, and subscriptions, start with QUICKSTART.md.

import { defineSubgraph } from "@secondlayer/subgraphs";

export default defineSubgraph({
  name: "token-transfers",
  version: "1.0.0",
  sources: {
    // Named event sources — the key becomes the handler name.
    transfer: {
      type: "ft_transfer",
      assetIdentifier: "SP2X0TZ59D5SZ8ACQ6YMCHHNR2ZN51Z32E2CJ173.stx-token::stx",
    },
  },
  schema: {
    transfers: {
      columns: {
        sender: { type: "principal" },
        recipient: { type: "principal" },
        amount: { type: "uint" },
      },
      // Auto-added: _block_height, _tx_id, _created_at
    },
  },
  handlers: {
    async transfer(event, ctx) {
      ctx.insert("transfers", {
        sender: event.sender,
        recipient: event.recipient,
        amount: event.amount,
      });
    },
  },
});

Deploy via CLI (sl subgraphs deploy path/to/definition.ts), SDK (sl.subgraphs.deploy({...})), or MCP (subgraphs_deploy). The CLI can also scaffold from a deployed contract with sl subgraphs scaffold <contract> -o subgraphs/name.ts; scaffold writes or amends package.json and runs bun install unless --no-install is passed. The dashboard is read-only — creation always happens through an API surface.

Exports

Subpath	Description
`.`	`defineSubgraph`, `validateSubgraphDefinition`, `deploySchema`, `diffSchema`, `reindexSubgraph`, `backfillSubgraph`, `generateSubgraphSQL`, `pgSchemaName`
`./types`	All schema + filter + handler types (`SubgraphDefinition`, `SubgraphFilter`, `StxTransferFilter`, etc.)
`./schema`	Generator + deployer internals
`./validate`	Shape + filter validation for deploys
`./triggers`	Typed `on.*` helpers for all `SubgraphFilter` variants
`./runtime/source-matcher`	Pure fn: match txs+events against a `SubgraphFilter` — used by the processor hot path
`./runtime/replay`	`replaySubscription({ accountId, subscriptionId, fromBlock, toBlock })` — re-enqueue historical rows as outbox entries

Runtime components

The runtime ships behind these entrypoints (import from the package root):

startSubgraphProcessor(opts?) — boots the block processor. LISTENs on indexer:new_block, matches sources, runs handlers, flushes writes inside a transaction, and emits outbox rows for matching subscriptions. Also boots the emitter worker.
processBlock(subgraph, name, height, opts?) — single-block entry point used by catch-up, reindex, and tests.
catchUpSubgraph(def, name) — drains pending blocks up to chain tip.
reindexSubgraph(def, opts) — drop + rebuild schema tables from a start block. Breaking schema changes trigger this automatically on deploy.

Subscription emitter

Every row written through ctx.insert() / ctx.upsert() is atomically enqueued to subscription_outbox for every active subscription whose filter matches — inside the same transaction as the flush, so a processor crash rolls back both.

The emitter drains the outbox via LISTEN subscriptions:new_outbox and FOR UPDATE SKIP LOCKED batch claims. Live deliveries win a 90/10 split over replays. Each row dispatches through the format builder matching the subscription's format column (standard-webhooks, inngest, trigger, cloudflare, cloudevents, raw). Retries follow 30s → 2m → 10m → 1h → 6h → 24h → 72h. Twenty consecutive failures trips the per-sub circuit breaker and pauses the subscription.

Delivery bodies and response previews land in subscription_deliveries. Rows whose retries exhaust mark status = 'dead' in the outbox and surface in the dashboard's dead-letter queue for one-click requeue.

Environment

Variable	Default	Description
`SECONDLAYER_EMIT_OUTBOX`	`true`	Set `false` to bypass outbox emission on every block (kill-switch).
`SECONDLAYER_ALLOW_PRIVATE_EGRESS`	`false`	Allow the emitter to deliver to private IP ranges (localhost, 10/8, 172.16/12, 192.168/16, link-local, v6 mapped). Leave off in production.
`SECONDLAYER_SECRETS_KEY`	—	32-byte hex key for the AES-GCM envelope around subscription signing secrets. OSS mode auto-generates + persists to `.env.local`.
`TENANT_PLAN`	unset	Dedicated hosting plan injected by the provisioner. Unset and paid plans use standard batches.
`SUBGRAPH_REINDEX_BATCH_SIZE`	plan-based	Override the default historical block batch size used by reindex/backfill.
`SUBGRAPH_REINDEX_MIN_BATCH_SIZE`	plan-based	Override the adaptive lower bound for reindex/backfill batches.
`SUBGRAPH_REINDEX_MAX_BATCH_SIZE`	plan-based	Override the adaptive upper bound for reindex/backfill batches.
`DATABASE_MAX_POOLS`	`25`	Max cached connection pools. With BYO subgraphs (one pool per user DB) the least-recently-used pool is evicted past this cap; the source/target pools are never evicted.
`DATABASE_IDLE_TIMEOUT`	`300`	Seconds before idle connections are closed (`0` = never). Keeps a fleet of BYO pools from pinning connections.

Bring your own database (BYO data plane)

There are three tiers, not two:

Tier	Indexer / decode	Handler exec	Database	Serving API
Managed (default)	ours	ours	ours	ours
BYO data plane	ours	ours	yours	yours
Self-host	yours	yours	yours	yours

With BYO, the managed pipeline (ingest → decode → match → run your handler) is unchanged, but your handler's rows land in your Postgres and the serving API reads from there. Deploy with a connection string:

sl subgraphs deploy subgraphs/my.ts --database-url "postgres://user:pass@your-host:5432/db"

BYO deploys default visibility: private (managed deploys default public); pass --visibility public or sl subgraphs publish <name> to open reads.

The connection string is stored encrypted at rest (AES-GCM, keyed by SECONDLAYER_SECRETS_KEY) and never returned in API responses. The server verifies the connection before deploying. Once deployed, query the subgraph_… schema directly with any ORM/GraphQL/REST — we're no longer in the serving path.

Preview before it touches your DB. POST /api/subgraphs with {"dryRun": true, "databaseUrl": "…"} returns the exact DDL plus a grant script and verifies the connection — without writing anything.

Constraints (v1):

Idempotent handlers only. A BYO block write can't share the managed transaction, so a crash replays the block (at-least-once). ctx.insert and ctx.upsert (with a unique key) are safe — flush is replace-per-height. A deploy with non-idempotent ctx.update / ctx.patchOrInsert is rejected.
No reindex. A breaking change (removed table/column, changed type, or a forced reindex) would drop + rebuild the schema in your DB, so the deploy is refused with HTTP 422 — nothing is touched. The refusal carries a migration plan: the SDK throws a typed ByoBreakingChangeError (details.reasons + details.plan with dropStatement/statements/grantScript) and the CLI prints the exact DROP SCHEMA … CASCADE + rebuild DDL to run yourself, then re-deploy. A destructive --force rebuild on your DB is not yet supported.
Delete leaves your data. Deleting the subgraph removes our registry row (and the stored connection) and pauses subscriptions, but never drops the schema in your database.

Recommended: give Secondlayer a least-privilege role scoped to its own schema, and point --database-url at a session-mode pooler endpoint (PgBouncer/Neon/Supabase) rather than raw Postgres.

ORM codegen

Once rows land in your DB, generate a typed schema for your ORM:

sl subgraphs codegen subgraphs/my.ts --target prisma  -o prisma/schema.prisma
sl subgraphs codegen subgraphs/my.ts --target drizzle -o db/schema.ts

Prisma and Drizzle have first-class generators (generatePrismaSchema / generateDrizzleSchema are exported); both emit relations/@relation from the schema's relations metadata. For Kysely, run kysely-codegen against the DB. Output mirrors the deployed DDL — prisma db pull should be a no-op; treat the tables as read-only (the processor owns them) and never migrate/push. uint→Decimal/numeric and the BigInt id need .toString() for JSON.

Trait-scoped sources

A source can target a SIP standard instead of a fixed contract — it indexes every contract the registry classifies as that standard (incl. ones deployed later):

sources: {
  tokens: { type: "ft_transfer", trait: "sip-010" }, // all SIP-010 tokens
}

trait (sip-009 | sip-010 | sip-013) is supported on FT/NFT/contract_call/ print_event filters and composes (AND) with other fields. Token filters match the asset-identifier's contract; contract_call/print match contract_id. Resolution is as-of-block, so a reindex backfills a contract's full history even if it was classified after deploy. Requires the contract registry to be populated. Discover the set via GET /v1/contracts?trait=sip-010.

Postgres + pool mode

The emitter holds a persistent LISTEN on subscriptions:new_outbox and subscriptions:changed, so it MUST connect through a session-mode pool. pgbouncer in transaction mode silently breaks it. Run the emitter against a session-mode port (pool_mode = session), or connect directly to Postgres as the default docker-compose setup does.

License

MIT