TOON Format Documentation

September 10, 2026 · View on GitHub

Token-Oriented Object Notation (TOON)

Overview

TOON (Token-Oriented Object Notation) is a compact serialization format for structured data such as dependency graphs, symbol lists, and file analysis results. Its size relative to JSON depends on the payload.

Format Specification

Structure

TOON consists of two parts:

  1. Header: Defines the object name and field names

    objectName(field1,field2,field3)

  2. Data Lines: One line per object, with values in brackets

    [value1,value2,value3]

Example

JSON (Original):

[
  { "file": "main.ts", "deps": ["fs", "path"], "line": 10 },
  { "file": "utils.ts", "deps": ["os", "crypto"], "line": 20 }
]

TOON (Compact):

files(file,deps,line)
[main.ts,fs|path,10]
[utils.ts,os|crypto,20]

Features

1. Array Serialization

Arrays within values are joined with pipe | delimiter:

  • ["fs", "path"] → fs|path
  • ["a", "b", "c"] → a|b|c

2. Escaping

Special characters are escaped when escapeValues: true (default):

  • Comma: , → \,
  • Pipe: | → \|
  • Brackets: [ → \[, ] → \]
  • Backslash: \ → \\

3. Type Preservation

TOON automatically handles:

  • Numbers: 42, 3.14, -5
  • Booleans: true, false
  • Strings: Any text
  • Arrays: Multi-value fields with pipe delimiter
  • Objects: Serialized as JSON strings
  • Empty values: Empty string ''

Token Savings

estimateTokens() uses gpt-tokenizer/encoding/cl100k_base. The resulting counts measure serialized representation size; they are not provider billing tokens.

In the fixed graph-context corpus run on 2026-09-04, six JSON responses totaled 8,780 cl100k_base tokens and the corresponding TOON encodings totaled 1,072, a measured reduction of 87.8%. That example applies only to that corpus and run. Different schemas and values can produce smaller savings or make TOON larger than JSON.

Measurement Protocol and Limits

TOON changes the representation returned by a local analysis. It does not make architecture, codemap, impact, or call-graph analysis depend on an LLM. estimateTokenSavings() compares JSON and TOON with the shared cl100k_base tokenizer. Keep these representation counts separate from provider-reported usage.

Run the reproducible local corpus with:

npm run test:context-economy

It removes known LLM credentials from child processes, runs six graph-context workflows in JSON and TOON on a fixed twelve-file corpus, and writes raw output plus report.json under .reports/context-economy/. The report records quality, bounds, freshness, latency, and cl100k_base representation counts. MCP initialization, MCP tool-call, continuation, and provider billing-token metrics stay null because the independent CLI runner cannot observe them.

The optional natural-language query command is the narrow exception: an available provider can help extract search keywords. Its graph scoring and BFS remain local, and missing credentials activate the heuristic keyword fallback.

Graph context encoding

graphitlive_graph_context uses separate TOON blocks so an agent can consume the response without repeating field names:

graph_context(indexRevision,fresh,mode,tokenEstimate,truncated,nextCursor)
seeds(id,kind,name,path,startLine,endLine,language,score,isSeed)
nodes(id,kind,name,path,startLine,endLine,language,score,isSeed)
edges(source,target,relation,confidence,sourcePath,sourceLine,sourceEndLine,evidence)
paths(nodeIds,edgeIndexes,hops)
ambiguous(node,score,reason)
omitted(nodes,edges)
nextQueries(query)
errors(message)

The gateway budgeter and tokenEstimate use gpt-tokenizer/encoding/cl100k_base. The # Token Savings footer compares serialized JSON and TOON with the same encoding. Neither value is provider billing usage or a universal percentage. The gateway preserves requested seeds and path endpoints; when the budget is exceeded it reports omissions and exposes an opaque cursor for continuation.

Usage in MCP Server

Request Format Parameter

All MCP tools now accept a format parameter:

{
  "tool": "crawl_dependency_graph",
  "params": {
    "entryFile": "/path/to/main.ts",
    "format": "toon"
  }
}

Available formats:

  • json (default): Standard JSON output
  • toon: Compact TOON format
  • markdown: JSON wrapped in markdown code blocks

Automatic Format Suggestion

The server automatically suggests TOON format for large datasets (>10 items).

Fixed — Breaking: query_natural_language toon field (v1.1.0, ADR-S2-01)

Prior to this fix, the query_natural_language MCP tool's outputFormat: 'toon' branch (the default) returned a JSON passthrough of the analyzer's compact payload — not real TOON — despite the field being named toon. A client doing JSON.parse(result.toon) worked by accident; a client expecting header+rows TOON got JSON instead.

The field name toon is unchanged (see docs/architecture/ADR-S2-01-toon-field-mcp-compat.md for the rationale), but its content is now a real TOON encoding: a #-prefixed meta line followed by two TOON blocks (nodes(...) / edges(...)).

Before (JSON passthrough, mislabeled as toon):

{"nodes":[{"id":"a","n":"funcA","t":"function","p":"src/a.ts","l":10,"r":1}],"edges":[{"src":"a","tgt":"b","rel":"CALLS"}],"nodeCount":1,"edgeCount":1,"truncated":false}

After (real TOON):

# nodeCount=1 edgeCount=1 truncated=false
nodes(id,n,t,p,l,r)
[a,funcA,function,src/a.ts,10,1]
edges(src,tgt,rel)
[a,b,CALLS]

Detect the fixed format by the leading # nodeCount=... edgeCount=... truncated=... meta line, or by the presence of nodes(...) / edges(...) TOON headers. The MCP_TOOL_VERSION constant (src/mcp/types.ts) was bumped 1.0.0 → 1.1.0 as an informational signal only — it is not read by the MCP SDK for protocol negotiation.

Token Savings Metadata

When using TOON format, responses can include representation counts. This is an illustrative shape, not a benchmark result:

files(file,deps,line)
[main.ts,fs|path,10]
[utils.ts,os|crypto,20]

# Token Savings
JSON: 125 tokens
TOON: 48 tokens
Savings: 77 tokens (61.6%)

API Reference

jsonToToon(data, options)

Converts JSON array to TOON format.

Parameters:

  • data: unknown[] - Array of objects to convert
  • options?: ToonOptions
    • objectName?: string - Name for the object type (default: 'data')
    • escapeValues?: boolean - Whether to escape special characters (default: true)

Returns: string - TOON formatted string

Example:

import { jsonToToon } from '@/shared/toon';

const data = [
  { file: 'main.ts', line: 10 },
  { file: 'utils.ts', line: 20 },
];

const toon = jsonToToon(data, { objectName: 'files' });
// Result: "files(file,line)\n[main.ts,10]\n[utils.ts,20]"

toonToJson(toonStr, options)

Parses TOON format back to JSON array.

Parameters:

  • toonStr: string - TOON formatted string
  • options?: ToonOptions - Same as jsonToToon

Returns: unknown[] - Parsed array of objects

Example:

import { toonToJson } from '@/shared/toon';

const toon = 'files(file,line)\n[main.ts,10]\n[utils.ts,20]';
const data = toonToJson(toon);
// Result: [{ file: 'main.ts', line: 10 }, { file: 'utils.ts', line: 20 }]

estimateTokenSavings(jsonStr, toonStr)

Counts JSON and TOON representations with cl100k_base and reports their difference.

Parameters:

  • jsonStr: string - JSON formatted string
  • toonStr: string - TOON formatted string

Returns: Object with:

  • jsonTokens: number - cl100k_base tokens for JSON
  • toonTokens: number - cl100k_base tokens for TOON
  • savings: number - Token difference
  • savingsPercent: number - Percentage saved

Best Practices

When to Use TOON

✅ Use TOON for:

  • Large arrays (>10 items)
  • Structured data (dependencies, symbols, nodes)
  • Repeated API calls with similar data
  • Token-sensitive operations

❌ Avoid TOON for:

  • Small datasets (<5 items)
  • Highly nested structures
  • One-off queries
  • Human-readable output

Limitations

  1. Single-Element Arrays: Arrays with one element (e.g., ["os"]) are indistinguishable from strings after parsing. Use multi-element arrays for proper array detection.

  2. Nested Objects: Complex nested objects are serialized as JSON strings within TOON, reducing efficiency.

  3. Type Loss: Some type information may be lost in round-trip conversion (e.g., null becomes '').

Integration Example

MCP Tool Handler

import { formatDataAsToon } from '@/mcp/responseFormatter';

// In your tool handler
const result = await analyze(filePath);

// Format based on user preference
const formatted = params.format === 'toon'
  ? formatDataAsToon(result.dependencies, 'dependencies')
  : JSON.stringify(result, null, 2);

return { content: formatted };

Extension Usage

import { jsonToToon, toonToJson } from '@/shared/toon';

// Send data in TOON format
const toon = jsonToToon(dependencies, { objectName: 'deps' });
await sendToMcp(toon);

// Receive and parse TOON data
const received = await receiveFromMcp();
const data = toonToJson(received);

Testing

Comprehensive test suite available in:

  • tests/shared/toon.test.ts - Core TOON functionality
  • tests/mcp/responseFormatter.test.ts - Integration with MCP

Run tests:

npm test -- tests/shared/toon.test.ts
npm test -- tests/mcp/responseFormatter.test.ts

Contributing

When modifying the TOON module:

  1. Ensure cross-platform compatibility (Windows, Linux, macOS)
  2. Add tests for new features
  3. Update token savings benchmarks
  4. Run security scan: npm run snyk
  5. Verify types: npm run check:types
  6. Lint: npm run lint