LynxDB

Log analytics in a single binary. No dependencies. Lynx Flow query language.

LynxDB is in active development. APIs, storage format, and query behavior may change without notice between releases. Feedback and contributions are welcome

Lynx Flow

Lynx Flow is LynxDB's query language - a pipeline language where data flows left-to-right through commands separated by |. Commands are named for what they do: parse, let, where, group, order by, take.

from nginx
| parse combined(_raw)
| status >= 500
| group by uri compute count() as hits, avg(duration_ms) as latency
| order by hits desc
| take 10

Quick start

curl -fsSL https://lynxdb.org/install.sh | sh

Development setup

LynxDB pins every required toolchain and CLI in mise.toml: Go, Bun, Rust, Node, Python, rclone, golangci-lint, GoReleaser, gh, and rsigma-cli. Install mise once, then:

curl https://mise.run | sh
mise install
mise run build
mise run test

Pipe logs through lynxdb - no server, no config:

# From raw logs to p99 latency in one line
kubectl logs deploy/api | lynxdb query '
      | group by endpoint compute avg(duration_ms), perc99(duration_ms)'

# Three nested formats, one pipeline, zero config
docker logs api-server 2>&1 | lynxdb query '
      | parse docker(_raw)
      | parse json(message)
      | explode errors
      | group by errors.code, errors.service compute count() as cnt
      | order by cnt desc | take 10'

# Wildcard array extraction - like jq, but with aggregation:
cat orders.json | lynxdb query '
      | json items[*].price AS price, items[*].product AS product
      | explode product, price                     
      | let revenue = price * qty                          
      | group by product compute sum(revenue) as total_revenue          
      | order by total_revenue desc''

Or run as a persistent server:

lynxdb server
lynxdb ingest nginx_access.log --source nginx_access --index balancer --batch-size 100000
lynxdb query '
      | parse combined(_raw)
      | method="POST" AND status < 300
      | parse json(request_body)
      | json items[*].sku AS skus
      | explode skus
      | group by skus compute count() as purchases, dc(client_ip) as unique_buyers
      | order by purchases desc
      | take 20'

Generate sample data and explore:

# Start the demo (streams realistic logs from 4 sources at 200 events/sec)
lynxdb demo

# Try in another terminal:
lynxdb query 'from nginx | group by status compute count()'
lynxdb query '| level="ERROR" | group by host compute count()' --since 5m
lynxdb tail 'level=ERROR'

Features

• Pipe mode - reads from stdin or files, works like grep. No server, no config.
• Lynx Flow - group, let, parse, order by, join, CTEs, domain sugar, and more. Partial SPL2 compatibility.
• Full-text search - FST inverted index + roaring bitmaps, bloom filters for segment skipping
• Columnar storage - custom .lsg format, delta-varint timestamps, dictionary encoding, Gorilla XOR, LZ4
• Materialized views - precomputed aggregations with automatic query rewrite, up to ~400x speedup
• Cluster mode - add --cluster.seeds to go distributed; S3-backed shared storage
• Drop-in ingestion - Elasticsearch _bulk, OpenTelemetry OTLP, Splunk HEC
• Sigma rule support - run rsigma-generated SPL2, for example lynxdb query "$(rsigma convert -t lynxdb rule.yml)"; see docs/site/docs/sigma

Comparison

Configuration

Zero config needed - sensible defaults for everything. Customize in ~/.config/lynxdb/config.yaml. The values below are the defaults - override only what you need:

listen: "localhost:3100"
data_dir: "~/.local/share/lynxdb"   # $XDG_DATA_HOME/lynxdb if set
retention: 7d
log_level: info

storage:
  compression: lz4                  # lz4 | zstd
  cache_max_bytes: 1gb

Cascade: CLI flags -> LYNXDB_* env vars -> config file -> defaults.

listen: "localhost:3100"
data_dir: "~/.local/share/lynxdb"   # $XDG_DATA_HOME/lynxdb if set
retention: 7d
log_level: info

storage:
  compression: lz4          # lz4 | zstd
  flush_threshold: 512mb
  cache_max_bytes: 1gb
  s3_bucket: ""             # no default - set to enable S3 tiering
  s3_region: us-east-1

query:
  max_concurrent: 32
  max_query_runtime: 5m
  default_result_limit: 1000

ingest:
  max_body_size: 100mb

CLI reference

lynxdb server                start server
lynxdb query <query>         run a query (Lynx Flow or SPL2)
lynxdb tail <query>          live tail
lynxdb ingest <file>         ingest a file
lynxdb shell                 interactive REPL (tab completion, /help, PgUp scrollback)
lynxdb count <query>         quick event count
lynxdb sample N <query>      peek at data shape
lynxdb watch <query> -i 5s   periodic refresh with deltas
lynxdb diff <query> -p 1h    this period vs previous period
lynxdb explain <query>       query plan without executing
lynxdb mv create/list        materialized views
lynxdb status                server metrics
lynxdb bench                 benchmark
lynxdb demo                  generate sample data

Build from source

git clone https://github.com/lynxbase/lynxdb && cd lynxdb
go build -o lynxdb ./cmd/lynxdb/
go test ./...

Contributing

See CONTRIBUTING.md

Feedback

• Issues

LynxDB wouldn't exist without the projects that inspired it:

• Splunk - for creating SPL, the most expressive log query language. LynxDB's SPL2 compatibility and Lynx Flow design owe everything to Splunk's query model.
• ClickHouse - for proving that a single-binary analytical database with incredible performance is possible. The MergeTree architecture deeply influenced LynxDB's storage engine design.
• VictoriaLogs - for showing that log analytics can be resource-efficient and operationally simple.
• grep, awk, sed - for the Unix philosophy of composable tools and piping. LynxDB's pipe mode is a direct homage to this tradition.

This project started in early 2025 out of a deep appreciation for these tools and a desire to bring Splunk-level analytics to everyone in a single, lightweight binary.

Star History

License

Apache 2.0