Server-side conversion (never round-trips through stryke memory).

June 24, 2026 · View on GitHub

 ███████╗████████╗██████╗ ██╗   ██╗██╗  ██╗███████╗
 ██╔════╝╚══██╔══╝██╔══██╗╚██╗ ██╔╝██║ ██╔╝██╔════╝
 ███████╗   ██║   ██████╔╝ ╚████╔╝ █████╔╝ █████╗
 ╚════██║   ██║   ██╔══██╗  ╚██╔╝  ██╔═██╗ ██╔══╝
 ███████║   ██║   ██║  ██║   ██║   ██║  ██╗███████╗
 ╚══════╝   ╚═╝   ╚═╝  ╚═╝   ╚═╝   ╚═╝  ╚═╝╚══════╝
                   [ a r r o w ]

CI License: MIT stryke

[APACHE ARROW + PARQUET + FEATHER + ARROW-CSV/JSON // STRYKE PACKAGE]

"Columnar data, on demand. No daily-driver weight."

Apache Arrow + Parquet + Arrow IPC + Feather + arrow-CSV + arrow-JSON for stryke. Opt-in package, kept out of the stryke core binary so the daily-driver install stays slim. Created by MenkeTechnologies.

strykelang · MenkeTechnologiesMeta · stryke-parquet · stryke-duckdb · stryke-demo

Read the Docs · Engineering Report

Table of Contents

[0x00] Why a Package, Not a Builtin

stryke's core stays small on purpose — most one-liner / awk-replacement work doesn't need 200 transitive crates of columnar data infrastructure. arrow-rs + parquet hit a different scale:

TierPropertiesThis package
Core builtins (~40 MB stryke)small deps, used everywherestring, math, regex, parallel ops, scipy-class math
Package tier (opt-in)heavy deps, narrow use casesparquet, arrow, big-ML, cloud SDKs, niche formats

stryke-arrow ships as a local stryke package + a Rust cdylib (libstryke_arrow.{dylib,so}) built from this repo. The stryke side is a thin FFI wrapper; the heavy arrow-rs/parquet code lives in the cdylib and is dlopened on demand. Core stryke is never linked against arrow.

[0x01] Install

From a release (no rustc on the consumer machine):

s pkg install -g github.com/MenkeTechnologies/stryke-arrow

From a local checkout (publisher / contributor workflow):

cd ~/projects/stryke-arrow
cargo build --release           # produces target/release/libstryke_arrow.{dylib,so}
s pkg install -g .              # installs into ~/.stryke/store/arrow@<version>/

Or:

make install

The cdylib is dlopened in-process on first use Arrow (stryke's FFI bridge resolves the symbols at module load — no helper binary, no subprocess fork per call).

[0x02] Quick Start

use Arrow

# Read parquet/arrow/feather/csv/json — format detected from extension.
val @rows = Arrow::read("sales.parquet")
p $rows[0]

# Stream huge files without buffering.
Arrow::read_stream("events.parquet", callback => fn ($row) {
    process($row)
})

# Cheap metadata: footer-only for parquet/ipc.
val $sch = Arrow::schema("sales.parquet")
p "fields: #{join(", ", map { "$_->{name}:$_->{type}" } @{ $sch->{fields} })}"

p Arrow::row_count("sales.parquet")           # parquet footer, no scan

# Stats: row count + per-column null counts, min, max, distinct (parquet
# uses footer metadata, other formats scan once).
p to_json(Arrow::stats("sales.parquet"))

# Write — schema inferred from the first NDJSON batch.
Arrow::write("out.parquet", \@rows, compression => "zstd")

# Server-side conversion (never round-trips through stryke memory).
Arrow::convert("in.csv", "out.parquet", compression => "zstd")

# Compute: filter → sort → top-N, each step file→file, no data in stryke.
Arrow::filter("sales.parquet", "big.parquet", "amount", "ge", 1000)
Arrow::sort("big.parquet", "ranked.parquet", [{ column => "amount", descending => 1 }])
Arrow::head("ranked.parquet", "top10.parquet", 10)

Per-format aliases when you want it explicit:

use Arrow::Parquet
use Arrow::IPC
use Arrow::Feather
use Arrow::CSV
use Arrow::JSON

Arrow::Parquet::read("x.parquet")
Arrow::IPC::write("x.arrow", \@rows)
Arrow::CSV::stats("x.csv")

DataFrame bridge:

use Arrow::DataFrame

val $df = Arrow::DataFrame::load("sales.parquet")
# $df is a stryke DataFrame when the builtin is available; otherwise a
# { col => [vals] } columnar hash.

[0x03] Options

Every Arrow::* op accepts %opts. Read fields:

format       → parquet|ipc|arrow|feather|csv|tsv|json|ndjson  (default: extension-detected)
columns      → \@names  — projection at the source format
limit        → max rows
skip         → rows to skip from the start
batch_size   → reader batch size (default 8192)

Write fields:

format       → as above
compression  → snappy|gzip|zstd|lz4|brotli|none  (parquet only; default snappy)
row_group    → max rows per parquet row group (default 65536)

Convert fields:

src_format, dst_format, compression, row_group

Sublibraries (Arrow::Parquet, Arrow::IPC, Arrow::Feather, Arrow::CSV, Arrow::JSON, Arrow::DataFrame) pin format automatically — see lib/Parquet.stk etc.

[0x04] API Reference

Arrow::read(PATH, %opts) → @rows | \@rows

Load every row as a hashref. Options: format, columns (array or comma string), limit, skip, batch_size.

Arrow::read_stream(PATH, callback => sub ($row) { … }, %opts) → $count

Same options as read; calls the callback once per row without buffering.

Arrow::read_columnar(PATH, %opts) → { schema, num_rows, columns }

Single columnar object. Faster than read when you want column-major access.

Arrow::schema(PATH, %opts) → { fields, metadata }

Schema only; no data scan.

Arrow::stats(PATH, %opts) → { num_rows, num_columns, file_size, columns }

Parquet pulls min/max/null_count from row-group statistics. CSV/JSON/IPC scan once. Each entry in columns is { name, type, nullable, null_count, distinct_count, min, max }.

Arrow::row_count(PATH, %opts) → $n

Shortcut around stats.

Arrow::column_names(PATH, %opts) → \@names

Schema field names in file order — pure-stryke over schema.

Arrow::column_count(PATH, %opts) → $n

Number of columns in the schema.

Arrow::is_empty(PATH, %opts) → 1 | 0

True when the file has zero data rows (the schema may still be present).

Arrow::write(PATH, \@rows, %opts) → $n

Options: format, compression (parquet only: snappy|gzip|zstd|lz4|brotli|uncompressed), row_group, schema (path to a JSON schema spec to skip inference on huge inputs).

Arrow::write_iter(PATH, sub { … } , %opts) → $n

Iterator form: subref returns one row per call, undef to stop. Streams to the helper without holding all rows in stryke.

Arrow::convert(SRC, DST, %opts) → DST

Server-side reader-to-writer pipeline. Options: src_format, dst_format, compression, row_group. Doesn't round-trip data through the stryke process.

Arrow::version() → $string

The cdylib's package version (env!("CARGO_PKG_VERSION")).

Compute (file → file)

Server-side transforms over Arrow's compute kernels — read SRC, apply the kernel, write DST. None round-trips data through the stryke process. All accept src_format, dst_format, compression, row_group in %opts; DST format defaults to the destination extension, else the source format.

Arrow::filter(SRC, DST, COLUMN, OP, VALUE, %opts) → { dst, rows }

Keep rows where COLUMN OP VALUE. OP ∈ eq|ne|lt|le|gt|ge. VALUE is typed against the column (int/float/string/bool).

Arrow::filter_in(SRC, DST, COLUMN, \@values, %opts) → { dst, rows }

Keep rows where COLUMN is in \@values — SQL IN. Each value is typed against the column; an empty set matches no rows. Unknown column errors.

Arrow::filter_not_in(SRC, DST, COLUMN, \@values, %opts) → { dst, rows }

Keep rows where COLUMN is NOT in \@values — SQL NOT IN, the complement of filter_in. A null is kept (matches nothing, like pandas ~isin); an empty set keeps every row. Unknown column errors.

Arrow::filter_str(SRC, DST, COLUMN, OP, VALUE, %opts) → { dst, rows }

The string-search filter that complements filter (numeric/ordered comparison only). Keep rows whose string COLUMN matches VALUE under OP ∈ contains|starts_with|ends_with|like|ilike: contains/starts_with/ends_with test substrings; like/ilike use SQL %/_ wildcards (ilike case-insensitive). The column must be a string column; a null cell never matches.

Arrow::select(SRC, DST, \@cols, %opts) → { dst, rows, columns }

Project and reorder to \@cols (output order = request order).

Arrow::drop(SRC, DST, \@cols, %opts) → { dst, rows, columns }

Complement of select: remove \@cols, keep the rest in original order. Each named column must exist (a typo errors). columns lists the survivors.

Arrow::distinct(SRC, DST, %opts) → { dst, rows, dropped }

Drop duplicate rows, keeping the first occurrence of each distinct row in input order. Compares every column (all types + nulls, via Arrow's row encoding). dropped is the number removed.

Arrow::drop_nulls(SRC, DST, \@cols?, %opts) → { dst, rows, dropped }

Drop rows that are null in any of \@cols (omit to check every column). A row survives only when all target columns are present. dropped is the number removed. Unknown column names error.

Arrow::keep_nulls(SRC, DST, \@cols?, %opts) → { dst, rows, dropped }

The complement of drop_nulls: keep only rows that are null in at least one of \@cols (omit for any column), for isolating the incomplete rows. dropped is the number of fully-populated rows removed. Unknown column names error.

Arrow::fill_null(SRC, DST, VALUE, \@cols?, %opts) → { dst, rows, filled }

Fill nulls in \@cols (omit for every column) with the constant VALUE, typed to match each column (integer/float/string/bool). The fill companion to drop_nulls — row count is unchanged; filled is the number of null cells replaced. Unknown column names error.

Arrow::sort(SRC, DST, \@by, %opts) → { dst, rows }

Lexicographic sort. \@by is [{ column => NAME, descending => 1, nulls_first => 0 }, …].

Arrow::reverse(SRC, DST, %opts) → { dst, rows }

Reverse the row order (last row first), independent of any sort key — just flips whatever order the rows are already in. A double reverse is the identity.

Arrow::gather(SRC, DST, \@indices, %opts) → { dst, rows }

Select rows by an explicit list of 0-based indices (polars gather / pandas .iloc[[…]]). Unlike slice (a contiguous window), the index list is arbitrary: it may repeat a row and emits rows in the order given. Out-of-range indices die.

Arrow::top_k(SRC, DST, COLUMN, K, %opts) → { dst, rows }

The K rows with the largest values in COLUMN (polars top_k). Pass descending => 0 for the smallest (bottom_k). Nulls sort last so they never take a top slot, and K caps at the row count. The sort is limited to K indices up front, so it does less work than sort-then-head.

Arrow::value_counts(SRC, DST, COLUMN, %opts) → { dst, rows, distinct }

Frequency of each distinct value in COLUMN (pandas/polars value_counts). Writes a two-column table — COLUMN (original type, distinct values) and a count (Int64) — one row per distinct value, sorted by count descending then value ascending. Nulls form their own group.

Arrow::head(SRC, DST, N, %opts) / Arrow::tail(SRC, DST, N, %opts) → { dst, rows }

First / last N rows.

Arrow::slice(SRC, DST, OFFSET, LENGTH, %opts) → { dst, rows }

Half-open row window [OFFSET, OFFSET+LENGTH).

Arrow::with_row_index(SRC, DST, %opts) → { dst, rows }

Prepend a 0-based UInt64 row-index column (polars with_row_index / pandas reset_index). opts: name (default index), offset (default 0).

Arrow::concat(\@srcs, DST, %opts) → { dst, rows, sources }

Concatenate sources with identical schemas into one DST.

Arrow::hstack(SRC, OTHER, DST, %opts) → { dst, rows, columns }

Horizontally stack OTHER's columns onto SRC — the column-wise counterpart of concat. Both files must have the same row count; a column-name collision is rejected.

Arrow::rename(SRC, DST, \%map, %opts) → { dst, rows }

Rename columns via { old => new }; unmapped columns pass through.

Arrow::cast(SRC, DST, \%casts, %opts) → { dst, rows }

Cast columns via { column => type } where type ∈ int|int32|float|float32|str|bool.

Arrow::count(PATH, %opts) → $n

Row count read straight from the source, no JSON materialization.

Arrow::null_counts(PATH, %opts) → { null_counts => { col => n, … }, rows }

Per-column null count (pandas isnull().sum(), polars null_count()), accumulated from each column's native Arrow null_count() without materializing.

Arrow::shape(PATH, %opts) → { rows, columns }

The dataset's (rows, columns) shape (pandas/polars .shape) — column count from the schema, rows streamed without materializing.

Aggregation + numeric transforms

Read-only aggregates return their result in the payload (no DST); the rest are file → file like the compute ops above. Numeric work casts the target column to Float64 so one code path covers every integer/float width.

Arrow::sum(PATH, COLUMN, %opts) → { column, sum, count }

Sum of one numeric COLUMN (polars/pandas sum); count excludes nulls. A non-numeric column errors.

Arrow::mean(PATH, COLUMN, %opts) → { column, mean, count }

Arithmetic mean of one numeric COLUMN; nulls are excluded from both sum and divisor. mean is undef for an all-null or empty column.

Arrow::min_max(PATH, COLUMN, %opts) → { column, min, max }

Minimum and maximum of one numeric COLUMN in a single pass; both undef when the column is all-null.

Arrow::std(PATH, COLUMN, %opts) → { column, std, var, mean, count }

Standard deviation + variance of one numeric COLUMN (pandas/polars std/var). Sample by default (n-1 divisor); pass population => 1 for the population n divisor. Nulls are excluded; all stats are undef when count is below the divisor floor (0 population, 1 sample).

Arrow::median(PATH, COLUMN, %opts) → { column, median, count }

Median (50th percentile) of one numeric COLUMN (pandas/polars median); an even count averages the two middle values. Nulls are excluded; median is undef for an all-null column.

Arrow::quantile(PATH, COLUMN, Q, %opts) → { column, q, quantile, count }

The Q-quantile of one numeric COLUMN (Q ∈ [0,1]) using numpy's default linear interpolation — q=0 min, q=0.5 median, q=1 max. Nulls are excluded; quantile is undef for an all-null column. Q outside [0,1] errors.

Arrow::corr(PATH, X, Y, %opts) → { x, y, corr, count }

Pearson correlation between numeric columns X and Y (pandas/polars corr). Only pairwise-complete rows (both non-null) contribute. corr is undef with fewer than two complete pairs or when either column has zero variance over them.

Arrow::describe(PATH, %opts) → { columns }

Per-column summary over every numeric column (pandas DataFrame.describe). Each entry is { column, count, nulls, min, max, mean, sum }; non-numeric columns are skipped.

Arrow::aggregate(SRC, DST, %opts) → { dst, rows, agg }

Column-wise aggregate into a two-column { column, value } table. agg ∈ sum|mean|min|max (default sum); pass columns => \@cols (each numeric) to limit it, else every numeric column is aggregated.

Arrow::unique(SRC, DST, COLUMN, %opts) → { dst, rows }

Distinct values of a single COLUMN (SQL SELECT DISTINCT col / polars Series.unique), written as a one-column table sorted ascending (nulls last). Unlike distinct (whole-row dedupe), this dedupes one column.

Arrow::clip(SRC, DST, COLUMN, LOWER?, UPPER?, %opts) → { dst, rows }

Clamp a numeric COLUMN into [LOWER, UPPER] (pandas/polars clip). Pass undef for either bound to leave that side open; at least one is required. Nulls stay null.

Arrow::scale(SRC, DST, COLUMN, FACTOR, %opts) → { dst, rows }

Multiply a numeric COLUMN by the constant FACTOR in place (polars col * k). The column keeps its width (integer columns stay integer, truncating).

Arrow::add_const(SRC, DST, COLUMN, VALUE, %opts) → { dst, rows }

Add the constant VALUE to a numeric COLUMN in place — the additive counterpart of scale. The column keeps its width (integer columns stay integer).

Arrow::abs(SRC, DST, COLUMN, %opts) → { dst, rows }

Absolute value of a numeric COLUMN in place (pandas/polars abs). The column keeps its on-disk width; nulls stay null.

Arrow::round(SRC, DST, COLUMN, %opts) → { dst, rows }

Round a numeric COLUMN to decimals places in place (pandas/polars round; decimals default 0, half-away-from-zero). The column keeps its width; nulls stay null.

Arrow::add_column(SRC, DST, NAME, VALUE, TYPE, %opts) → { dst, rows, columns }

Append a new column NAME filled with the constant VALUE, typed by TYPE ∈ int|int32|float|float32|str|bool (polars with_columns(lit(...))). The name must not already exist.

Arrow::sample(SRC, DST, STEP, %opts) → { dst, rows }

Systematic sample: keep every STEP-th row starting at offset (default 0) — df[offset::step]. STEP must be at least 1. Unlike head/slice (contiguous) or gather (explicit list), this thins a table uniformly.

Arrow::fold_case(SRC, DST, %opts) → { dst, rows, columns }

Lower- or upper-case every column NAME (data untouched) — case ∈ lower|upper (default lower). A case-fold that collides two names is rejected.

Arrow::metadata(PATH, %opts) → { src, format, bytes, rows, columns, types }

File-level metadata in one call: detected format, on-disk byte size, row count, column count, and a { name => type } schema map.

[0x05] FFI Layer

Each Arrow::* wrapper builds a JSON args dict and calls a sibling arrow__* symbol resolved out of libstryke_arrow.{dylib,so}. The cdylib is dlopened in-process on first use Arrow (via stryke's pkg::commands::try_load_ffi_for resolver hook). Its exports span four groups: read (version, read, read_columnar, schema, stats), write (write), conversion (convert), and compute (filter, select, sort, slice, head, tail, count, concat, rename, cast, plus aggregation + numeric transforms: sum, mean, min_max, describe, aggregate, unique, clip, scale, add_column, sample, fold_case, metadata). The authoritative list is [ffi].exports in stryke.toml.

Wire shape (cdylib responses):

  • read → {"columns": [...], "rows": [{col: val, ...}, ...]}
  • read_columnar → {"columns": [...], "num_rows": N, "data": {col: [...]}}
  • schema → {"fields": [{name, type, nullable}, ...]}
  • stats (parquet) → {"num_rows", "num_row_groups", "columns": [{name, null_count, min, max}, ...]}
  • write, convert → {"path": ..., "rows": N}
  • compute (filter/sort/select/…) → {"dst": ..., "rows": N}; count → {"src": ..., "rows": N}
  • Errors → {"error": "<msg>"} — the wrapper dies with it

Stateless package — arrow operations are pure file transforms, no process-level cache.

[0x06] Supported Formats

FormatReadWriteNotes
Parquet✓✓snappy / gzip / zstd / lz4 / brotli / uncompressed
Arrow IPC✓✓.arrow extension; no compression
Feather✓✓alias for Arrow IPC v2
CSV✓✓header row mandatory; schema inferred from first 1024 rows
JSON (NDJSON)✓✓line-delimited only

[0x07] Compression

CodecLibraryNotes
snappysnapdefault; fast, modest ratio
zstdzstdbest ratio per CPU
gzipflate2broad compatibility
lz4lz4_flexLZ4_RAW frame
brotlibrotlihigh ratio, slow
uncompressed—fastest write, biggest file

[0x08] Discovery

The cdylib lives next to Arrow.stk inside the package install dir:

~/.stryke/store/arrow@<version>/
  stryke.toml
  lib/
    Arrow.stk
    libstryke_arrow.{dylib,so}

On use Arrow, stryke's pkg::commands::try_load_ffi_for reads the sibling stryke.toml, finds the [ffi] table, and dlopens the cdylib once for the life of the process. No env vars, no PATH probing, no helper-binary discovery — the install dir IS the discovery answer.

[0x09] Tests

cargo test                       # helper CLI contract tests (tests/)
s test t/                        # end-to-end round-trip per format

t/test_arrow.stk writes a small dataset in every format, reads it back, checks shape + values, and exits TAP-style.

[0x0A] Dev Workflow

make             # release build
make debug       # faster compile
make test        # cargo test + s test t/
make install     # release + pkg install -g .
make clean

[0x0B] Layout

stryke-arrow/
  stryke.toml                  # stryke package manifest
  Cargo.toml                   # cdylib crate manifest
  Makefile                     # convenience targets
  src/lib.rs                   # cdylib — arrow__* extern "C" exports
  lib/
    Arrow.stk                  # `use Arrow`
    Parquet.stk                # `use Arrow::Parquet`
    IPC.stk                    # `use Arrow::IPC`
    Feather.stk                # `use Arrow::Feather`
    CSV.stk                    # `use Arrow::CSV`
    JSON.stk                   # `use Arrow::JSON`
    DataFrame.stk              # `use Arrow::DataFrame`
  t/
    test_arrow.stk             # round-trip tests per format
    test_stryke_arrow_surface.stk  # wrapper-completeness pin
  examples/
    csv_to_parquet.stk
    dataframe_bridge.stk
    discover.stk
    json_lines.stk
    read_parquet.stk

[0xFF] License

MIT.