Anofox Statistics

September 2, 2026 · View on GitHub

A statistical analysis extension for DuckDB, providing regression analysis, diagnostics, and inference capabilities directly within your database.

Important

This extension is in early development, so bugs and breaking changes are expected. Please use the issues page to report bugs or request features.

✨ Key Features

Regression Methods

Method	Function	Description
OLS	`ols_fit`, `ols_fit_agg`	Ordinary Least Squares
Huber	`huber_fit`, `huber_fit_agg`	Robust M-estimator (outlier-resistant; reports MAD scale + outlier mask)
RANSAC	`ransac_fit`, `ransac_fit_agg`	Robust consensus regression (outlier-resistant; reports inlier mask + trial count)
Theil-Sen	`theil_sen_fit`, `theil_sen_fit_agg`	Robust nonparametric regression via spatial-median over OLS subsamples
Ridge	`ridge_fit`, `ridge_fit_agg`	L2 regularization
Elastic Net	`elasticnet_fit`, `elasticnet_fit_agg`	Combined L1+L2 regularization
WLS	`wls_fit`, `wls_fit_agg`	Weighted Least Squares
RLS	`rls_fit`, `rls_fit_agg`	Recursive Least Squares (online)
Poisson	`poisson_fit_agg`	GLM for count data
Binomial	`binomial_fit_agg`	GLM for success-rate data (logit / probit / cloglog links)
Negative Binomial	`negbinom_fit_agg`	GLM for overdispersed counts (dispersion α estimated jointly)
Tweedie	`tweedie_fit_agg`	GLM for positive-skew continuous outcomes
Gamma	`gamma_fit_agg`	GLM for strictly-positive continuous outcomes (claims, durations)
Logistic	`logistic_fit_agg`	Binary classification (binomial GLM with logit link); reports accuracy + threshold echo, optional L2 penalty
ALM	`alm_fit_agg`	24 error distributions
BLS/NNLS	`bls_fit_agg`, `nnls_fit_agg`	Bounded/Non-negative LS
PLS	`pls_fit`, `pls_fit_agg`	Partial Least Squares
Isotonic	`isotonic_fit`, `isotonic_fit_agg`	Monotonic regression
Quantile	`quantile_fit`, `quantile_fit_agg`	Quantile/median regression
AFT survival	`aft_fit_agg`	Duration models with right censoring
Mixed effects	`glmm_fit_agg`	Random intercept over a grouping factor
EB shrinkage	`eb_shrink_agg`	Partial pooling of per-group estimates

Statistical Hypothesis Tests

Category	Function	Description
Normality	`shapiro_wilk_agg`, `jarque_bera_agg`, `dagostino_k2_agg`	Normality tests
Parametric	`t_test_agg`, `one_way_anova_agg`, `yuen_agg`, `brown_forsythe_agg`	Parametric tests
Nonparametric	`mann_whitney_u_agg`, `kruskal_wallis_agg`, `wilcoxon_signed_rank_agg`, `brunner_munzel_agg`, `permutation_t_test_agg`	Nonparametric tests
Correlation	`pearson_agg`, `spearman_agg`, `kendall_agg`, `distance_cor_agg`, `icc_agg`	Correlation tests
Categorical	`chisq_test_agg`, `chisq_gof_agg`, `g_test_agg`, `fisher_exact_agg`, `mcnemar_agg`	Contingency table tests
Effect Size	`cramers_v_agg`, `phi_coefficient_agg`, `contingency_coef_agg`, `cohen_kappa_agg`	Association measures
Proportion	`prop_test_one_agg`, `prop_test_two_agg`, `binom_test_agg`	Proportion tests
Equivalence	`tost_t_test_agg`, `tost_paired_agg`, `tost_correlation_agg`	TOST equivalence tests
Distribution	`energy_distance_agg`, `mmd_agg`	Distribution comparison
Forecast	`diebold_mariano_agg`, `clark_west_agg`	Forecast evaluation

Diagnostics & Utilities

Function	Description
`vif`, `vif_agg`	Variance Inflation Factor
`aic`, `bic`	Model selection criteria
`residuals_diagnostics_agg`	Residual analysis (raw, standardized, studentized, leverage arrays)
`aid_agg`, `aid_anomaly_agg`	Demand pattern classification

Fit-Predict Table Macros (`*_fit_predict_by`)

Table macros for easy per-group model fitting and prediction with a single function call:

Macro	Description
`ols_fit_predict_by`	OLS per-group fit + predict
`huber_fit_predict_by`	Huber robust per-group fit + predict
`ransac_fit_predict_by`	RANSAC robust per-group fit + predict
`theil_sen_fit_predict_by`	Theil-Sen robust per-group fit + predict
`ridge_fit_predict_by`	Ridge per-group fit + predict
`elasticnet_fit_predict_by`	ElasticNet per-group fit + predict
`wls_fit_predict_by`	WLS per-group fit + predict
`rls_fit_predict_by`	RLS per-group fit + predict
`bls_fit_predict_by`	Bounded LS per-group fit + predict
`alm_fit_predict_by`	ALM per-group fit + predict
`poisson_fit_predict_by`	Poisson GLM per-group fit + predict
`pls_fit_predict_by`	PLS per-group fit + predict
`isotonic_fit_predict_by`	Isotonic per-group fit + predict
`quantile_fit_predict_by`	Quantile per-group fit + predict

⚡ Performance

The extension is built with performance as a first-class concern. The Phase-4 benchmark harness (scripts/bench.sh) covers three representative workloads:

Workload	Scale	Per-group cost
W1 — aggregate dispatch	10K groups / 1M rows	~3.2 µs per OLS fit (3 features)
W2 — fit + predict	10K groups / 1M rows	fit → predict → marshal pipeline
W3 — FFI micro (inference)	500 groups / 50K rows	~4.8 µs per fit with `compute_inference: true`

Profiling showed that the dominant cost is DuckDB's own HASH_GROUP_BY dispatch (~66% of query time) — the extension's per-call overhead is a minority. Two optimizations landed in Phase 4:

DataArray::to_vec bulk-copy fast path (no-NULL path): eliminated per-element branching for dense columns, yielding a consistent ~3–4% query-time reduction at 5M rows / 50K groups (controlled A/B).
FfiVec<T> RAII wrapper + alloc_inference_arrays! macro: replaced 6 hand-written libc::malloc inference blocks, removing manual free/OOM boilerplate without changing allocation count (inherent to the FFI ABI).

Benchmark results and before/after numbers live in bench/PROFILING.md and bench/baseline/. Run bash scripts/bench.sh to reproduce.

🎨 User-Friendly API

Phase 5 (ERGO) made the API consistent and ergonomic:

Unprefixed function names: all functions are now ols_fit_agg(...), theil_sen_fit(...), etc. The old anofox_stats_ prefix is gone — see docs/API_CONVENTIONS.md §5 for migration.
MAP-style options with documented keys (e.g. {'fit_intercept': true, 'compute_inference': true}); unknown keys raise InvalidInputException at bind time instead of being silently ignored.
Early input validation: dimension mismatches, insufficient rows, all-non-finite input, and constant columns are caught with a descriptive message before any computation.
Consistent return-struct field names across families: r_squared (not .r2), residual_std_error, n_observations, z_values for GLM/AFT (not t_values).

The regression algorithms are validated against R's lm(), glmnet, and other standard statistical packages in the anofox-regression Rust crate.

🚀 Quick Start

All functions use the v0.3.0 API — unprefixed names and MAP-style options. See docs/API_CONVENTIONS.md for the full naming and options reference.

Step 1 — Create a small dataset and fit an OLS model

-- Dataset: house size (sqm) and sale price (kEUR)
CREATE TABLE houses AS SELECT * FROM (VALUES
    (50.0, 120.0), (65.0, 155.0), (80.0, 190.0),
    (95.0, 225.0), (110.0, 265.0), (125.0, 300.0)
) t(sqm, price_keur);

-- Fit: OLS regression (price_keur ~ sqm) via the aggregate form
SELECT
    round((ols_fit_agg(price_keur, [sqm])).r_squared, 4) AS r_squared,
    (ols_fit_agg(price_keur, [sqm])).coefficients[1]      AS slope,
    (ols_fit_agg(price_keur, [sqm])).intercept            AS intercept
FROM houses;
-- r_squared ≈ 0.9995, slope ≈ 2.41, intercept ≈ -1.67

The scalar ols_fit takes y and X in column-major format (each inner array is one feature column across all observations). Use the scalar predict(X_new, coefficients, intercept) to score new points:

-- Predict prices for three new house sizes (70, 100, 140 sqm)
-- ols_fit column-major X: [[sqm_col]] = one feature column with all 6 training values
SELECT
    unnest([70.0, 100.0, 140.0]) AS new_sqm,
    unnest(predict(
        [[70.0, 100.0, 140.0]]::DOUBLE[][],
        (ols_fit([120.0, 155.0, 190.0, 225.0, 265.0, 300.0],
                 [[50.0, 65.0, 80.0, 95.0, 110.0, 125.0]])).coefficients,
        (ols_fit([120.0, 155.0, 190.0, 225.0, 265.0, 300.0],
                 [[50.0, 65.0, 80.0, 95.0, 110.0, 125.0]])).intercept
    )) AS predicted_keur;
-- 70 sqm → ~167 kEUR, 100 sqm → ~239 kEUR, 140 sqm → ~336 kEUR

Step 3 — Inspect residuals

-- Residual diagnostics: raw and standardized residuals from in-sample predictions
WITH preds AS (
    SELECT
        unnest([120.0, 155.0, 190.0, 225.0, 265.0, 300.0])::DOUBLE AS actual,
        unnest(predict(
            [[50.0, 65.0, 80.0, 95.0, 110.0, 125.0]]::DOUBLE[][],
            (ols_fit([120.0, 155.0, 190.0, 225.0, 265.0, 300.0],
                     [[50.0, 65.0, 80.0, 95.0, 110.0, 125.0]])).coefficients,
            (ols_fit([120.0, 155.0, 190.0, 225.0, 265.0, 300.0],
                     [[50.0, 65.0, 80.0, 95.0, 110.0, 125.0]])).intercept
        )) AS yhat
)
SELECT
    (residuals_diagnostics_agg(actual, yhat)).raw AS raw_residuals
FROM preds;

Per-group regression with `GROUP BY`

All *_fit_agg functions support GROUP BY for per-segment models:

-- Fit a separate OLS model per product category
SELECT
    category,
    (ols_fit_agg(revenue, [units_sold], {'compute_inference': true})).r_squared AS r_squared,
    (ols_fit_agg(revenue, [units_sold], {'compute_inference': true})).coefficients[1] AS slope
FROM sales_data
GROUP BY category;

📦 Installation

From erpl.io (recommended)

Install directly from our public distribution bucket. DuckDB must be started with the -unsigned flag, since the extension binary is not signed by the DuckDB Foundation:

INSTALL 'anofox_statistics' FROM 'http://get.erpl.io';
LOAD 'anofox_statistics';

This pulls the right binary for your platform (Linux amd64/arm64, macOS amd64/arm64, Windows amd64). No build toolchain, no vcpkg, no submodules required.

Community Extension

INSTALL anofox_statistics FROM community;
LOAD anofox_statistics;

Telemetry

This extension collects anonymous usage telemetry to help improve the product. Telemetry is enabled by default and includes:

Extension load events (extension name, version, platform)
Function execution events (which functions are used)
No personal data or query contents are collected

Disable telemetry:

export DATAZOO_DISABLE_TELEMETRY=1

SET anofox_telemetry_enabled = false;

For more information, see the posthog-telemetry repository.

📚 API Reference

The authoritative function reference and naming conventions live in the docs/ directory — not duplicated here to avoid drift:

docs/API_REFERENCE.md — complete function signatures, option keys, and return-struct fields for all 100+ functions.
docs/API_CONVENTIONS.md — naming convention, MAP-option keys, return-struct field names, error taxonomy, and the v0.3.0 breaking-changes migration guide.

User-facing guides are in the guides/ directory:

guides/01_quick_start.md — getting started with worked examples
guides/02_technical_guide.md — architecture and implementation details
guides/03_business_guide.md — real-world business use cases
guides/04_advanced_use_cases.md — complex analytical workflows

🛠️ Development

Building from source

Prerequisites: Rust stable toolchain, a C++17 compiler, and CMake (all provided by extension-ci-tools):

git clone --recurse-submodules https://github.com/DataZooDE/anofox-statistics.git
cd anofox-statistics
make release

This produces build/release/duckdb (the CLI) and build/release/extension/anofox_statistics/anofox_statistics.duckdb_extension.

Rust unit tests (the core regression library):

cargo test

DuckDB SQL test suite (2 000+ assertions across all function families):

build/release/test/unittest --test-dir=test/sql

Doc-SQL validation (all SQL examples in README + guides + API docs must pass):

python3 scripts/validate_docs_sql.py          # full 7-file sweep
python3 scripts/validate_docs_sql.py --file README.md   # single-file fast path

Benchmark harness (repeatable timing across three workloads):

bash scripts/bench.sh          # default: ~1 s total
bash scripts/bench.sh --full   # adds 1M-group workload (~8 GB RAM, ~160 s)

Results are written to bench/results/ as diffable markdown files. See bench/README.md for details.

Contributing

Contributions are welcome:

Fork the repository
Create a feature branch
Make your changes with tests
Submit a pull request

Areas for contribution: additional statistical tests, visualization helpers for diagnostics, documentation and examples, bug reports and fixes, performance optimizations.

💬 Support

API docs: docs/API_REFERENCE.md and docs/API_CONVENTIONS.md
Guides: guides/
Issues: GitHub Issues
Discussions: GitHub Discussions
Email: contact@datazoo.de

If a fit misbehaves or a result looks wrong, please open an issue — regression against real data has failure modes we cannot reproduce from synthetic tests, so a report with your data shape is the fastest path to a fix. Errors from the fit and predict functions include that link.

If it saved you time, a star on the repo helps other people find it.

The first time you load the extension in an interactive terminal each day, a small banner says the same thing. It never prints when output is piped, in notebooks, or in CI. Silence it with SET datazoo_banner = false; or DATAZOO_NO_BANNER=1.

📖 Citation

If you use this extension in research, please cite:

@software{anofox_statistics,
  title = {Anofox Statistics: Statistical Analysis Extension for DuckDB},
  author = {DataZoo DE},
  year = {2025},
  url = {https://github.com/DataZooDE/anofox-statistics},
  version = {1.0.0}
}

⚖️ License

This project is licensed under the Business Source License 1.1 (BSL 1.1).

Key Terms

Usage Grant: Free to use, modify, and distribute for non-production purposes
Production Use: Permitted after 4 years from release date, or under a commercial license
Change Date: [Release Date + 4 years]
Change License: Apache License 2.0

See LICENSE for full terms.

Why BSL?

The BSL allows:

Free use for development, testing, and research
Open source collaboration and contributions
Academic and educational use
Small-scale production use

While ensuring:

Sustainable development funding
Protection for the project's long-term viability
Future conversion to fully open source (Apache 2.0)

For commercial production use before the Change Date, please contact: contact@datazoo.de