Public-Data Validation Benchmark

This page describes the larger public-data validation path. It is designed to move beyond the tiny IC smoke test while staying honest about what public data can and cannot prove.

The validation script compares:

• equal-weight baseline
• 20-day momentum baseline
• Alpha101 subset score baseline
• walk-forward MLP baseline on Alpha101 features
• walk-forward Transformer baseline on Alpha101 features

It reports annual return, volatility, Sharpe, max drawdown, turnover, cost drag, gross return, optional bootstrap uncertainty intervals, and active metrics versus equal weight. Transaction costs and slippage are combined into the effective cost charged on weight changes.

Quick Synthetic Check

Use this first to verify that the validation harness works without downloading data:

python scripts/public_data_validation.py \
  --source synthetic \
  --models equal_weight,momentum_20,alpha101_mean

The command writes:

artifacts/public_data_validation/summary.md
artifacts/public_data_validation/summary.csv
artifacts/public_data_validation/summary.json
artifacts/public_data_validation/metadata.json
artifacts/public_data_validation/submission.md

Use submission.md when opening a GitHub issue. It contains the command, environment, data coverage, result table, and interpretation notes in one copy-ready report.

Larger Public-Data Run

The default public run uses a 100-name US large-cap preset from yfinance:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset us-large-100 \
  --max-tickers 100 \
  --start 2021-01-01 \
  --end 2025-01-01

For an ETF universe:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset etf-50 \
  --max-tickers 50

For a mixed stock/ETF universe:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset mixed-150 \
  --max-tickers 150

For a zero-auth A-share public-data run through AkShare, use the CSI 300 dynamic preset:

python scripts/public_data_validation.py \
  --source akshare \
  --preset csi-300 \
  --max-tickers 300 \
  --start 2021-01-01 \
  --end 2025-01-01 \
  --models equal_weight,momentum_20,alpha101_mean \
  --epochs 1 \
  --batch-size 4096 \
  --hidden 32 \
  --cost-grid-bps 0,7,15,30 \
  --bootstrap-samples 100 \
  --bootstrap-block-size 20

csi-300 and hs300 are aliases. The script resolves the current CSI 300 constituents from AkShare/CSI public endpoints at runtime, normalizes them to six-digit A-share codes, and records the resolved ticker list in metadata.json. This is useful for public reproducibility because it requires no Baostock login and no proprietary data dump.

Important limitation: this is not historical point-in-time index membership. The resolved universe is the currently published CSI 300 constituent list available when the command is run, then backfilled over the requested date range. Treat the result as a public-data validation diagnostic, not as a full paper reproduction or deployable alpha claim.

For a paper-style public-data approximation with the full 213-factor library and daily evaluation, use the full-pipeline script:

python scripts/akshare_csi300_full_pipeline.py \
  --factor-set all \
  --preset csi-300 \
  --max-tickers 300 \
  --start 2021-01-01 \
  --end 2025-01-01 \
  --rebalance-step 1 \
  --cost-grid-bps 0,7,15,30

The maintained report is validation_akshare_csi300_full_pipeline_20260729.md. The homepage-facing snapshot is tracked in validation_dashboard.md. It shows why daily public-data runs need a portfolio layer: the naive daily 213-factor portfolio has strong gross return but high turnover, while the buffered daily factor rule preserves enough of the factor edge to beat equal weight after costs. The main comparison intentionally stays daily; weekly rebalancing is useful as a sensitivity check, but it can make high-turnover factor selection look cleaner than it really is.

You can also pass your own comma-separated ticker list:

python scripts/public_data_validation.py \
  --source yfinance \
  --tickers AAPL,MSFT,NVDA,SPY,QQQ,IWM

Maintainer 100-Stock Reference Run

This reference run was executed on the built-in us-large-100 yfinance preset. It is included to make the benchmark concrete and comparable. It is not evidence of deployable alpha.

Command:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset us-large-100 \
  --max-tickers 100 \
  --start 2021-01-01 \
  --end 2025-01-01 \
  --models equal_weight,momentum_20,alpha101_mean,mlp_alpha101,transformer_alpha101 \
  --epochs 1 \
  --batch-size 4096 \
  --hidden 32

Environment:

Field	Value
Data source	yfinance
Universe	100 US large-cap tickers
Returned panel	1005 dates x 100 stocks
Returned date range	2021-01-04 to 2024-12-31
Costs + slippage	5.00 + 2.00 bps
Walk-forward train/test/step	504 / 63 / 63 trading days
Python	3.9.6
Platform	macOS 26.5.1 arm64
PyTorch	2.8.0

Results:

Strategy	Ann Return	Ann Vol	Sharpe	Max DD	Turnover	Cost Drag	Gross Ann Return	Gross Sharpe	Info Ratio vs EW	Active Ann Return	Final Equity
equal_weight	0.1818	0.1531	1.1681	0.1997	0.0005	0.0007	0.1820	1.1693			1.9470
momentum_20	0.1072	0.1719	0.6782	0.2781	0.1675	0.2355	0.1745	1.0232	-0.6280	-0.0650	1.5008
alpha101_mean	-0.0529	0.1608	-0.2577	0.3043	0.5401	0.7592	0.1456	0.9266	-3.4446	-0.1994	0.8050
mlp_alpha101	0.0323	0.0914	0.3934	0.1729	0.2915	0.4097	0.1439	1.5164	-1.0431	-0.1413	1.1351
transformer_alpha101	-0.0157	0.0932	-0.1227	0.1603	0.2921	0.4105	0.0910	0.9793	-1.3872	-0.1811	0.9390

In this reference run, equal weight is the strongest net baseline. That is useful negative evidence: the current public-data validation harness is not being used to claim that the ML baselines already beat simple portfolios.

Walk-Forward Design

By default, the ML baselines train on roughly two years of daily observations and predict the next quarter:

train window: 504 trading days
test window:   63 trading days
step:          63 trading days

Each split trains only on dates before the test window. The script then turns predictions into long-only top-quantile portfolios and runs the same vectorized backtest path as the rest of the repository.

Cost And Slippage Analysis

The script exposes both transaction costs and slippage:

python scripts/public_data_validation.py \
  --costs-bps 5 \
  --slippage-bps 2

The effective cost is:

effective_cost_bps = costs_bps + slippage_bps

To check whether the same strategies are fragile to different effective cost assumptions, add a cost grid:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset us-large-100 \
  --cost-grid-bps 0,7,15,30

This does not retrain the ML models or reselect parameters. It re-scores the same generated portfolio weights under each effective cost scenario and writes:

artifacts/public_data_validation/cost_sensitivity.md
artifacts/public_data_validation/cost_sensitivity.csv
artifacts/public_data_validation/cost_sensitivity.json

The same rows are also embedded in summary.json under cost_sensitivity so maintainers can audit or aggregate them later. Use this table to spot strategies whose net results depend on one narrow cost assumption.

This is intentionally simple. It is useful for sensitivity checks, but it is not a substitute for a broker-, exchange-, order-size-, and liquidity-aware execution model.

Bootstrap Uncertainty Intervals

Point estimates can be noisy, especially for short public-data windows. Add a block bootstrap to report 95% intervals for annualized return and Sharpe:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset us-large-100 \
  --bootstrap-samples 500 \
  --bootstrap-block-size 20

The bootstrap samples contiguous return blocks instead of individual days so the resampled paths preserve some short-horizon autocorrelation. The output adds:

ann_return_ci_low
ann_return_ci_high
sharpe_ci_low
sharpe_ci_high

These intervals are diagnostics, not formal proof of statistical significance. They are most useful for spotting fragile point estimates and for comparing community reports that use the same universe and date range.

Metric Glossary

Every column in the summary table, with its unit and time basis. The distinction that matters most is the last column: most figures are per year, one is a total over the run, and comparing across those two without noticing is easy.

column	unit	basis
ann_return	fraction	per year, geometric
gross_ann_return	fraction	per year, geometric, before costs
ann_vol	fraction	per year
sharpe, gross_sharpe	ratio	per year
info_ratio	ratio	per year, against the benchmark
alpha_ann	fraction	per year, against the benchmark
max_dd	fraction	worst peak-to-trough over the run
turnover	fraction	average per rebalance
cost_drag_cumulative	fraction	total over the whole run, arithmetic sum
final_equity	multiple	total over the run, starting at 1.0
effective_costs_bps	bps per side	input, not a result

cost_drag_cumulative is not annualised. It is sum(daily cost) across every period in the backtest, so a strategy tested over eight years reports roughly twice the drag of the same strategy over four — at identical turnover and identical fees. Two consequences:

• Do not read it as a rate beside ann_return. On a four-year run a cost_drag_cumulative of 0.20 is about 5 percentage points a year, not 20.
• Do not compare it between reports of different lengths. turnover and effective_costs_bps are the length-independent figures; use those.

It also will not reconcile exactly with gross_ann_return − ann_return even after dividing by the number of years, because the annualised returns compound while the cost sum is arithmetic. The gap is small on long runs and material on short ones. No derived "annualised cost drag" is published for that reason: an approximate number under an exact-sounding name is worse than an explicit one.

cost_drag remains as a deprecated alias of cost_drag_cumulative for one release so archived reports and existing tooling keep working. New output should use the explicit name.

Interpreting Results

This benchmark is stronger than the tiny public-data IC note because it includes a larger universe, walk-forward prediction, portfolio construction, turnover, drawdown, costs, uncertainty intervals, and baseline comparisons.

When interpreting a factor strategy, always read net return together with gross_ann_return and turnover — all three are per-year figures, so they compare directly. cost_drag_cumulative is a total over the run rather than a rate (see the glossary above), so divide it by the number of years before holding it beside an annualised return. A strategy can have a positive gross return but still fail after costs if it trades too aggressively. That is especially important for naive factor blends such as alpha101_mean: a negative net result may be a turnover-control and portfolio-construction problem rather than evidence that the factor family has no research value.

It still should not be read as proof of deployable alpha:

• yfinance data is convenient but not institutional-grade research data.
• AkShare is a useful zero-auth A-share source, but public endpoints can change, throttle, or revise records.
• Current presets are survivorship-biased because membership is fixed today.
• Slippage is modeled as a simple basis-point charge, not a market-impact model.
• The ML baselines are intentionally small and should be treated as reference comparisons, not tuned production models.
• Public benchmark results can drift when yfinance revises data.

The most valuable community contributions are additional benchmark reports with clear universe definitions, dependency versions, hardware details, and the exact command used.

Sharing A Community Report

1. Run the validation command.
2. Audit artifacts/public_data_validation/summary.json.
3. Open artifacts/public_data_validation/submission.md.
4. Paste it into the Public-data validation report issue template.
5. Attach or paste metadata.json when the run uses a custom universe.

The generated summary.json is intended for future aggregation scripts and leaderboards. It includes metadata, data coverage, and strategy metrics in a machine-readable format.

Troubleshooting yfinance Rate Limiting

Yahoo Finance may throttle requests from certain networks or IP ranges. When this happens, the validation script fails during the data-download stage before any backtest or report is generated. If no public-data result was produced, report the blocker first instead of opening a validation PR.

How to recognise rate limiting

Rate limiting can surface in several misleading forms. The underlying cause is usually an HTTP 429 (Edge: Too Many Requests) response from Yahoo, but yfinance may report it as one or more of:

• YFRateLimitError: Too Many Requests. Rate limited. Try after a while.
• JSONDecodeError: Expecting value: line 1 column 1 (char 0) — the response body is not valid JSON because Yahoo returned an error page
• YFTzMissingError: possibly delisted; No timezone found — tickers that normally download fine suddenly appear "delisted"
• request timeouts or connection errors around the same run

Dozens of tickers failing at once — especially a preset like etf-50 or us-large-100 that has worked in prior runs — is a strong signal that the problem is network-side, not ticker-side.

What to do

1. Stop. Do not retry the full 50-ticker run immediately — repeated requests may extend the rate-limit window.
2. Report the blocker in the relevant issue rather than opening a PR. An incomplete validation run is still useful as a blocker report, but it should not be submitted as a benchmark result. Mention the exact errors, the date/time, yfinance version, and any non-sensitive network context you are comfortable sharing.
3. Wait and retry with a smoke test before scaling back up:

python scripts/public_data_validation.py \
  --source yfinance \
  --tickers SPY,QQQ,TLT,GLD,AGG \
  --start 2021-01-01 \
  --end 2025-01-01 \
  --models equal_weight,momentum_20 \
  --epochs 1 \
  --batch-size 4096 \
  --hidden 32 \
  --cost-grid-bps 0,7,15,30 \
  --bootstrap-samples 50 \
  --bootstrap-block-size 20

4. If the 5-ticker smoke test succeeds, scale to 20 and then to the full preset. If it still returns 429, wait longer or retry from a different network.
5. Once the full run completes, open a PR with the generated submission.md report.

Fallback: synthetic validation

If yfinance remains unavailable, the synthetic data path is always usable as a reproducibility check:

python scripts/public_data_validation.py \
  --source synthetic \
  --models equal_weight,momentum_20,alpha101_mean

A synthetic run cannot replace a public-data report, but it confirms that the validation harness itself is working on your platform.

Auditing Reports

Before sharing a validation run, audit the generated summary.json:

python scripts/audit_validation_report.py \
  artifacts/public_data_validation/summary.json \
  --output-md artifacts/public_data_validation/audit.md \
  --output-json artifacts/public_data_validation/audit.json

The audit checks for missing metadata, missing result rows, low data coverage, tickers with no data, missing equal-weight baseline, non-finite metrics, negative cost settings, unusual turnover/drawdown, and non-positive final equity. It is a quality gate for reproducibility reports, not a judgement about whether a strategy is good.

For the default artifact path, the Make target is:

make audit-validation

Aggregating Reports

Maintainers can aggregate one or more validation result directories:

python scripts/aggregate_validation_reports.py \
  artifacts/public_data_validation \
  --output-md artifacts/public_data_validation/leaderboard.md \
  --output-csv artifacts/public_data_validation/leaderboard.csv

The script scans for summary.json files and writes a compact leaderboard with source, preset, date range, panel size, strategy, return, Sharpe, drawdown, turnover, data coverage, and environment columns.

For the default artifact path, the Make target is:

make aggregate-validation