Method, API behaviour and caveats

This is an unofficial project. It is not published by, affiliated with or endorsed by data.gov.il or any Israeli government body. Everything here is derived from the portal's public CKAN API by a third party.

Verified against the live portal on 2026-07-27. Statements below were confirmed by request unless marked inferred.

Nothing is scraped

The whole changelog comes from one CKAN endpoint. package_search accepts a sort parameter and honours creation order:

GET https://data.gov.il/api/3/action/package_search?rows=1000&start=0&sort=metadata_created%20desc

The response echoes back "sort": "metadata_created desc", which is how you can tell the parameter was accepted rather than silently ignored.

• rows is capped at 1000 per request — paginate with start=.
• 1197 datasets at the time of writing, so a full pull is two requests.
• fl (field list) is not supported and returns HTTP 500, not a 400 with an error body. If a request suddenly 500s, suspect an unsupported parameter before suspecting an outage.

What metadata_created is, and what it is not

This is the load-bearing assumption, and it was checked before anything was built, because government CKAN instances frequently bulk-import their whole catalogue on migration day and flatten the real publication history.

On data.gov.il it behaves well:

• 0 of 1197 datasets are missing metadata_created.
• Timestamps are microsecond-resolution and land minutes apart within a publishing session (three Judicial Authority datasets at 12:24, 12:27, 12:29 on 2026-07-15) — real individual publication events, not a batch stamp.
• Additions spread across 124 distinct months from 2015-12 to 2026-07, with only 4 empty months in a 128-month span. Median 6 datasets/month.

But it is the record's creation timestamp in the portal's database, not a warranted publication date. Nobody at the portal guarantees it, and it is not documented as a publication date. A record could in principle be created before a dataset is made public, recreated during a migration, or edited in ways the API does not expose. So:

Treat the dates as indicative. They are the best signal the API offers and they hold up well under inspection, but this is a tool for keeping on top of the catalogue — not a citable record of when a dataset was released. If a specific date matters, verify it on the dataset page.

Two lumps are visible and are genuine portal history rather than data problems:

Month	Datasets	Reading
2015-12	113	Portal launch seed load
2017-11	91	Second large onboarding wave

What this cannot see

• Deletions and withdrawals. metadata_created only ever grows. A dataset removed from the portal simply stops appearing in future pulls; it does not produce a "removed in month X" entry. Catching that needs a diff between two data/catalogue.json commits — the git history makes it possible, the build does not do it yet.
• Updates to existing datasets. metadata_modified is captured in data/catalogue.json but is deliberately not used for the month tree; it churns on every resource refresh and would swamp the actual additions.
• Private/unpublished datasets, by definition.
• Backdated entries. Inferred: nothing observed suggests publishers can set metadata_created retroactively, but if one ever did, a past month's file would change on rebuild. That shows up as a diff in an old month — worth a glance rather than assuming old months are frozen.

Also worth knowing: status_show returns HTTP 403 on this portal while the rest of the action API is open, and per-organisation package_count comes back null from organization_list, so publisher counts must be computed from search results. Both were first documented in Israel-Open-Data-Resources.

English is machine translation

data.gov.il publishes no English metadata at all. Checked across all 1197 packages on 2026-07-27: there is no title_translated field, and extras contains only ContainPrivateData, is_geographic, Accessibility, coordinates, desc_spatial_cover, geodetic_ref_sys, ref_number and tags_in_dataset — nothing translation-shaped.

So every English title and description here is machine-translated, and the Hebrew is the authoritative text. datasets.csv carries a title_en_source column per row:

title_en_source	Rows	Meaning
google/gemini-2.0-flash-001	1168	Inherited from the sibling translations repo (snapshot 2026-05-06)
gpt-4.1-mini	23	Translated by this repo, for datasets the snapshot did not cover
as published	6	Publisher wrote a Latin-script title; passed through unchanged

A caution on as published: it means "the title contained no Hebrew", not "the title is English". A handful of portal titles are published in Arabic, Spanish and French (tqar_a3, tqes, tqfr — theory-test question banks in those languages), and 24 more mix Hebrew and English in one string. Those are translated like anything else; only titles that survived translation unchanged are marked as published.

scripts/translate.py is hash-incremental: a dataset is re-translated only when it has no cached English, or when its Hebrew title/description has changed since it was last translated. Steady state is a handful of datasets per month.

Provider and a real failure mode

The script takes OPENAI_API_KEY (default gpt-4.1-mini) or OPENROUTER_API_KEY (default google/gemini-2.0-flash-001), OpenAI first. Override with OPENAI_MODEL or OR_MODEL.

gpt-4.1-mini intermittently returns finish_reason: "tool_calls" with message.content: None on requests that define no tools at all. It reproduces on particular batches, not particular datasets, and retrying the identical batch does not clear it — the same batch of 11 failed four times in a row. Splitting the batch fixes it: all 11 succeeded when sent individually. The script therefore retries failures one at a time before giving up, and treats content is None as a retryable error rather than crashing on .strip() of None.

Second, subtler bug worth not reintroducing: the singles-retry pass must select records whose source hash is stale, not only records with no English at all. Datasets whose Hebrew description changed already have English cached, so a "missing English" filter skips them and silently leaves the English describing an older version of the dataset.

Folder naming

Months are months/YYYY-MM/, not the MMYY form used elsewhere in these projects (060526 = 6 May 2026). With 124 months spanning 11 years, MMYY does not sort chronologically as a string — 0126 sorts before 1225 — so ls and GitHub's file listing would both scramble the timeline. YYYY-MM sorts correctly everywhere.

Rebuilding

./build.sh                 # fetch, translate what's new, rebuild
./build.sh --offline       # rebuild from cached catalogue + translations
./build.sh --no-translate  # fetch and rebuild, leave translations untouched

Needs the venv for the translation step:

uv venv .venv && uv pip install -r requirements.txt

The build is idempotent — it rewrites every month from scratch, so a run with no upstream change produces no diff. That is what makes git diff after a build a meaningful answer to "what did the portal add since last time".

data/translations.json is this repo's own cache and the only copy of the English text. It was seeded once from the sibling repo and must never be overwritten from there again — an earlier version of build.sh re-downloaded it on every run, which would have discarded every locally-translated entry.