Contributing to shlib

August 30, 2026 · View on GitHub

Everything you need to change this library without breaking somebody else's install script. It is worth reading The vendoring problem first: copies of this code are frozen into hundreds of third-party installers that will never be updated, and that single fact explains most of the rules here.

Two sections are worth knowing about before you open a pull request:

Deliberate choices that look like bugs -- several things here are wrong on purpose, and have been "fixed" before.
Where help is wanted -- scoped, measured tasks if you are looking for somewhere to start.

Companion documents: docs/PORTABILITY.md for shell and platform behaviour, docs/INSTALLERS.md for building a curl | sh installer, docs/API.md for the function index.

What this is

shlib is a set of portable shell functions for curl | sh installers -- detect the platform, download a file, verify a checksum, unpack an archive, place a binary. Each function lives in its own <name>.sh file and is meant to be concatenated into somebody else's script, not sourced as a dependency. A curl ... | sh installer cannot have dependencies, so the code has to travel with it.

That single fact drives almost every design decision here.

The code is written in POSIX sh, but that is the constraint, not the claim. POSIX describes neither where this has to run nor what it has to defend against: Solaris and illumos ship pre-POSIX tools, Windows via git bash is not a POSIX environment at all, and several shells in the matrix are not POSIX shells by default. Writing to the smallest common subset is how the functions reach those places -- see Conventions for the rule and docs/PORTABILITY.md for where reality falls short of the standard.

Shell- and platform-specific gotchas live in docs/PORTABILITY.md, not here: which shells break which idiom, which systems ship pre-POSIX tools, and how to rehearse a platform you cannot boot. Reach for it when one CI leg is red and the rest are green. This file stays on conventions, architecture and open work.

The vendoring problem

Because copies travel, they go stale, and the copies never hear about fixes. Measured with gh code search (Aug 2026):

what	files
carry shlib's `not a GOOS value` message	920
still say `file bug at .../client9/shlib`	812
generated by `godownloader`	526
contain the MinGW fix (Dec 2018)	662 — so ~258 still lack it
contain `illumos` / `midnightbsd` (2021)	97 / 55

The not a GOOS value phrasing was retired after v2026.08.28 (the checks now say not a recognized OS name / architecture name), so it stays a usable search fingerprint for the existing population and additionally dates a copy as pre-2026.08.29. Newer copies are dated by their version marker instead.

Two real bug reports arrived in 2026 for bugs fixed in 2018 and 2021. Neither reporter ran shlib; both ran a frozen copy embedded in an install script (fossa-cli, generated 2019-07-23; Bearer, copied from Trivy's copy).

v2026.08.28 is the first bundle that is not silently broken. v2026.08.27 shipped with the minifier bug below -- valid checksums, missing code -- so anyone who vendored that tag should re-vendor. Verified on the published artifacts: Windows_NT resolves to windows under v2026.08.28 and to windows_nt under v2026.08.27.

When triaging a bug report, first establish which vintage the reporter is running. Since v2026.08.27 the bundle self-identifies:

sed -n 's/^shlib \(.*\)/\1/p' their-install.sh

Older copies must be dated by diffing uname_os against this repo's history.

Conventions

POSIX sh only. No local, no [[, no == in [, no arrays, no $'...'. Variables are effectively global; that is accepted style here, not a bug to fix.
One function (or small family) per file, named after the file.
Functions report errors with log_err / log_crit from log.sh, which needs echoerr.sh. Anything with a failure path therefore depends on both — this is the most common mistake when hand-assembling a bundle.
Error messages are generic. Do not put a bug-reporting URL in them: 812 stale copies already do that and misroute other projects' bugs here. The shlib URL belongs only in license.sh / license_end.sh.
Internal variables are prefixed _shlib_. The library is concatenated into other people's scripts, so a bare os= or version= clobbers the caller. It used to: github_release overwrote $version and uname_os overwrote $os. Every internal scratch variable now carries the prefix, and dist_test.sh fails if any unprefixed lowercase global escapes.

Not prefixed, because they are the documented contract: the installer config names (OWNER REPO BINARY BINARIES FORMAT BINDIR PLATFORMS PLATFORM OS ARCH VERSION TAG NAME TARBALL CHECKSUM …) and standard environment (TMPDIR, GITHUB_TOKEN). local is not POSIX, which is why prefixing is the mechanism.
Optional arguments must be written ${2-}, never $2. The library is concatenated into other people's scripts, and an install script is exactly the kind of thing that starts with set -eu. Under nounset a bare $2 for an argument the docs call optional does not default to empty -- it aborts the shell. github_release owner/repo (tag omitted) did precisely that, and it took the whole NetBSD and OpenBSD legs down before a single test ran, because the vmactions runner executes its script under set -u. Same for an optional environment variable: ${GITHUB_TOKEN-}. nounset_test.sh pins every documented call form; add to it when adding an optional argument.
awk, grep, cut, tr, mktemp, tar are fair game. Assume busybox versions of all of them -- and Solaris's pre-POSIX ones, which is where the surprises are: see docs/PORTABILITY.md.

Commands

make test                        # all tests under /bin/sh
make test TEST_SHELL=dash        # ... under one shell
make test TESTS=untar_test.sh    # ... one file
make test-all                    # ... every shell installed locally
make lint                        # shellcheck (sh/bash/dash/ksh) + scripts/ + install/ + dist/ + shfmt
make fmt                         # shfmt -ci -p -i 2 -w
make dist                        # rebuild dist/ bundles
make docs                        # regenerate docs/API.md
make hooks                       # enable the pre-commit hook
make tools                       # pinned shellcheck + shfmt into ./bin (gitignored)

make lint needs ./bin/shellcheck; run make tools first.

Pre-commit hook

make hooks points core.hooksPath at .githooks/, enabling a pre-commit hook that runs the fast, offline half of CI:

make dist docs and a diff against dist/ + docs/API.md (0.1s). This is the mistake most easily made -- edit a library file, forget to regenerate, and the generated CI job goes red after the push.
make lint (4.4s), but only when ./bin/shellcheck exists. On a fresh clone it warns and allows the commit rather than blocking on a tool that has not been downloaded yet; CI is the real enforcement.

Deliberately excluded:

make test needs the network and fails offline (verified: 3 assertions in http_download_test.sh). A hook that blocks commits on a train gets disabled.
make fmt rewrites files mid-commit, changing what is being committed. make lint already fails on unformatted files and says to run it.

Known limitation: the hook checks the working tree, not precisely what is staged, so an unstaged edit can mask a problem CI then catches. Fixing that means stashing mid-hook, which risks losing work. git commit --no-verify bypasses it.

Testing

assert.sh provides assertTrue/assertFalse/assertEquals/assertNotEquals plus assert_skip. Assertions are non-fatal — a failure is recorded and the file continues, so one bad assertion no longer hides everything after it. An EXIT trap prints totals and sets the exit status, so a test file cannot forget to report. (Verified: exit inside an EXIT trap behaves identically in all 13 shells in the matrix.)

Conventions that matter:

Stub uname to test mapping tables. uname_os_test.sh and uname_arch_test.sh define a local uname() and unset -f uname afterwards, so the whole case statement is exercised on any machine. Without this the suite only ever tested the host's own branch — which is exactly how the illumos bashism survived from 2021 to 2026.
Every test_foo invocation needs a test_foo() definition. A lost definition prints "command not found" and the file can still report ok. Both scripts/lint.sh (static check) and assert.sh (zero assertions is a failure) now guard this; it slipped through twice before that.
A regression test must fail against the old code. Every fix in this repo was checked that way; several "tests" passed both before and after until corrected. Prove the discriminating power, do not assume it.
Assertions hold back stderr and show it only on failure. Negative tests are expected to make things complain -- assertFalse "hash_sha256 NONEXISTANT" really does run sha256sum on a missing file. That produced 25 lines of stderr against 14 of stdout on a fully green run, which made passing runs look broken and buried real failures. _assert_run stashes stderr to a temp file and _assert_show_stderr prints it, indented, under the FAIL line. A passing run is now silent on stderr in every shell.

eval still runs in the current shell, not a subshell, so assertions with side effects behave exactly as before.
Skip, don't fail, on a missing dependency (assert_skip), e.g. openssl or zstd. But make CI install those tools, or coverage is hollow.
A stub that always succeeds cannot test a refusal.
Tests run from the repo root and use fixtures/.

Shell-specific traps in the harness itself -- how eval hides a nounset abort on ksh93 and zsh, why a stub cannot be scoped with ( … ) -- are in docs/PORTABILITY.md.

CI

Eleven CI workflows (28 test legs), plus release.yml on v* tags. Badges are workflow-scoped — no per-job or per-matrix-leg badge exists — which is why they are split by platform rather than one ci.yml.

workflow	covers
`lint$	\text{shellcheck} \times 4 \text{dialects}, $scripts/`,` dist/shlib.sh`, shfmt, a static check for flags Solaris does not have, and the` dist` sync job
`linux`	dash, bash, ksh93, mksh, yash, posh, busybox ash
`macos`	sh, bash 3.2, ksh, zsh, dash
`freebsd`	14.4 + 15.1 via `vmactions/freebsd-vm` (QEMU on a Linux runner; FreeBSD `sh` is a distinct ash)
`openbsd`	7.9 + 7.8 via `vmactions/openbsd-vm` — full suite under `sh` (OpenBSD ksh), and the only leg where `http_download`'s `ftp(1)` branch runs without header support
`netbsd`	11.0 + 10.1 via `vmactions/netbsd-vm` — full suite under `sh` (ash) and `ksh` (pdksh); the leg where `ftp -H` puts a real header on the wire, and where the deliberate refusal of `Accept` is pinned
`dragonflybsd`	6.4.2 via `vmactions/dragonflybsd-vm` — full suite under `sh`; `DragonFly` → `dragonfly` needs no fixup. The image ships curl, so this leg calls `http_download_fetch` directly rather than asserting curl's absence
`sunos`	Solaris 11.4 and OmniOS via `vmactions/*-vm` — full suite under `sh`, plus an assertion that `SunOS` resolves to `solaris`/`illumos` for real rather than stubbed
`alpine`	busybox ash on musl, via `docker run` (not `container:` — `actions/checkout` needs glibc)
`windows`	git bash + msys2 — full suite, plus an assertion that `MINGW64_NT-`/`MSYS_NT-` map to `windows`
`runtimes`	`python:3.12-slim` and `node:22-slim` via `docker run` — each asserts curl, wget, fetch and ftp are all absent, then runs the full suite and a real install through `http_download_python` / `http_download_node`

Notes below are about how the legs are built. What each platform actually does differently -- the reason a leg goes red -- is in docs/PORTABILITY.md.

Windows is supported through git bash and msys2, where the full suite runs and the uname mapping is asserted against the real thing. It is not a POSIX environment, and native cmd/PowerShell is not a target. The leg exists because a git-bash user's report is what surfaced a stale-copy bug, and the uname mapping had previously only been tested with a stubbed uname. WSL is deliberately excluded: it reports Linux, so the linux workflow already covers it.
The sunos legs assert the uname_os result before running the suite, so a failure later still leaves the mapping result in the log. Both legs now run the full suite; they install no packages and use no gmake, which removed a failure mode unrelated to what they test.
uses: cannot take a matrix expression, which is why solaris and omnios are two jobs rather than two legs of one.
A negative assertion must pin the reason, not just the failure. The openbsd leg checks that github_release cannot resolve latest without an Accept header. Its first version only checked that the call failed, and it duly went green while the real cause was the set -u abort above. It now matches the error message.
An example config that makes an ABI claim has to be proved on hardware that has that ABI. install/examples/hugo.sh maps illumos onto hugo's solaris build, because hugo publishes no illumos asset. Whether a Solaris-built Go binary runs on illumos is not a question about shell, so install_test.sh can only check that the config produces hugo_..._solaris-amd64.tar.gz. The omnios leg therefore assembles that config into a real installer, runs it, and executes the binary. The static evidence pointed the same way -- interpreter /lib/amd64/ld.so.1, NEEDED libsendfile.so/libsocket.so/libc.so, no versioned symbol needs -- but that is an argument, not a result. The result: green on OmniOS r151054 on 2026-08-28, the binary reporting itself as hugo v0.165.0 solaris/amd64 while running on illumos.
Do not assert the absence of an alternative when you can exercise the branch directly. The dragonflybsd leg first asserted no curl, copying the freebsd leg; the vmactions DragonFly image ships curl in /usr/local/bin, so the leg failed on its own premise. It now calls http_download_fetch directly, which tests the branch rather than a property of somebody else's disk image. The freebsd/openbsd/netbsd legs keep their absence assertions because there the absence IS the thing under test.
The QEMU legs run as root, which quietly changes what a test means: any assertion that something is forbidden is vacuous there unless it guards for it. docker run alpine is a cheap way to check the root path of such a guard without booting a VM.

Installers (`install/`)

The replacement for the archived godownloader. An install script is config.sh + dist/install-base.sh concatenated -- no Go, no YAML, no template language, because a shell function is already a lazily-evaluated template.

install/runner.sh defines functions only, so tests can source it.
install/main.sh runs the flow and is concatenated LAST, so a truncated curl | sh cannot do a partial install.
The config is concatenated FIRST, so runner.sh fills in adjust_* hooks only when absent (via command -v, not is_command -- if the latter were missing the guard would fail open and clobber the project's hook).
PLATFORMS is declared by the project, giving a real error instead of a 404.
FORMAT is a filename suffix, not a format identifier. It is consumed in exactly one place -- tarball_name appends it to what archive_name returned -- and is never compared against anything, so its legal values are precisely the suffixes untar matches. The name is inaccurate (TARBALL is too, when FORMAT=zip) but both are in the documented contract and in every vendored config, so they are documented rather than renamed.
Whether the download needs unpacking is a SEPARATE axis from what it is called, which is why unpack is a hook and there is no FORMAT=binary sentinel. hadolint proves they are independent: its windows asset is hadolint-windows-x86_64.exe -- a non-empty suffix that is still not an archive -- so no single field can express both. A sentinel would also have needed special-casing in two places (suppress the dot, skip the untar).
Clean up around a function, not at the end of it. execute freed its temp directory in its last statement, so every || return 1 leaked one. The fix is a wrapper (execute) around a body (_shlib_execute) so cleanup runs on all paths. An EXIT trap -- the idiom mktmpdir.sh documents -- is not usable here: assert.sh installs its own EXIT trap to print test totals and install_test.sh calls execute directly, so a trap set inside would silence the whole test report.
In install/runner.sh, unprefixed function names are the ones main.sh calls; _shlib_-prefixed ones are private. That distinguishes them from the hooks a config is meant to override (unpack, binary_path, adjust_*, latest_version). install_test.sh's untar stub originally never failed, so test_execute_bare_binary passed even with execute still calling untar directly -- the bug it exists to catch. The stub now refuses an unrecognised suffix the way the real one does. Checked by reverting each of the three changes in turn and confirming the matching test goes red.

The platform traps this code has hit -- Solaris's /usr/bin/unpack shadowing the hook, posh's getopts, Windows [ -x ] -- are in docs/PORTABILITY.md.

See docs/INSTALLERS.md.

Documentation

docs/API.md is a generated index: function, one-line summary, link to source. scripts/mkapi.sh takes the first meaningful line of the comment block above each function, so the source stays the single place documentation is written.

Run make docs after adding or renaming a function, and commit the result -- the generated CI job runs make dist docs and fails on any diff.
A function with no summary comment makes make docs fail outright, so new code cannot land undocumented.
Deliberately an index, not a reference: arguments, returns and side effects live in the source comments. A fuller convention (Arguments: / Returns: / Side effects: headers, extracted into the table) is a possible next step.
install/runner.sh is excluded; it is internal to a generated installer and documented in docs/INSTALLERS.md.

Releases and `dist/`

dist/shlib.sh (the functions) and dist/install-base.sh (functions plus the installer flow) are generated by scripts/dist.sh and committed, so consumers can fetch a stable raw URL at build time instead of hand-vendoring:

https://raw.githubusercontent.com/client9/shlib/master/dist/shlib.sh

Nothing is stripped or minified. cat is the only transformation, and that is deliberate. There used to be a dist/shlib.min.sh built by piping the bundle through grep -v '^[[:space:]]*#':

It shipped a silently broken release. The original idiom also filtered #, which deleted code lines carrying trailing comments -- v2026.08.27 went out with valid checksums, no win*) os="windows" mapping, and no gitrepo= assignments in git_clone_or_update.
Even fixed, it constrained what the library could contain: any embedded awk, sed or python carrying a whole-line # would be silently gutted.
It bought about 10 KB gzipped, once, at install time. GitHub raw serves gzip; the TLS handshake costs more than the difference.
A curl | sh script that users are told to read before running is more useful with its comments intact.

Do not reintroduce it.

The version marker still lives inside a cat /dev/null <<EOF heredoc rather than a # comment. That began as a way to survive stripping; it stays because sed -n 's/^shlib $.*$/\1/p' is the documented way to date a vendored copy and must keep matching the copies already in the wild.

The version comes from the VERSION file. Do not "improve" this. Both obvious alternatives were tried and both break the dist sync job:

date -u changes daily, so dist/ shows a diff every day.
git log counts only committed history while make dist bundles the working tree, so the version lags a commit and then jumps — making freshly committed dist/ instantly stale. Chicken-and-egg.

A plain file is stable, needs no git, and works in shallow clones and tarballs.

CalVer (2026.08.27), tagged v2026.08.27. Step-by-step in docs/RELEASING.md; consumer guidance in docs/EMBEDDING.md.

If you edit any library .sh, run make dist and commit the result or the dist CI job fails.

Deliberate choices that look like bugs

The platform names are shlib's, not Go's. They match GOOS/GOARCH because that is where the artifact-naming convention came from and compatibility is worth keeping -- but Go is provenance, not authority. The admission rule, written down in uname_os_check.sh and generated into docs/API.md: a name is recognized when some real system's uname maps to it AND it is the spelling projects use for release artifacts. Nothing is added because Go added it or dropped because Go dropped it. The three consequences:
- midnightbsd is accepted by uname_os_check but is not a GOOS. Added by MidnightBSD's maintainer (PR #33); kept on purpose.
- nacl and amd64p32 were removed from Go in 1.14 but are still accepted, to avoid breaking unknown downstreams. Commented as historical.
- uname_arch returns armv5/armv6/armv7, not Go's arm + GOARM. Intentional -- artifacts are named that way.
sunos is the reverse case: real uname -s, but no project names an artifact for it, so it is resolved to solaris/illumos and never returned.

Projects using raw kernel spellings (x86_64, aarch64) map back with the installer's adjust_arch hook. That round-trip is the cost of the Go-derived vocabulary for non-Go projects; it is accepted, not overlooked.
hash_* functions avoid pipes into cut on the same line as the hashing command, so failures surface without pipefail.

Where help is wanted

In rough priority order. These are real, scoped tasks, not aspirations -- each one below has been measured rather than guessed at.

1. Resolving `latest` without content negotiation

github_release asks https://github.com/OWNER/REPO/releases/latest for Accept: application/json. Neither OpenBSD's ftp nor NetBSD's tnftp can deliver a usable Accept (see the note in the CI section), so latest is dead on both — which is most of what an installer needs.

Three findings from measuring this, rather than guessing:

The specific-tag case needs no JSON at all. releases/tag/<tag> answers 200 for a tag that exists and 404 for one that does not, so github_release owner/repo v1.2.3 — which only validates existence — could be a plain http_download status check on every downloader we support. This is the easy half and could be done on its own.

releases.atom is header-free but not prerelease-aware. It needs no Accept, so every downloader can read it, and entries are newest-first with the tag in a parseable id:

sed -n 's/.*<id>tag:github.com,2008:Repository\/[0-9]*\/\(.*\)<\/id>/\1/p' | head -1

But it is chronological, not filtered the way releases/latest is. Measured: neovim/neovim reports v0.12.5 for latest and nightly as its first atom entry. Taking the first entry would install a nightly build. There is no prerelease marker in the feed to filter on. It is also fat — the feed embeds full release-note HTML for the last several releases, 290 KB for cli/cli against a few KB of JSON, which matters for curl | sh.

The clean answer exists but is out of reach. releases/latest is a 302 to releases/tag/<tag> — with correct prerelease and draft semantics, and regardless of Accept. The tag is right there in the Location header. But no BSD base downloader can expose a response header or stop at a redirect: curl has -w '%{redirect_url}' and -I, fetch and both ftps have nothing equivalent. http_last_modified has the same shape of problem.

So the realistic shape is: status check for an explicit tag, atom feed for latest with the prerelease caveat documented and probably a log_err when the chosen entry cannot be confirmed non-prerelease. Worth doing only if BSD installers matter; a config can already override latest_version() today.

2. Untested functions

git_clone_or_update has no test, plus the http_download_curl/_wget branches (exercised indirectly through http_download, and by nounset_test.sh, but never directly for what they download). github_api has nounset coverage only.

http_last_modified was on this list until a head -c in it -- a GNU/BSD extension Solaris does not have -- shipped for years without anything catching it. http_last_modified_test.sh stubs curl with canned header blocks, so it needs no network; that is what had kept the function untested.

3. Possible, not planned

Outreach to the ~258 repos still carrying the pre-2018 MinGW bug. gh code search finds them; see the table at the top.