Testing Documentation for WifiWand

June 22, 2026 · View on GitHub

This document describes how WifiWand's test suite is organized and how to run the right scope for local development, CI, and native-host verification.

Overview

The suite has three practical buckets:

Default tests: mocked or otherwise hermetic tests. These are safe for CI and run by default.
:real_env_read_only: tests that read the real host environment but do not intentionally mutate it.
:real_env_read_write: tests that mutate the real host environment and therefore require capture/restore safeguards.

The :real_env tag is derived automatically whenever either :real_env_read_only or :real_env_read_write is present. That makes it easy for the harness to exclude all real-host tests by default.

Quick Start

# Default safe suite
bundle exec rspec

# Include real-host read-only checks
WIFIWAND_REAL_ENV_TESTS=read_only bundle exec rspec

# Include the full native-host suite, including read-write tests
WIFIWAND_REAL_ENV_TESTS=all bundle exec rspec

Fail-Fast And Verbose Debugging

When debugging real-environment failures, it is often best to stop on the first failure and print the underlying OS commands.

Use fail-fast directly with RSpec:

# Stop on the first failure in the full real-host suite
WIFIWAND_REAL_ENV_TESTS=all bundle exec rspec --fail-fast

# Equivalent explicit one-failure form
WIFIWAND_REAL_ENV_TESTS=all bundle exec rspec --fail-fast=1

If you prefer the Rake wrappers, pass fail-fast through RSPEC_OPTS:

RSPEC_OPTS="--fail-fast" bundle exec rake test:all

To print the underlying OS commands while the suite runs:

WIFIWAND_REAL_ENV_TESTS=all WIFIWAND_VERBOSE=true bundle exec rspec --fail-fast

Or with the Rake wrapper:

WIFIWAND_VERBOSE=true RSPEC_OPTS="--fail-fast" bundle exec rake test:all

For narrower debugging, combine these with a single spec file:

WIFIWAND_REAL_ENV_TESTS=all WIFIWAND_VERBOSE=true bundle exec rspec \
  spec/wifi_wand/platforms/mac/model_spec.rb --fail-fast

Targeted Rake Tasks

Use the Rake tasks when you want the convenience of scope selection plus a specific spec file or files.

Examples:

# Read-only real-host tests for one file
bundle exec rake 'test:read_only_target[./spec/wifi_wand/platforms/mac/model_spec.rb]'

# Full real-host tests for one file
bundle exec rake 'test:real[./spec/wifi_wand/platforms/mac/model_spec.rb]'

# Multiple targets are passed through to RSpec
bundle exec rake 'test:real[./spec/a_spec.rb ./spec/b_spec.rb]'

Shell behavior matters here:

In bash, the quotes are optional.
In zsh, the quotes are required because [ and ] are treated as glob syntax.
Preferred cross-shell form: bundle exec rake 'test:real[./spec/path_spec.rb]'
zsh alternatives: escape the brackets or prefix with noglob

Examples for zsh:

bundle exec rake test:real\[./spec/wifi_wand/platforms/mac/model_spec.rb\]
noglob bundle exec rake test:real[./spec/wifi_wand/platforms/mac/model_spec.rb]

CI Guidance

CI should run only the default suite.

Do not enable WIFIWAND_REAL_ENV_TESTS in CI:

CI runners often lack WiFi hardware.
Even read-only real-host tests depend on machine-specific state.
Read-write real-host tests can disrupt connectivity and require restoration.

Tag Model

`:real_env_read_only`

Use this for tests that touch the real machine but only read from it.

Examples:

detect the WiFi interface
read IP address or nameservers
read macOS version from the host
scan networks when WiFi is already on

These tests are not suitable for CI, but they do not need the suite-level restore harness.

`:real_env_read_write`

Use this for tests that intentionally change real machine state.

Examples:

turn WiFi on or off
disconnect from the current network
connect to a network
change nameservers

These tests trigger the suite's real-host restoration flow.

`:real_env`

This tag is derived automatically from the two tags above. Do not set it manually.

It is used for:

excluding all real-host tests by default
OS-specific gating for real-host examples
deciding whether macOS auth preflight should run

OS Gating

Mocked tests run on any supported host.

Real-host tests can additionally declare real_env_os: :os_mac or real_env_os: :os_ubuntu. The suite detects the current OS and skips foreign real-host examples automatically.

Network Restore Behavior

Only :real_env_read_write tests participate in suite-level network capture and restoration.

Behavior:

before the suite, the current network state is captured once
after each :real_env_read_write example, the suite restores that state
at suite end, the suite attempts a final restoration and raises if restoration fails

Recommended Real-Host Coverage

When validating :real_env_read_write behavior locally, test both of these network types when possible:

an open network that does not require a password
a password-protected network that requires stored or interactive credentials

Both cases matter. Open networks exercise disconnect/reconnect behavior without keychain-backed password lookup, while password-protected networks exercise saved-password capture, restore-time reconnects, and macOS authentication/keychain edge cases. A change that passes on one type can still fail on the other.

If macOS redacts the current SSID during preflight and the suite aborts because it cannot capture a restorable network name, fix helper permissions before running the real-host suite. Run wifiwand-macos-setup, grant Location Services access to wifiwand-helper, then rerun the tests.

There is no supported environment-variable override for the restore target. The suite must be able to read and later verify the exact original SSID so it can restore host network state safely.

macOS Preferred-Network Removal Moved Out Of Automated Test Suite

The automated suite no longer exercises the privileged macOS preferred-network removal path.

That change is intentional:

rspec must not silently enter sudo mode.
Local test runs must not prompt for elevated credentials.
CI and default local testing must stay non-privileged and predictable.

The production code path still exists in WifiWand::Platforms::Mac::Model#remove_preferred_network, which calls:

sudo networksetup -removepreferredwirelessnetwork <iface> <ssid>

Verification for that path is now manual-only. When you want occasional coverage of the real macOS removal flow, use an SSID you genuinely no longer need and verify removal through the CLI:

bundle exec exe/wifiwand forget "Example SSID"
bundle exec exe/wifiwand pref_nets | grep "Example SSID"

Interpretation:

If forget succeeds and grep finds no match, the preferred-network entry was removed.
If forget fails with an authentication or networksetup error, investigate that production path directly.

Use this carefully. It mutates the host's saved preferred-network state by removing the named SSID from macOS's preferred wireless network list.

This manual check preserves occasional coverage only for the preferred-network removal path. It does not replace automated tests for ordinary behavior, and it must not expand into a general privileged host-mutation harness.

In particular, it is not for:

nameserver or DNS mutation checks
other system-configuration mutation checks
CI or default local test flows

Coverage Artifacts

Two resultset filenames are used depending on whether the run touches the real host:

coverage/.resultset.json This is used for ordinary runs, including the default safe suite.
coverage/.resultset.<os>.json This is used for any real-environment run, including both: WIFIWAND_REAL_ENV_TESTS=read_only bundle exec rspec and WIFIWAND_REAL_ENV_TESTS=all bundle exec rspec

The OS suffix appears only for real-environment runs so those host-dependent artifacts do not overwrite the default mocked/hermetic resultset.

Interpreting Coverage Correctly

Coverage artifacts are only authoritative for the exact test run that generated them.

If you run a filtered test subset, the resultset reflects that subset only.
If source files change after the resultset is written, coverage tools may report stale entries such as length_mismatch.
Developers and agents must not treat an existing coverage file as proof of whole-codebase coverage unless they intentionally ran a fresh unfiltered suite for that purpose.

Practical rule:

For codebase-wide review or planning, first run a fresh unfiltered suite such as bundle exec rspec or bundle exec rake test:safe, then inspect the newly generated resultset.
Treat the overall SimpleCov percentage as incomplete unless the review also checks tracked runtime files. SimpleCov starts from lib/**/*.rb and exe/*, then excludes maintainer-only files that are also excluded by the gemspec. Use cov-loupe when deciding whether packaged runtime files are covered, uncovered, or missing from the resultset.
Branch coverage is enabled by default.
bin/* contains maintainer tooling that is not shipped as gem executables, so it is not part of normal runtime coverage accounting.
For targeted local work, it is fine to rely on targeted coverage artifacts, but only for the code exercised by that run.

This is an operator responsibility, not an automatic guarantee provided by the resultset filenames.

Modifier Env Vars

These env vars are orthogonal to test scope and can be combined with any rake task or rspec invocation:

# Show underlying OS commands during tests
WIFIWAND_VERBOSE=true bundle exec rake test:read_only

# Disable Cobertura formatter loading and XML output
WIFIWAND_COBERTURA_COVERAGE=false bundle exec rake test:safe

WIFIWAND_COBERTURA_COVERAGE=false only affects runtime formatter loading. Because Gemfile declares simplecov-cobertura as a development/test dependency, Bundler still resolves it before tests run. When testing a SimpleCov version that is incompatible with simplecov-cobertura, modify Gemfile too and refresh the bundle.

Writing Tests

Use the smallest scope that matches reality:

# Default mocked/hermetic test
it 'returns wifi status' do
  expect(subject.wifi_on?).to be(true).or(be(false))
end

# Real-host read-only test
it 'reads the current IP addresses', :real_env_read_only, real_env_os: :os_ubuntu do
  expect(subject.ipv4_addresses).to all(match(/^\d+\.\d+\.\d+\.\d+$/))
end

# Real-host read-write test
it 'disconnects from the current network', :real_env_read_write do
  expect { subject.disconnect }.not_to raise_error
end

Guidelines:

Prefer mocked tests whenever behavior can be exercised without the host.
Use :real_env_read_only for host-dependent checks that must stay read-only.
Use :real_env_read_write only for the small set of tests that truly need mutation.
Add real_env_os: when a real-host test only makes sense on one OS.

Why macOS WiFi-Toggling Operations Have No Real-Environment Tests

macOS only. This limitation is specific to macOS. The same operations may be testable under real-environment conditions on other platforms.

The following methods have no real-environment tests on macOS:

#disconnect (BaseModel)
#wifi_on / #wifi_off (WifiWand::Platforms::Mac::Model)

All are fully covered by mocked unit tests that exercise every code path. Real-environment tests were written and investigated but ultimately removed.

Root cause: macOS airportd auto-reconnect

On macOS, the airportd daemon monitors preferred-network associations and starts an auto-reconnect cycle within milliseconds of any programmatic disassociation or WiFi toggle. This creates two problems:

Problem 1 — the test itself becomes unreliable. For #disconnect, the test always lands in the error branch because airportd reconnects faster than our stability check can confirm disassociation. The clean-disconnect path is never reachable.

Problem 2 — the restore phase fails. After any test that toggles WiFi off or calls disconnect, the suite's global restore hook tries to reconnect explicitly via networksetup. This call races with airportd's own reconnect and produces -3900 tmpErr errors every time. The restore was failing for #wifi_on, #wifi_off, and #disconnect tests for this reason.

Alternatives considered

Approach	Problem
Remove network from preferred list before disconnecting	macOS reconnects to another preferred network instead
Turn WiFi off to prevent reconnect	Tests a different operation; same restore problem reappears
Private `airportd` API to suppress reconnect	Not accessible without private frameworks
Increase restore settle window (e.g. 60 s)	Only delays the inevitable; macOS always wins the race
Retry loop in restore	Retries still race with airportd; each attempt can -3900

Conclusion

On macOS, any real-environment test that leaves WiFi toggled off or the interface disassociated will trigger airportd's reconnect before our restore hook can act, causing non-deterministic -3900 tmpErr failures in the restore phase. There is no public API to suppress this behavior.

The policy for macOS is: remove real-environment tests for any operation whose post-test state triggers airportd competition. Mocked tests cover the logic fully. Real-environment tests for these operations on macOS would only test macOS's auto-reconnect behavior, not our code.

The same reasoning applies to any future macOS test that toggles WiFi state or calls disconnect — mocked coverage is sufficient.