Beamtalk Testing Strategy

This document describes the testing approach for the Beamtalk compiler and runtime.

Overview

Beamtalk uses a multi-layered testing strategy covering the Rust compiler, Erlang runtime, and language features:

Layer	Technology	Location	Purpose
Unit Tests	Rust #[test]	crates/*/src/*.rs	Test individual functions and modules
Snapshot Tests	insta	test-package-compiler/	Validate lexer, parser, and codegen output
Compilation Tests	erlc	test-package-compiler/	Verify generated Core Erlang compiles
Stdlib Tests	EUnit (compiled)	stdlib/bootstrap-test/*.btscript	Bootstrap primitive validation (no REPL needed)
BUnit Tests	EUnit (TestCase)	stdlib/test/*.bt	Language feature tests via TestCase (beamtalk test)
Runtime Unit Tests	EUnit	runtime/apps/beamtalk_runtime/test/*_tests.erl	Test Erlang runtime modules
Integration Tests	EUnit + daemon	runtime/apps/beamtalk_runtime/test/*_integration_tests.erl	Test REPL ↔ daemon communication
Codegen Simulation Tests	EUnit	runtime/apps/beamtalk_runtime/test/beamtalk_codegen_simulation_tests.erl	Simulate compiler output, test runtime behavior
REPL Protocol Tests (Rust)	Rust + REPL	tests/repl-protocol/	REPL TCP-protocol integration tests
Parity Tests	Rust + REPL/MCP/CLI/LSP	tests/parity/ (cases) + crates/beamtalk-parity-tests/ (harness)	Cross-surface equivalence checks (BT-2077)

Running Tests

Quick Check (CI equivalent)

just ci                  # Build, lint, test, test-stdlib, test-repl-protocol

Or individual steps:

cargo build --all-targets
cargo clippy --all-targets -- -D warnings
cargo fmt --all -- --check
cargo test --all-targets
just test-stdlib         # Bootstrap expression tests (fast, ~14s)
just test-bunit          # BUnit TestCase tests (~85 files)
just test-parity         # Cross-surface parity tests (BT-2077, ~30s)
just test-repl-protocol  # REPL TCP-protocol tests (slower, ~50s)

Code Coverage

Generate coverage reports for both Rust and Erlang tests:

Rust coverage:

# Text output with summary
cargo llvm-cov --all-targets --workspace \
  -- --skip commands::build::tests::test_build_single_file \
     --skip commands::build::tests::test_build_multiple_files \
     --skip commands::run::tests::test_run_calls_build \
     --skip erlang_runtime_unit_tests

# HTML report (opens in browser)
cargo llvm-cov --all-targets --workspace --html --open \
  -- --skip commands::build::tests::test_build_single_file \
     --skip commands::build::tests::test_build_multiple_files \
     --skip commands::run::tests::test_run_calls_build \
     --skip erlang_runtime_unit_tests

# Cobertura XML format for CI integration
cargo llvm-cov --all-targets --workspace --cobertura --output-path coverage.cobertura.xml \
  -- --skip commands::build::tests::test_build_single_file \
     --skip commands::build::tests::test_build_multiple_files \
     --skip commands::run::tests::test_run_calls_build \
     --skip erlang_runtime_unit_tests

Erlang coverage:

# Runs unit + E2E + stdlib suites under cover, merges, and generates XMLs
just coverage-all

The Erlang coverage badge blends all four runtime apps — beamtalk_runtime, beamtalk_workspace, beamtalk_stdlib, and beamtalk_compiler. The beamtalk_stdlib figure reflects only the hand-written FFI .erl modules: the bt@* compiled-Beamtalk modules carry no Erlang abstract code (erlc +from_core emits empty abstract forms), so cover cannot instrument them and they are auto-excluded (BT-1672). They are exercised instead by the .bt BUnit suite.

Integration-shaped modules (BT-2389)

Some modules are only reached by a live external client, not by the TCP repl_protocol E2E suite or plain unit tests:

• beamtalk_ws_handler — the Cowboy WebSocket handler for the REPL protocol (used by MCP/LSP/browser clients). Its callbacks (init/2, websocket_init/1, websocket_handle/2, websocket_info/2, terminate/3) are largely pure frame-builders over a #ws_state{} record. Rather than stand up a live WS client, they are driven directly by EUnit: the record is shared via apps/beamtalk_workspace/include/beamtalk_ws_state.hrl, and tests construct handler state + decode real protocol messages (beamtalk_repl_protocol:decode/1) to exercise each clause. The post-auth create_session/resume paths use a lightweight fixture that starts only beamtalk_session_sup. This is deterministic and needs no socket.
• beamtalk_build_worker — its pure compilation helpers (compile_core_erlang/1, compile_core_file/2, compile_modules/2, handle_read_specs/1) are unit-tested via -ifdef(TEST) exports; only the escript stdin loop (main/0/compile_loop/0) needs a real port.
• beamtalk_compiler_port / beamtalk_compiler_server — success paths are covered by live-port integration EUnit tests (they spawn the real Rust compiler port).

--app discovery gotcha: coverage-runtime runs rebar3 eunit --app=<app>, which only auto-discovers a source module's Module_tests companion. A standalone *_callbacks_tests suite compiles fine but is silently skipped under --app, so its coverage never reaches the merged badge. Keep callback/unit tests for a source module in that module's _tests companion (e.g. beamtalk_ws_handler_tests).

Genuinely unreachable (documented out-of-scope, not forced): the create_session {error, Reason} arm (the session supervisor's simple_one_for_one child start does not fail deterministically), the actor_snapshot_frames/0 live-registry branch, the compile_core_file/2 beam-write-error arm, and root-only permission_denied / TOCTOU error:badarg / exit:{noproc} defensive catches in beamtalk_file.

Coverage reports are saved to:

• Rust HTML: target/llvm-cov/html/index.html
• Erlang HTML: runtime/_build/test/cover/index.html
• Rust Cobertura XML: coverage.cobertura.xml
• Erlang Cobertura XML: one per app under runtime/_build/test/covertool/ (beamtalk_runtime, beamtalk_workspace, beamtalk_stdlib, beamtalk_compiler)

CI Integration:

Coverage metrics are automatically displayed in:

• GitHub Actions Summary - View in the "Summary" tab of any workflow run
• PR Comments - Sticky comment with coverage badges and details on all pull requests

No external services required - all coverage reporting is handled within GitHub Actions using $GITHUB_STEP_SUMMARY and PR comments.

Coverage Thresholds:

Metric	Minimum	Target	Action if Below Minimum
Overall Line Coverage	70%	80%	Fail CI build
Branch Coverage	80%	90%	Fail CI build
Unit Test Coverage	80%	90%	Flag in PR review

Current Coverage (as of BT-136):

Language	Line Coverage	Notes
Rust	81.98%	Overall workspace coverage
Erlang	34%	Overall runtime coverage; some modules unused in tests

Target Coverage:

• Rust unit tests: >90% line coverage
• Overall: >80% coverage for all test types

Note: Some Rust tests are skipped in coverage due to pre-existing failures unrelated to the test framework. See BT-136 for details.

Erlang Runtime Tests

cd runtime
rebar3 eunit

Individual Test Suites

# Just compiler snapshot tests
cargo test -p test-package-compiler

# Just core library unit tests
cargo test -p beamtalk-core

# Specific Erlang test module
cd runtime && rebar3 eunit --module=beamtalk_actor_tests

---

Test Types

1. Rust Unit Tests

Standard Rust #[test] functions colocated with the code they test.

Location: crates/*/src/*.rs (in #[cfg(test)] mod tests { ... })

Count: ~200 tests

Example (erlang.rs):

#[test]
fn test_generate_literal_integer() {
    let mut generator = CoreErlangGenerator::new("test");
    let lit = Literal::Integer(42);
    let result = generator.generate_literal(&lit);
    assert!(result.is_ok());
    assert_eq!(generator.output, "42");
}

What they test:

• Lexer token generation
• Parser AST construction
• Code generation helpers
• Type conversions
• Error handling

---

2. Compiler Snapshot Tests

Snapshot tests validate the compiler's output at each stage. Changes are reviewed in PRs.

Location: test-package-compiler/

Test cases: test-package-compiler/cases/*/main.bt

Snapshots: test-package-compiler/tests/snapshots/

Generated tests per case:

Test	Snapshot File	Purpose
test_{case}_lexer	*_lexer.snap	Token stream from lexer
test_{case}_parser	*_parser.snap	AST structure from parser
test_{case}_codegen	*_codegen.snap	Generated Core Erlang
test_{case}_compiles	(none)	Verifies erlc +from_core succeeds

Adding a new test case:

# 1. Create test directory
mkdir -p test-package-compiler/cases/my_feature

# 2. Add source file
cat > test-package-compiler/cases/my_feature/main.bt << 'EOF'
// Copyright 2026 James Casey
// SPDX-License-Identifier: Apache-2.0

// Test description
myMethod := [ ^42 ]
EOF

# 3. Generate snapshots
cargo test -p test-package-compiler

# 4. Review and accept
cargo insta review

Managing snapshots:

cargo insta review    # Interactive review
cargo insta accept    # Accept all pending
cargo insta reject    # Reject all pending

---

3. Compilation Verification Tests

These tests verify that generated Core Erlang actually compiles with erlc.

Location: test-package-compiler/tests/compiler_tests.rs (test_codegen_compiles)

Process:

1. Parse Beamtalk source
2. Generate Core Erlang
3. Write to temp file
4. Run erlc +from_core
5. Assert compilation succeeds

Skipping: Tests are skipped gracefully if erlc is not available.

Why this matters: Snapshots can drift from actual erlc requirements. These tests catch syntax errors that snapshots miss.

---

4. Stdlib Tests (Bootstrap Expression Tests)

Expression tests for bootstrap-critical primitives that TestCase transitively depends on. These must remain as expression tests because TestCase itself relies on these features working correctly.

Location: stdlib/bootstrap-test/*.btscript

Count: ~11 test files

Command: just test-stdlib

How it works: The beamtalk test-stdlib command compiles each .bt file, parses // => assertion comments, generates a thin EUnit wrapper, and runs via eunit:test/1. No REPL daemon is involved — tests compile and execute directly on BEAM.

Test file format (same // => format as E2E):

// Basic arithmetic
1 + 2
// => 3

5 negated
// => -5

// String operations
'hello' size
// => 5

With fixtures (@load directive, used in bootstrap tests only):

// @load tests/repl-protocol/fixtures/counter.bt
// (path is CWD-relative; the bootstrap suite resolves from the workspace root.)

Counter spawn
// => _

// Wildcard _ means "runs but don't check result"

Bootstrap files (DO NOT migrate to BUnit): arithmetic.bt, booleans.bt, equality.bt, errors.bt, exceptions.bt, custom_exceptions.bt, erlang_exceptions.bt, integer_test.bt, float.bt, string_methods.bt, string_ops.bt, symbol.bt, literals.bt, value_types.bt

Also kept as expression tests due to compile-time type check constraints: stack_frames.bt, class_hierarchy.bt

When to use stdlib expression tests:

Test needs...	Use stdlib test?
Bootstrap-critical primitives (arithmetic, strings, booleans)	✅ Yes
Tests that TestCase depends on transitively	✅ Yes
Tests that fail static type checks in BUnit (e.g. Number subclasses)	✅ Yes
All other language feature tests	❌ No — use BUnit tests

Adding a new stdlib test:

1. Create stdlib/bootstrap-test/my_feature.bt
2. Add expressions with // => expected_result annotations
3. Run just test-stdlib

Design: See ADR 0014 for the full rationale behind compiled expression tests vs E2E tests.

---

4b. BUnit Tests (TestCase Classes)

SUnit-style test classes that subclass TestCase. The primary home for language feature tests — collections, closures, regex, actors, reflection, and more.

Location: stdlib/test/*.bt (project test directory)

Count: ~85 test files

Command: just test-bunit or beamtalk test

How it works: The beamtalk test command first pre-compiles all .bt files in the fixtures/ subdirectory, making fixture classes available on the BEAM code path — similar to how all classes exist in a Smalltalk image. It then discovers .bt files containing TestCase subclass: definitions, compiles them through the normal pipeline, generates EUnit wrapper modules, and runs all test methods. Each test method starting with test is auto-discovered and run with a fresh instance. Limitation: currently only the first TestCase subclass in each .bt file is compiled (a warning is emitted if more are found), so put each test class in its own file.

Test fixtures: Place fixture classes in stdlib/test/fixtures/. All .bt files in this directory are automatically compiled and made available to all test files — no explicit loading needed. Just use the class name directly in your tests.

Test file format:

// stdlib/test/counter_test.bt
// Counter class is available from stdlib/test/fixtures/counter.bt — no @load needed

TestCase subclass: CounterTest

  testInitialValue =>
    self assert: (Counter spawn getValue await) equals: 0

  testIncrement =>
    self assert: (Counter spawn increment await) equals: 1

  testMultipleIncrements =>
    | counter |
    counter := Counter spawn.
    3 timesRepeat: [counter increment await].
    self assert: (counter getValue await) equals: 3

Lifecycle: For each test method: create fresh instance → setUp → test method → tearDown

Assertion methods:

Method	Description	Example
assert:	Assert condition is true	self assert: (x > 0)
assert:equals:	Assert two values equal	self assert: result equals: 42
deny:	Assert condition is false	self deny: list isEmpty
should:raise:	Assert block raises error	self should: [1 / 0] raise: #badarith
fail:	Unconditional failure	self fail: 'not implemented'

REPL integration: TestCase classes can also be run interactively:

> :load stdlib/test/counter_test.bt
> CounterTest runAll        // Run all tests in class
> CounterTest run: #testIncrement  // Run single test

When to use BUnit tests:

Test needs...	Use BUnit?
Language features (collections, closures, regex, etc.)	✅ Yes
Stateful test setup/teardown	✅ Yes
Multiple assertions per test	✅ Yes
Testing complex actor interactions	✅ Yes
Bootstrap-critical primitives	❌ No — use stdlib tests
REPL command testing	❌ No — use E2E

Adding a BUnit test:

1. Create stdlib/test/my_feature_test.bt with TestCase subclass: MyFeatureTest
2. Add test methods prefixed with test
3. Run just test-bunit

Design: See ADR 0014 Phase 2 for the full TestCase framework rationale.

---

5. Erlang Runtime Unit Tests

EUnit tests for the Erlang runtime modules.

Location: runtime/apps/beamtalk_runtime/test/

Test File	Tests
beamtalk_actor_tests.erl	Actor lifecycle, message dispatch, doesNotUnderstand
beamtalk_future_tests.erl	Future creation, resolution, rejection, await
beamtalk_hot_reload_tests.erl	Hot code reload, state migration
beamtalk_codegen_simulation_tests.erl	Codegen round-trip via EUnit simulation

Running:

cd runtime
rebar3 eunit --module=beamtalk_actor_tests

Example (beamtalk_future_tests.erl):

resolve_sets_value_test() ->
    {ok, Future} = beamtalk_future:new(),
    ok = beamtalk_future:resolve(Future, 42),
    ?assertEqual(42, beamtalk_future:await(Future)).

Test fixtures: test_counter.erl, test_throwing_actor.erl, etc.

---

6. Integration Tests

Test the interaction between the Rust compiler daemon and Erlang runtime.

Location: runtime/apps/beamtalk_runtime/test/beamtalk_repl_integration_tests.erl

Requires: Compiler daemon running (beamtalk daemon start)

What they test:

• TCP connection to daemon
• Expression compilation via daemon
• Hot code loading
• Error handling across Rust/Erlang boundary

Running:

# Terminal 1: Start daemon
./target/debug/beamtalk daemon start --foreground

# Terminal 2: Run tests
cd runtime
rebar3 eunit --module=beamtalk_repl_integration_tests

CI runs these with the daemon started in background mode.

---

7. Codegen Simulation Tests

Tests runtime behavior using real compiled Beamtalk code and simulated patterns.

Location: runtime/apps/beamtalk_runtime/test/beamtalk_codegen_simulation_tests.erl

What they test:

• spawn/0 and spawn/1 tests use real compiled counter.bt (unified E2E fixture - BT-239)
  • Validates actual #beamtalk_object{} record generation
  • Tests counter:spawn() from compiled module
• Other tests use simulated state for complex scenarios
• Method invocation (sync and async)
• State initialization and mutation
• Interaction between multiple actors

Test Fixtures: Compiled automatically by rebar3 pre-hook

• Source: tests/repl-protocol/fixtures/counter.bt (canonical implementation - BT-239)
• Compiled by: runtime/apps/beamtalk_runtime/test_fixtures/compile_fixtures.escript (runs automatically)
• Output: runtime/_build/*/test/bt@counter.beam
• No manual compilation needed - hook runs before every rebar3 eunit

Compilation Workflow:

Developer runs: cargo test OR rebar3 eunit
  └─> cargo build (if needed) - creates ./target/debug/beamtalk
  └─> rebar3 pre-hook runs: escript runtime/apps/beamtalk_runtime/test_fixtures/compile_fixtures.escript
      └─> Uses ./target/debug/beamtalk to compile tests/repl-protocol/fixtures/counter.bt
      └─> Copies bt@counter.beam to runtime/_build/*/test/
  └─> Tests run with compiled fixtures available

Note: For REPL TCP-protocol tests with full compilation pipeline, see tests/repl-protocol/.

Example:

spawn_zero_uses_default_state_test() ->
    %% Uses real compiled counter module
    Object = counter:spawn(),
    ?assertMatch({beamtalk_object, 'Counter', counter, _Pid}, Object),
    
    %% Extract pid from #beamtalk_object{} record
    Pid = element(4, Object),
    
    %% Verify default value
    {ok, Value} = gen_server:call(Pid, {getValue, []}),
    ?assertEqual(0, Value).

---

8. REPL Protocol Tests

REPL TCP-protocol integration tests that require a running REPL daemon. (Previously called "E2E tests" — renamed in BT-2085 because they exercise one specific surface, not "end-to-end across surfaces". Cross-surface parity tests live under tests/parity/.)

Location: tests/repl-protocol/

Test cases: tests/repl-protocol/cases/*.btscript (~23 files)

Test harness: crates/beamtalk-cli/tests/repl_protocol.rs

What they test:

• Workspace bindings (Transcript, Beamtalk globals)
• REPL commands (:load, variable persistence)
• Actor auto-await behavior
• ERROR: assertion patterns
• Integration between compiler daemon and runtime

When to use REPL-protocol tests: Only for tests that genuinely need the REPL daemon. Most language feature tests belong in stdlib/test/*.bt as BUnit tests (see section 4b).

Test file format:

// Test workspace bindings
Transcript show: 'Hello'
// => nil

// Variable persistence across expressions
x := 42
// => 42

x + 1
// => 43

Running:

# Run REPL protocol tests only
just test-repl-protocol

# Or via cargo directly
cargo test --test repl_protocol -- --ignored

# Run with verbose output
cargo test --test repl_protocol -- --ignored --nocapture

Adding a new REPL-protocol test case:

1. Create tests/repl-protocol/cases/my_feature.btscript
2. Add expressions with // => expected results
3. Run just test-repl-protocol

Note: Before adding here, consider whether the test needs the REPL. If it tests pure language features, add it to stdlib/test/*.bt as a BUnit test instead (see section 4b).

Error testing:

undefined_var
// => ERROR: Undefined variable

See tests/repl-protocol/README.md for full documentation.

9. Cross-surface Parity Tests (BT-2077)

Drives the same input through every public surface of the compiler stack (REPL, MCP, CLI, LSP) and asserts the observable behaviour is equivalent. Catches surface drift early — without this layer, a regression that affects only the MCP evaluate tool would slip through both the REPL E2E suite and the BUnit suite.

Location:

• Cases: tests/parity/cases/*.parity.bt
• Fixtures: tests/parity/fixtures/
• Harness crate: crates/beamtalk-parity-tests/

What they test:

• Literal evaluation (REPL + MCP) — same value
• Project loading (REPL + MCP + CLI) — same class set
• Lint (MCP + CLI) — same diagnostic count on a clean project
• BUnit test execution (REPL + MCP) — same pass/fail outcome
• Diagnostics on a broken file (CLI + MCP + LSP) — every surface flags it

Case file format:

// @input
3 + 4
// @surfaces repl, mcp
// @expect 7

Three expectation directives are supported:

Directive	Meaning
// @expect <text>	Every surface must produce this normalized value
// @expect-classes A, B, …	Every surface must observe at least these class names
// @expect-diagnostics N	Every surface must report (≥) N diagnostics; 0 means exactly zero

The harness recognises two placeholder tokens in the input:

• <project> — replaced with the staged temp copy of tests/parity/fixtures/simple_project/
• <bad_file> — replaced with the staged copy of tests/parity/fixtures/diagnostic/BadSyntax.bt

Workspace pool: all REPL and MCP cases share a single workspace started once per harness run. The pattern is borrowed from crates/beamtalk-mcp/src/client.rs::tests.

Running:

just test-parity

When to add a new case:

• A new operation appears on more than one surface
• A bug fix corrected a divergence between surfaces — add a regression case
• A surface gets a new tool/command that maps to an existing REPL op

10. LiveView IDE (Cockpit) Tests

The Phoenix LiveView IDE under editors/liveview is tested in four layers, gated by tags in test/test_helper.exs:

Layer	Tag	Needs	Example
Pure / unit	(none — bare mix test)	nothing	test/bt_attach/doc_format_test.exs
LiveView integration	(none)	the StubWorkspaceClient stub	test/bt_attach_web/workspace_doc_block_test.exs
Workspace integration	:workspace	a live workspace node + BT_WORKSPACE_COOKIE	test/bt_attach_web/workspace_live_test.exs
Browser e2e	:playwright	a workspace node and Playwright/Chromium (PHX_PLAYWRIGHT=1)	test/bt_attach_web/workspace_browser_test.exs

The bare mix test lane (the liveview CI job) runs the first two layers against the stub, so the full LiveView render path is covered without a node; the :workspace / :playwright lanes run in the e2e CI job.

Known coverage note — doc-comment rendering (BT-2558): the System Browser doc block is covered at two seams — beamtalk_repl_ops_browse_tests asserts browse-method-source carries doc/signature (from a beamtalk_object_class fixture that populates __doc__ directly), and workspace_doc_block_test.exs asserts the LiveView renders that payload as escaped HTML (against a stub). The compiler seam — that a /// comment in .bt source actually flows through to __doc__ / get_doc at runtime — is exercised by the help: tests (beamtalk_repl_docs / beamtalk_interface_test.bt), not through the browser. There is intentionally no browser e2e walking source → codegen → render (an earlier attempt was fragile against the mount-time class tree); if doc-comment rendering regresses, check those three suites in that order.

---

CI Pipeline

The CI workflow runs on every PR:

just ci
# Equivalent to:
#   just build           # Build Rust + Erlang
#   just lint            # Clippy + fmt-check + dialyzer
#   just test            # Rust unit tests + runtime EUnit
#   just test-stdlib     # Bootstrap expression tests (~14s)
#   just test-bunit      # BUnit TestCase tests (~85 files)
#   just test-repl-protocol  # REPL TCP-protocol tests (~50s)

Testing Pyramid

The test suite follows a proper testing pyramid after ADR 0014:

            ╱╲
           ╱  ╲        E2E Tests (~36 files)
          ╱    ╲       REPL/workspace integration — slow (~50s)
         ╱──────╲
        ╱        ╲     BUnit Tests (~85 files)
       ╱          ╲    Language feature tests — fast (`just test-bunit`)
      ╱────────────╲
     ╱              ╲  Stdlib Tests (~11 files)
    ╱                ╲ Bootstrap expression tests — fast (~14s)
   ╱──────────────────╲
  ╱                    ╲ Rust + Erlang Unit Tests (~600+ tests)
 ╱                      ╲ Parser, codegen, runtime modules — fast (~10s)
╱────────────────────────╲

Layer	Count	Speed	What it tests
Rust unit tests	~600 tests	~5s	Parser, AST, codegen
Erlang unit tests	~100 tests	~3s	Runtime, primitives, object system
Compiler snapshots	~51 cases	~2s	Codegen output stability
Stdlib tests	~11 files	~14s	Bootstrap primitives (expression tests)
BUnit tests	~85 files	—	Language features (TestCase classes)
E2E tests	~36 files	~50s	REPL/workspace integration

Cross-Repo Package Tests

The cross-repo workflow tests first-party packages that have been extracted to their own repositories (per ADR 0073). It builds the compiler from source, then checks out and tests each package against it.

Runs on: push to main + nightly schedule (7am UTC)

Current packages tested:

• beamtalk-http — HTTP client and server

Adding a new package: Add a new job to cross-repo.yml following the beamtalk-http job as a template. Each job checks out the compiler, builds it, installs the binary, then checks out and tests the package.

---

Test Organization Conventions

Naming

• Rust tests: test_descriptive_name or fn feature_behavior_context()
• EUnit tests: descriptive_name_test() (EUnit auto-discovers *_test functions)
• Snapshot test cases: snake_case directory names

File Structure

crates/beamtalk-core/src/
├── erlang.rs           # Code
└── erlang.rs           # Tests in same file (#[cfg(test)])

runtime/apps/beamtalk_runtime/test/
├── beamtalk_actor_tests.erl      # Tests for beamtalk_actor.erl
├── test_counter.erl              # Test fixture actor
└── ...

Serial Test Locks

Tests that manipulate process-global state (environment variables, current working directory, shared filesystem state) must use named serial_test locks to prevent conflicts while allowing parallelism between non-conflicting tests.

Lock naming guidelines:

Lock Name	Use When	Example
erlang_runtime	Running rebar3/erlc in runtime/ directory	Build/compile operations
e2e	Full E2E test with escript compilation	End-to-end language tests
env_var	Modifying environment variables	std::env::set_var, std::env::remove_var
cwd	Changing current working directory	std::env::set_current_dir
daemon_lockfile	Manipulating ~/.beamtalk/ directory	Daemon state management

Why named locks?

Previously, all serialized tests used #[serial_test::serial], which serialized all marked tests together. This reduced parallelism unnecessarily. For example, tests that manipulated environment variables would serialize with tests that changed the working directory, even though these operations don't conflict.

Named locks (e.g., #[serial(env_var)]) only serialize tests that actually conflict with each other. Tests in different groups can run in parallel, reducing CI time.

Example:

use serial_test::serial;

/// Uses `#[serial(env_var)]` because it modifies the `BEAMTALK_RUNTIME_DIR`
/// environment variable, which is process-global state.
#[test]
#[serial(env_var)]
fn test_env_modification() {
    unsafe { std::env::set_var("BEAMTALK_RUNTIME_DIR", "/tmp") };
    // ... test code ...
    unsafe { std::env::remove_var("BEAMTALK_RUNTIME_DIR") };
}

/// Uses `#[serial(cwd)]` because it changes the current working directory
/// (process-global state) using `std::env::set_current_dir`.
#[test]
#[serial(cwd)]
fn test_directory_change() {
    let original = std::env::current_dir().unwrap();
    std::env::set_current_dir("/tmp").unwrap();
    // ... test code ...
    std::env::set_current_dir(original).unwrap();
}

// These two tests can run in parallel because they use different locks

Guidelines for adding new serial tests:

1. Always use a named lock - Never use unnamed #[serial]
2. Document why - Add a doc comment explaining what global state is being manipulated
3. Choose the right lock - If manipulating a new type of global state, create a new lock name
4. Keep locks focused - Don't overload a lock name; create specific locks for specific conflicts

See BT-115 for the implementation details of the named lock system.

Test Fixtures

BUnit fixtures (Smalltalk image model): Place fixture .bt files in stdlib/test/fixtures/. These are automatically compiled and available to all BUnit test files — no @load directives needed. Just use the fixture class name directly in your test methods. This mirrors the Smalltalk approach where all classes exist in the running image.

stdlib/test/fixtures/
├── counter.bt            # Counter actor
├── typed_counter.bt      # Typed actor with Integer state
├── typed_account.bt      # Typed actor with Integer + String state
├── math_helper.bt        # Value type with recursion helpers
└── ...                   # ~46 fixture files

Erlang fixtures: test_*.erl in runtime/apps/beamtalk_runtime/test/ for reusable actors.

Compiler fixtures: test-package-compiler/cases/*/main.bt for compiler test inputs.

---

Adding New Tests

Adding a Rust Unit Test

Add to the existing #[cfg(test)] module in the source file:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn my_new_test() {
        // ...
    }
}

Adding a Compiler Snapshot Test

1. Create test-package-compiler/cases/my_feature/main.bt
2. Run cargo test -p test-package-compiler
3. Review with cargo insta review

Adding an Erlang Runtime Test

1. Add function to appropriate *_tests.erl file
2. Name it descriptive_name_test() (EUnit convention)
3. Run cd runtime && rebar3 eunit

Adding a Codegen Simulation Test

1. Add to runtime/apps/beamtalk_runtime/test/beamtalk_codegen_simulation_tests.erl
2. Manually construct state as compiler would generate
3. Run cd runtime && rebar3 eunit --module=beamtalk_codegen_simulation_tests

Adding a Stdlib Test (Bootstrap Primitives)

1. Create stdlib/bootstrap-test/my_feature.bt
2. Add expressions with // => expected_result annotations
3. Optionally use // @load path/to/fixture.bt for fixtures
4. Run just test-stdlib

Example test file:

// Test my new feature
myExpression
// => expected_result

// Wildcard (run but don't check value)
sideEffectExpression
// => _

Use this for: Bootstrap-critical primitives only (arithmetic, booleans, equality, strings, errors, exceptions). Most new tests should use BUnit instead.

Adding a BUnit Test (TestCase Classes)

1. Create stdlib/test/my_feature_test.bt with TestCase subclass: MyFeatureTest
2. If you need a helper class, add it to stdlib/test/fixtures/ — it will be auto-compiled
3. Add test methods prefixed with test (auto-discovered)
4. Optionally add setUp/tearDown for lifecycle
5. Run just test-bunit

Example test file:

// Fixture classes from stdlib/test/fixtures/ are automatically available
TestCase subclass: MyFeatureTest
  setUp =>
    self.thing := MyThing new

  testBasicBehavior =>
    self assert: (self.thing doSomething) equals: 42

  testErrorCase =>
    self should: [self.thing badMethod] raise: #does_not_understand

Use this for: Stateful tests with setup/teardown, complex scenarios with multiple assertions, actor interaction tests.

Adding a REPL Protocol Test (REPL/Workspace Integration)

1. Create or edit a .bt file in tests/repl-protocol/cases/
2. Add expressions with // => expected_result annotations
3. Run just test-repl-protocol

Use this for: Workspace bindings, REPL commands, variable persistence, auto-await, ERROR: patterns.

Example test file:

// Test workspace feature
Transcript show: 'hello'
// => nil

Cross-Platform Temp Paths

Never hardcode /tmp/ in tests. This breaks on Windows where /tmp does not exist.

Use File tempDirectory to get the OS temp directory, then build paths from it:

// In .btscript (e2e) or .bt (BUnit) tests:
tmp := File tempDirectory       // OS temp dir (e.g. /tmp or $TMPDIR on Unix, %TEMP% on Windows)
path := tmp ++ "/bt_my_test_file.txt"

// In Erlang tests:
TmpDir = beamtalk_file:'tempDirectory'(),
Path = <<TmpDir/binary, "/my_test_file.txt">>,

For BUnit tests (stdlib/test/*.bt), prefer relative paths under target/bt-test-tmp/ when possible — these don't need cross-platform handling. Use File tempDirectory only when an absolute path is required.

---

Debugging Test Failures

Rust tests

cargo test -- --nocapture              # Show println! output
cargo test test_name -- --nocapture    # Run specific test
RUST_BACKTRACE=1 cargo test            # Show backtraces

Snapshot differences

cargo insta review                     # Interactive diff viewer

Erlang tests

cd runtime
rebar3 eunit --module=module_name      # Run single module
rebar3 shell                           # Interactive debugging

Integration test failures

# Check daemon is running
ps aux | grep beamtalk

# Check socket exists  
ls -la ~/.beamtalk/daemon.sock

# Run daemon in foreground to see output
./target/debug/beamtalk daemon start --foreground

---

Fuzzing (Parser Crash Safety)

Fuzzing tests the parser's robustness by feeding it random or mutated input to detect crashes, infinite loops, and excessive memory use.

Technology: cargo-fuzz (libFuzzer)

Location: fuzz/fuzz_targets/parse_arbitrary.rs

Corpus: fuzz/corpus/parse_arbitrary/ (32 seed files from examples/ and tests/repl-protocol/cases/)

Running Locally

# Fuzz for 60 seconds (default)
just fuzz

# Fuzz for a specific duration
just fuzz 300  # 5 minutes

# Or use cargo directly
cargo +nightly fuzz run parse_arbitrary -- -max_total_time=60

Requirements:

• Rust nightly toolchain
• cargo-fuzz: cargo install cargo-fuzz

What Fuzzing Tests

Test Type	What It Catches
Crash safety	Panics on unexpected token sequences
Infinite loops	Hangs during error recovery
Stack overflow	Deeply nested expressions causing stack exhaustion
Out-of-memory	Excessive memory allocation on malformed input
Index bounds	Array/buffer out-of-bounds access

CI Integration

Fuzzing runs nightly (not per-PR) via .github/workflows/fuzz.yml:

• Duration: 10 minutes per run
• Memory limit: 4GB RSS
• Artifacts uploaded on failure
• Auto-creates GitHub issues for crashes

Why nightly? Fuzzing is too slow for per-PR CI (minutes to hours). Nightly runs catch regressions without blocking development.

Interpreting Results

Success: No artifacts produced, fuzzer completes normally

Done 17654 runs in 60 second(s)

Crash (CRITICAL): crash-* artifacts indicate parser panic

artifact_prefix='fuzz/artifacts/parse_arbitrary/'; Test unit written to crash-abc123

Action: Fix immediately, parser must never panic on user input.

Timeout (WARNING): timeout-* artifacts indicate infinite loop

SUMMARY: libFuzzer: timeout

Action: Investigate error recovery logic, add timeout limits.

OOM (INFO): oom-* artifacts indicate excessive memory use

SUMMARY: libFuzzer: out-of-memory

Action: Expected for extremely malformed input. Consider resource limits if frequent.

Reproducing Failures

# Reproduce exact crash
cargo +nightly fuzz run parse_arbitrary fuzz/artifacts/parse_arbitrary/crash-abc123

# Minimize test case to smallest reproducer
cargo +nightly fuzz tmin parse_arbitrary fuzz/artifacts/parse_arbitrary/crash-abc123

Adding Corpus Files

The corpus seeds fuzzing with realistic starting points. To add new files:

# Copy new .bt file to corpus
cp my_new_test.bt fuzz/corpus/parse_arbitrary/033_my_new_test.bt

# Fuzzer will use it as seed for mutation
just fuzz

Keep corpus in sync: When adding new .bt files to examples/ or tests/repl-protocol/cases/, also copy them to fuzz/corpus/parse_arbitrary/ so the fuzzer can use them as mutation seeds.

Corpus minimization: cargo +nightly fuzz cmin rewrites the corpus directory in-place. Run it on a temporary copy to avoid deleting tracked seed files:

# Safe minimization (don't run cmin directly on tracked corpus)
cp -r fuzz/corpus/parse_arbitrary /tmp/corpus-backup
cargo +nightly fuzz cmin parse_arbitrary
# Review changes, restore any deleted seeds if needed

Troubleshooting

"cargo-fuzz not found":

cargo install cargo-fuzz

"nightly toolchain required":

rustup toolchain install nightly

"workspace errors": Ensure fuzz is in workspace.exclude in root Cargo.toml.

Fuzzer runs too long: Use shorter duration for quick checks:

just fuzz 5  # 5 seconds

References

• Rust Fuzz Book - cargo-fuzz guide
• ADR 0011 - Robustness testing strategy
• Epic: BT-362

---

Property Testing (Nightly Extended)

Property tests use proptest to verify parser invariants over thousands of randomly generated inputs. Standard CI runs 512 cases per property (~0.4s). The nightly run extends this to 10,000 cases to catch rare edge cases.

Location: crates/beamtalk-core/src/source_analysis/parser/property_tests.rs

Properties tested:

Property	What It Verifies
parser_never_panics	Arbitrary UTF-8 input never causes a panic
parser_never_panics_near_valid	Near-valid Beamtalk fragments don't panic
diagnostic_spans_within_input	All diagnostic spans have end <= input.len()
error_nodes_produce_diagnostics	Every Expression::Error node has diagnostics
error_messages_are_user_facing	No internal type names leak into error messages

Running Locally

# Run with default 512 cases (fast, ~0.4s)
cargo test -p beamtalk-core property_tests

# Run with extended cases (matches nightly)
PROPTEST_CASES=10000 cargo test -p beamtalk-core property_tests

CI Integration

The extended proptest runs nightly alongside cargo-fuzz in the GitHub Actions workflow .github/workflows/fuzz.yml:

• Cases per property: 10,000 (vs 512 in standard CI)
• Schedule: 2 AM UTC daily (same as fuzzing)
• Can be triggered manually via workflow_dispatch
• Proptest automatically shrinks failures to minimal reproducing cases

Why nightly? 10,000 cases × 5 properties takes longer than is appropriate for per-PR CI. Nightly runs provide deeper exploration without slowing development.

Interpreting Results

Success: All 5 properties pass with 10,000 cases each.

Failure: Proptest finds a failing input and shrinks it to the smallest reproducer. The shrunk case and a seed are printed in the test output. Example:

proptest: Seed for failing test: 0x1234abcd...
proptest: Shrink failed: parser panicked on input "\x00\xff"

To reproduce a specific failure, use the seed from the output with PROPTEST_REPLAY:

PROPTEST_REPLAY="0x1234abcd..." cargo test -p beamtalk-core property_tests -- parser_never_panics

Proptest also persists failures in proptest-regressions/ files, so they are automatically replayed on subsequent test runs.

Standard CI vs Nightly

	Standard CI	Nightly
Cases per property	512	10,000
Duration	~0.4s	~10s
Runs on	Every PR	Daily at 2 AM UTC
Trigger	Automatic	Schedule + manual
Configured via	ProptestConfig in source	PROPTEST_CASES env var

References

• proptest documentation - Property testing framework
• ADR 0011 - Robustness testing strategy
• Epic: BT-362

---

Performance Testing (Future)

From AGENTS.md, targets for tooling responsiveness:

Operation	Target
Keystroke to diagnostics	<50ms
Single-file incremental	<50ms
Full file diagnostics	<100ms
Project-wide find references	<500ms

Performance regression tests are planned but not yet implemented.

---

Surface Parity

When adding or modifying operations across surfaces (CLI, REPL, MCP, LSP), consult the Surface Parity Map to ensure consistent coverage. Any operation not labelled surface-specific must produce equivalent output across all surfaces where it appears.

---

References

• Surface Parity Map - Cross-surface operation coverage matrix
• ADR 0014: Beamtalk Test Framework - Architecture decision for the three-layer test strategy
• test-package-compiler/README.md - Snapshot test details
• tests/repl-protocol/README.md - REPL protocol test framework details
• runtime/README.md - Erlang runtime test details
• AGENTS.md - Development guidelines
• insta documentation - Snapshot testing framework