Testing Guide

May 28, 2026 · View on GitHub

PerlOnJava provides a two-level testing strategy to balance development speed with comprehensive validation.

Quick Start

Fast Unit Tests (Recommended for Development)

make test-unit

Runs only the fast unit tests from src/test/resources/unit/. These tests:

✅ Run in seconds (not minutes)
✅ Cover core functionality
✅ Use parallel execution (8 jobs)
✅ Provide immediate feedback during development
✅ Output TAP format with detailed statistics

Comprehensive Test Suite

make test-all

Runs all tests including comprehensive module tests. These tests:

🔍 Include Benchmark.pm and other Perl core modules
⏱️ Take longer to complete (minutes)
📊 Generate detailed JSON report (test_results.json)
🎯 Identify high-priority opportunities (incomplete tests)
📈 Provide feature impact analysis

Testing Approaches

1. Perl-Style Testing (Default)

Uses dev/tools/perl_test_runner.pl - a prove-like test harness:

# Fast unit tests
make test-unit

# All tests with JSON report
make test-all

# Custom test run
perl dev/tools/perl_test_runner.pl --jobs 4 --timeout 20 src/test/resources/unit

Features:

TAP (Test Anything Protocol) output
Parallel test execution
Timeout protection
Feature impact analysis
Incomplete test detection
JSON reporting

Options:

--jobs|-j NUM      Number of parallel jobs (default: 5)
--timeout SEC      Timeout per test in seconds
--output FILE      Save detailed results to JSON file
--jperl PATH       Path to jperl executable (default: ./jperl)

2. Using jprove (Standard Perl prove)

PerlOnJava includes jprove (Unix) and jprove.bat (Windows), wrappers that run the standard Perl prove test harness with jperl:

# Run tests in a directory
./jprove src/test/resources/unit

# Run with verbose output
./jprove -v t/*.t

# Run specific test files
./jprove t/basic.t t/advanced.t

# Run recursively
./jprove -r t/

# Run with parallel jobs
./jprove -j4 t/

Common Options:

-v, --verbose      Print all test lines
-l, --lib          Add 'lib' to @INC
-r, --recurse      Recursively descend into directories
-j, --jobs N       Run N test jobs in parallel
-q, --quiet        Suppress some test output
--timer            Print elapsed time after each test
--color            Colored test output (default)
--nocolor          Disable colored output

Example Output:

./jprove src/test/resources/unit/array.t
src/test/resources/unit/array.t .. ok
All tests successful.
Files=1, Tests=15, 1 wallclock secs
Result: PASS

jprove is useful when you want standard Perl prove behavior and options, while perl_test_runner.pl provides additional features like JSON reporting and feature impact analysis.

3. JUnit/Gradle Testing (For CI/CD)

Uses JUnit 5 with tags for test filtering:

# Fast unit tests via Gradle
make test-gradle-unit

# All tests via Gradle
make test-gradle-all

# Direct Gradle commands
./gradlew testUnit    # Only unit tests
./gradlew testAll     # All tests
./gradlew test        # Default test task

Use Cases:

CI/CD pipeline integration
IDE integration (IntelliJ, VSCode)
JUnit test reports
Maven-style testing

Test Organization

src/test/resources/
├── unit/              # Fast unit tests (seconds)
│   ├── array.t
│   ├── hash.t
│   ├── regex/
│   └── ...

Test Categories

(Work in progress)

Category	Location	Speed	Purpose
Unit Tests	`unit/`	Fast (seconds)	Core functionality, operators, syntax
Module Tests	`Benchmark/`, `lib/`, etc.	Slow (minutes)	Perl core modules, CPAN compatibility
Integration Tests	`dist/`, `ext/`	Varies	Package integration, extensions

Development Workflow

During Development (Fast Feedback Loop)

# 1. Make changes
vim src/main/java/org/perlonjava/...

# 2. Build
make dev

# 3. Run fast tests
make test-unit

Before Committing (Comprehensive Validation)

# Run full test suite
make test-all

# Review test_results.json for any regressions

Tracking Test Progress Over Time

For long-running development work, track test results over time to monitor progress and catch regressions:

1. Run Tests with Timestamped Logs

# Clean up and run comprehensive tests
killall java
rm -rf perl5_t/t/tmp_*

# Run with extended timeout and save to dated log
perl dev/tools/perl_test_runner.pl \
  --timeout 300 \
  --output out.json \
  perl5_t/t \
  2>&1 > logs/test_$(date +%Y%m%d_%H%M%S).log

Workflow details:

killall java - Stop any hung Java processes from previous runs
rm -rf perl5_t/t/tmp_* - Clean temporary test files
--timeout 300 - Allow 5 minutes per test (for slower tests)
logs/test_YYYYMMDD_HHMMSS.log - Timestamped log for tracking history

2. Compare Test Runs

Compare two test runs to see what changed:

perl dev/tools/compare_test_logs.pl \
  logs/test_20260206_090000.log \
  logs/test_20260206_102400.log

Output shows:

Tests that started passing
Tests that started failing
Changes in test counts
New tests added or removed
Summary of improvements or regressions

Use cases:

Before/after implementing a feature
Daily progress tracking on a branch
Identifying when a regression was introduced
Measuring impact of optimizations

3. Log Organization

Suggested logs/ directory structure:

logs/
├── test_20260206_090000.log  # Baseline
├── test_20260206_102400.log  # After Feature A
├── test_20260206_153000.log  # After Bug Fix
└── test_20260207_101500.log  # Latest

Keep baseline logs for major milestones to track long-term progress.

CI/CD Pipeline

# Build and run all tests
make build
make test-gradle-all

Test Output

TAP Output (perl_test_runner.pl)

Finding test files in src/test/resources/unit...
Found 142 test files
Running tests with ./jperl (8 parallel jobs, 10s timeout)
------------------------------------------------------------
[  1/142] unit/array.t                     ... ✓ 15/15 ok (0.23s)
[  2/142] unit/hash.t                      ... ✓ 12/12 ok (0.18s)
[  3/142] unit/regex/basic.t              ... ✓ 25/25 ok (0.31s)
...

TEST SUMMARY:
  Total files: 142
  Passed:      140
  Failed:      2
  Errors:      0
  Timeouts:    0
  Incomplete:  0

  Total tests: 3,456
  OK:          3,421
  Not OK:      35
  Pass rate:   99.0%

JUnit Output (Gradle)

> Task :testUnit

PerlScriptExecutionTest > Unit test: unit/array.t PASSED
PerlScriptExecutionTest > Unit test: unit/hash.t PASSED
...

BUILD SUCCESSFUL in 12s
142 tests completed, 140 succeeded, 2 failed

Advanced Usage

Running Specific Tests

# Single test file
perl dev/tools/perl_test_runner.pl src/test/resources/unit/array.t

# Specific directory
perl dev/tools/perl_test_runner.pl src/test/resources/unit/regex

# With custom settings
perl dev/tools/perl_test_runner.pl --jobs 16 --timeout 60 --output mytest.json t/

Analyzing Test Results

After running make test-all, examine test_results.json:

# View summary
jq '.summary' test_results.json

# Find failing tests
jq '.results | to_entries | map(select(.value.status == "fail")) | .[].key' test_results.json

# Feature impact
jq '.feature_impact' test_results.json

Debugging Failures

# Run single test with full output
./jperl src/test/resources/unit/array.t

# Run with verbose output
./jperl -d src/test/resources/unit/array.t

# Check for syntax errors
./jperl -c src/test/resources/unit/array.t

Best Practices

Run test-unit frequently during development for fast feedback
Run test-all before commits to catch regressions
Use parallel jobs (--jobs 8) to speed up test runs
Set appropriate timeouts - short for unit tests (10s), longer for module tests (30s)
Review incomplete tests - they often indicate bugs that block many tests
Save JSON reports for trend analysis and debugging

Performance Tips

Unit tests: Should complete in < 5 minutes total
All tests: May take 10-30 minutes depending on system
Parallel jobs: Adjust --jobs based on CPU cores (typically 1-2x core count)
Timeouts: Increase for slow systems, decrease for fast feedback

Integration with IDEs

IntelliJ IDEA / VSCode

Open project
Right-click on PerlScriptExecutionTest.java
Select "Run tests"
Or run specific tags: @Tag("unit") or @Tag("full")

Command Line with Gradle

# Run specific test class
./gradlew test --tests PerlScriptExecutionTest

# Run with tags
./gradlew testUnit    # @Tag("unit")
./gradlew testAll     # @Tag("full")

Troubleshooting

Tests timing out

Increase timeout: --timeout 60

Too slow

Reduce parallel jobs on low-memory systems: --jobs 2

Out of memory

Reduce parallel jobs
Increase JVM heap: export JAVA_OPTS="-Xmx4g"

Test hangs

Check for infinite loops, use timeout command:

timeout 30s ./jperl problematic_test.t

Importing Perl5 Test Suite

PerlOnJava can import and run tests from the official Perl5 repository to verify compatibility and behavior.

Setup

To import Perl test files and verify their behavior under PerlOnJava:

Clone the Perl5 repository (if not already done):

rm -rf perl5  # if it exists
git clone https://github.com/Perl/perl5.git

Run the import script to copy tests and apply patches:
```
perl dev/import-perl5/sync.pl
```

This script reads dev/import-perl5/config.yaml and copies configured files from the perl5/ directory (Perl 5 source repository) to perl5_t/ at the project root, creating:

Core tests in perl5_t/t/
Module tests in perl5_t/[Module]/
Test infrastructure (TestInit.pm, MANIFEST)
Supporting files (Porting/ directory)

The script also applies patches from dev/import-perl5/patches/ for PerlOnJava compatibility.

For how this differs from CPAN install-time patches under jcpan, see CPAN Distroprefs for PerlOnJava and dev/design/patch-and-cpan-prefs-layout.md.

Running Imported Tests

To run the imported Perl5 tests:

perl dev/tools/perl_test_runner.pl --output out.json perl5_t/t

See dev/import-perl5/README.md for more details on:

The import system architecture
How to add patches for PerlOnJava compatibility
Managing test expectations