Testing

This directory contains both regression/feature testing and performance testing.

Regression Testing

Each test generates a single frame and writes it out to a png image. The glContext is initialized before and freed after each test to make sure each has a clean slate. The tests are executed using a series of programs compiled with different options:

• ./run_tests
• `./run_tests_d16
• `./run_tests_d16_no_stencil
• `./run_tests_rgb565
• `./run_tests_clamp_border
• `./run_tests_no_depth_no_stencil

Each one of the non-default configurations runs all the tests possible with that configuration. Each test compares the output image against the expected output and reports any failures. All expected outputs are the same except for the rgb565 tests (and any future 16-bit color buffer configs) which generate 2-channel output (grayscale + alpha) which obviously is not the correct visual but is good enough to memcmp against.

Here is what running the default tests looks like:

$ make run_tests
# build output
$ ./run_tests
All tests passed

In addition to running all the tests, you can pass 1 or more arguments to select specific tests. It will skip any tests it can't find:

$ ./run_tests blend_test test_edges stencal_test map_vbuffer
Could not find 1/4 tests
All 3 tests run passed

Each test suite returns 0 if there were no failures (or skips), 1 otherwise.

Finally if you want to be exhaustive, you can run the run_all_tests.sh which will run all test_suites and return the appropriate value:

$ ./run_all_tests.sh
All 105 tests run passed
All 105 tests run passed
All 104 tests run passed
All 105 tests run passed
All 107 tests run passed
All 99 tests run passed
$ echo $?
0

If you add a new function or feature, you can add a test following the structure shown in the existing tests.

You can look in testing/test_output to see the png's generated by run_tests which are compared with those in testing/expected_output. For each test that fails, two files will be generated: testname_diff.png and testname_diff.txt. The first is a black image with any pixels that didn't match the expected output being white. The second lists the exact pixel values and their expected values in the format:

Diff from (195, 67) to (195, 67):
(252 253 248 255)
(251 253 248 255)
Diff from (330, 67) to (331, 67):
(253 254 249 255) (252 253 249 255)
(253 253 249 255) (253 253 249 255)

This format is subject to change but for now it lists runs of consecutive mismatching pixels with the produced output on the first line and the expected output on the second line. The coordinates are image coordinates, from the top left, not the bottom left like when PGL actually generated the image.

Remember that for the 16-bit color buffer tests the 2, 8-bit values will not match up with the rgb channels.

Performance Testing

The current tests are just basics. If you make any significant changes to core pieces that might affect performance, try to make sure the results are the same or better afterward.

Any of the standalone demos could be used as performance test too and if you know one of them specifically stress the area of your changes, definitely take advantage of that. I may end up converting some to formal tests too (especially gears since it's so standard).

The actual tests do run in a visual window so you can see what they're actually doing (and I thought it was a more realistic test than rendering offscreen). Here's the process and what the output looks like on my laptop:

$ make config=release perf_tests
# build output
$ ./perf_tests
points_perf: 471.642 FPS
freeing buffer 1
pointsize_perf: 648.193 FPS
freeing buffer 1
lines_perf: 206.676 FPS
freeing buffer 1
triangles_perf: 33.036 FPS
freeing buffer 1
tri_interp_perf: 42.029 FPS
freeing buffer 1

Keep in mind that's with OpenMP enabled which does boost the last two tests by 40-60% in my experience though PortableGL works fine without compiling with OpenMP as well and it's only even used for filling triangles, so the best case scenario is a scene with just a few large triangles.