.. vim: ft=rst sts=2 sw=2 tw=70 .. default-role:: literal

====================================================================== rnt

---

         Automated testing of command line interfaces

---

:Author: Roman Neuhauser :Contact: rn+rnt@sigpipe.cz :Copyright: This document is in the public domain.

.. This file is marked up using reStructuredText. Lines beginning with ".." are reST directives. "foo_" or "foo bar_" is a link, defined at ".. _foo" or ".. _foo bar". "::" introduces a literal block (usually some form of code). "foo" is some kind of identifier. Suspicious backslashes in the text ("std::string\s") are required for reST to recognize the preceding character as syntax.

Overview

rnt_ is a tool for automated testing of command line interfaces. It comprises two shell scripts portable enough to run at least on FreeBSD, GNU/Linux and MSYS (Windows).

rnt_ has notions of test suite and test case.

.. _rnt: https://github.com/roman-neuhauser/rnt

Test Suite

---

A test suite is a directory containing zero or more test cases. A test suite may contain other entries beside test cases, they will be ignored.

Test Case

---

A test case is an immediate test suite subdirectory with a name matching ^[0-9]{3,}(-[a-z0-9]+)+$. It should contain these files:

README This file is output at the terminal if the test fails. cmd A script with the command(s) under test. exit.expected Expected exit code. out.expected Expected contents of standard output. err.expected Expected contents of standard output.

cmd is required, the remaining files are optional (with a caveat about exit.expected, see below). If any of the *.expected files is missing, it's treated as if it was present and empty. A test case may contain other files matching *.expected, these are treated the same as the usual *.expected files listed above (see below). Any number of other files and directories may be present in the test case directory.

Behavior

---

rnt runs each test case in the given test suite, printing . (a dot) for a successful case, F for a failed one. After all test cases have been run, rnt prints details of failures if any: for each failed test case, its README file is output if it exists, followed by the diffs.

For each test case, rnt chdirs into the test case directory and runs cmd, storing stdout, stderr and the exit code in out.actual, err.actual and exit.actual, respectively.

Then, rnt gathers all *.actual and *.expected files in the test case directory. The list of files considered is not limited to the out.*, err.* and exit.* files listed above: all files with either of those two extensions are included. It's not necessary for both foo.expected and foo.actual to be present; if either side is missing, it's treated as present and empty.

Each pair of foo.expected, foo.actual gets diffed; if any pair has non-empty diff, the test is considered to have failed. Note that since the actual exit code is always a number (never the empty string), a test case without exit.expected will always fail.

*.actual and *.diff files of successful tests are removed, while failed test cases are left as-is for examination.

Installation

Git Subtree

---

Use git subtree in a project using Git::

git remote add rnt https://github.com/roman-neuhauser/rnt.git git subtree add -P rnt --squash rnt master

Git Submodule

---

This method has several disadvantages: submodule trees do not show up in git-archive(1) output, and working copies of such a repository need to be created with git clone --recursive, otherwise git submodule update --init is necessary to clone and check out the submodule. ::

git submodule add https://github.com/roman-neuhauser/rnt.git rnt git submodule update --init

Plain Old Copy

---

Using another versioning control system? ::

mkdir rnt curl -s https://github.com/roman-neuhauser/rnt/tarball/master
| tar -xzf - -C rnt --strip-components=1 svn add rnt svn ci

Integration

Modify your Makefile to include a check target of the following structure:

::

check: all SHELL=$(SHELL)$(SHELL) /run-tests.sh [args...]

• <rnt> is the path to rnt.
• <suite> is the path to your tests.
• [args...] is any number of arguments (possibly none), these are passed to each test.

Alternatives

rnt is quite different from a number of other shell-oriented testing tools like

• Bats_
• cram_
• roundup_
• sharness_

.. _Bats: https://github.com/sstephenson/bats .. _cram: https://bitheap.org/cram .. _roundup: https://bmizerany.github.io/roundup .. _sharness: https://github.com/mlafeldt/sharness