4.3. Testing Tools#

These tools are used to run the continuous integration tests. We have unittests and script tests.

4.3.1. b2test-units: Run unittests#

This program runs all unittests of the Belle2 Software. It is based on Google Test. Usually it’s enough to call it without arguments. Here we just document the most useful arguments, please use --help to get a more detailed list of available options.

Optional Arguments

--help

Show a detailed list of all options

--gtest_list_tests

List all tests instead of running them

--gtest_filter=PATTERN

Run only the tests whose name matches the pattern

--gtest_output_xml=FILENAME

Write the test results into a JUnit compatible XML file

4.3.2. b2test-scripts: Run framework tests#

Run basf2 framework tests of the given packages. If no argument is given all tests are run.

For each package execute all files ending in *.py inside the tests/ directory as test cases. A return value of 0 is treaded as success, other return values are treated as failures. In addition if *.out or *.err files with the same base name are present in the tests directory they will be treated as expected output on stdout or stderr respectively. If the output of the script differes from the contents of these files the test is marked as failed as well. The first line from stdout will not be compared to allow executing the scripts from a different directory.

Note

Tests can decide that they need to be skipped under certain conditions. To do so a test script can write a line beginning with TEST SKIPPED: followed by the reason for skipping on stderr and exit with a non-zero return code.

Note

On Unix/Linux systems, exit codes are defined to be 8-bit unsigned value. Values outside the 0-255 range are truncated by taking modulo 256. If the passed code happens to be a multiple of 256, the test will be a wrongly tagged as a SUCCESS. Care should be taken when directly passing return codes from other processes.

See also

b2test_utils.skip_test

usage: b2test-scripts [-h] [-j N] [--xml XML] [-n DIFF_LINES] [-q]
                      [--filter FILTER] [--exclude EXCLUDE] [-t SECONDS]
                      [DIRECTORY [DIRECTORY ...]]

Required Arguments

DIRECTORY

run all tests included in this directory.If none are specified run all tests

Optional Arguments

-j

Number of test cases to run in parallel

Default: 1

--xml

Save results into given file using xml format compatible with JUnit parsers

-n, --diff-lines

Number of lines to show if output is different from template

Default: 10

-q, --quiet

if given don’t print output of failed tests

--filter

Only run tests where the name of the steering file matches the given regular expression

--exclude

Exclude all tests where the name of the steering file matches the given regular expression

-t, --timeout

if given this is the maximum runtime in seconds for each test case before they are killed and marked as failed

Examples

  • run all tests with 8 tests being run in parallel:

    $ b2test-scripts -j8

  • run the framework tests and save the output as XML:

    $ b2test-scripts --xml=framework-tests.xml framework

  • run only tests which where the python filename contains seqroot:

    $ b2test-scripts --filter=seqroot