Tools for Physics Validation of the Software

25. Tools for Physics Validation of the Software#

The validation package is used to validate the physics performance of the software. As the software is constantly developed further, algorithms are changed and we want to make sure that they don’t perform worse than before (e.g. lower reconstruction efficiencies or slower runtime) and that they work as intended.

To that end, every software package has a validation subdirectory that contains a number of simple steering files. The validation package collects these steering files and runs them. The resulting output *.root files can then be compared across software versions (“revisions”) to spot performance degradations and other problems.

To provide a simple user interface, the validation package contains a small webserver, that can also be run locally to show the comparison plots.

The validation framework is also run centrally on a nightly basis. The results are shown at https://validation.belle2.org/.

25.1. Command line utilities#

25.1.1. b2validation: Run Physics Validation#

This program is used to run the full physics validation suite

usage: b2validation [-h] [-d] [-m {local}] [-i INTERVALS]
                    [-o OPTIONS [OPTIONS ...]] [-p PARALLEL]
                    [-pkg PACKAGES [PACKAGES ...]] [-s SELECT [SELECT ...]]
                    [-si SELECT_IGNORE_DEPENDENCIES [SELECT_IGNORE_DEPENDENCIES ...]]
                    [--send-mails]
                    [--send-mails-mode {full,incremental,automatic}] [-q]
                    [-t TAG] [--test] [--use-cache] [--view]
                    [--max-run-time MAX_RUN_TIME]

options

-d, --dry

Perform a dry run, i.e. run the validation module without actually executing the steering files (for debugging purposes).

-m, --mode

Possible choices: local

The mode which will be used for running the validation. Possible values: local. Default is ‘local’

-i, --intervals

Comma separated list of intervals for which to execute the validation scripts. Default is ‘nightly’

-o, --options

One or more strings that will be passed to basf2 as arguments. Example: ‘-n 100’. Quotes are necessary!

-p, --parallel

The maximum number of parallel processes to run the validation. Only used for local execution. Default is number of CPU cores.

-pkg, --packages

The name(s) of one or multiple packages. Validation will be run only on these packages! E.g. -pkg analysis arich

-s, --select

The file name(s) of one or more space separated validation scripts that should be executed exclusively. All dependent scripts will also be executed. E.g. -s ECL2D.C

-si, --select-ignore-dependencies

The file name of one or more space separated validation scripts that should be executed exclusively. This will ignore all dependencies. This is useful if you modified a script that produces plots based on the output of its dependencies.

--send-mails

Send email to the contact persons who have failed comparison plots. Mail is sent from b2soft@mail.desy.de via /usr/sbin/sendmail.

--send-mails-mode

Possible choices: full, incremental, automatic

How to send mails: Full report, incremental report (new/changed warnings/failures only) or automatic (default; follow hard coded rule, e.g. full reports every Monday).

-q, --quiet

Suppress the progress bar

-t, --tag

The name that will be used for the current revision in the results folder. Default is ‘current’.

--test

Execute validation in testing mode where only the validation scripts contained in the validation package are executed. During regular validation, these scripts are ignored.

--use-cache

If validation scripts are marked as cacheable and their output files already exist, don’t execute these scripts again

--view

Once the validation is finished, start the local web server and display the validation results in the system’s default browser.

--max-run-time

By default, running scripts (that is, steering files executed bythe validation framework) are terminated after a certain time. Use this flag to change this setting by supplying the maximal run time in minutes. Value <=0 disables the run time upper limit entirely.

25.1.2. b2validation-plots: Compare and plot a selection of revisions#

Select several different software versions (revisions) for which output files were generated using b2validation and generate comparison plots:

usage: b2validation-plots [-h] [-r [REVISIONS ...]] [-f] [--popular]

options

-r, --revisions

Takes a list of revisions (separated by spaces) and generates the plots for them

-f, --force

Regenerates plots even if the requested combination exists already

--popular

Creates several sets of plots for popular combinations of revisions.

25.1.3. b2validation-server: Run Webserver Displaying the Physics Validation Results#

With b2validation-server you can run a local webserver to display the results of a physics validation performed with b2validation.

usage: b2validation-server [-h] [-ip IP] [-p PORT] [-v] [--production]
                           [-u USERMAP]

options

-ip, --ip

The IP address on which theserver starts. Default is ‘127.0.0.1’.

-p, --port

The port number on which the server starts. Default is ‘8000’.

-v, --view

Open validation website in the system’s default browser.

--production

Run in production environment: no log/error output via website and no auto-reload

-u, --usermap

Path of file containing <email gitlab_username> map.

25.1.4. b2validation-bisect: Find commits which introduced validation degradation#

Utility script to be used in conjunction with the git bisect functionality [1] to search for degradations in validation variables. To use it, you can set the last known good git commit and the first known bad commit and start the bisecting process. Git will then checkout commits between the good and the bad and perform a binary search to arrive at the first bad commit.

To give feed back to git, which is first bad commit, this script either returns the exit code 0 for good commit, code 125 to ignore the current commit or code 1 for a bad commit.

Information contained within .root files produced by the official validation scripts or your custom scripts can be used to distinguish between good and bad commits.

The syntax of the --check-quantity option is the following: <root filename>:<quantity>:<compare op>:<value>

  • quantity is the name of a histogram in the given root file. For histograms, the quantity to check for can be given in square brackets, for regular histograms, this is one of this strings:

    • mean_x = mean of the histogram

    • entries = number of entries in all bins

    • mean_y = the mean of the bin content

    • mean_y_zero_suppressed = the mean of all the bin contents > 0

  • compare op can be > or <

  • value is the reference value as a float to compare to

As an example, to identify the commit which made the tracking efficiency drop below 0.93, the following snippet can be used:

git bisect good <last good release>
git bisect bad <first bad release>
git bisect run bisect_validation.py --script 13_trackingEfficiency_createPlots.py
    --check-quantity "results/current/tracking/TrackingValidation.root:hEfficiency[mean_y]:>:0.93"

usage: b2validation-bisect [-h] [--skip-compile] [--report-compile-fail]
                           [--report-execution-fail]
                           [--check-quantity CHECK_QUANTITY]
                           [--check-failed-comparison]
                           [--max-failed-comparison MAX_FAILED_COMPARISON]
                           [--test-check] [--execute EXECUTE]
                           [--script SCRIPT] [--keep]

options

--skip-compile

Do not trigger a compile, useful for saving time when setting up the quantity checks.

--report-compile-fail

Report a bad revision to git if the code base did not compile. By default, failing compiles are ignored.

--report-execution-fail

Report a bad revision to git if the provided script has an error code != 0. By default, this is reported to git.

--check-quantity

Check for a quantity in validation files

--check-failed-comparison

Check if one comparison to a reference plot failed

--max-failed-comparison

Maximum number of comparisons allowed to fail

--test-check

Just check for the quantity, but do not run any git or compile commands

--execute

File to execute after the compile and before the quantity check

--script

Name of validation script to run after the compile and before the quantity check. Only this validation script and all scripts it depends on are executed.

--keep

Keep the result folder of each bisect step for later examination

25.1.5. b2validation-revisions: Extract quantities from each revision#

Tool to do for each git commit in a specific range: - check out the commit (configurable if you want to run git stash and git stash pop before and afterwards) - run scons or not (configurable) - run a script (given on command line) - extract the meanX, the meanY and the zeroSupressedMeanY from variables from ROOT files (also configured on the commandline) - write all variables together with their git hash and git log message into a pandas DataFrame and this dataframe into the results.pkl file as a pickle stream.

For example if you want to examine some commits related to tracking, you can use:

b2validation-revisions 5e066d31b391bb240cb2b381e67c5488681a8ec6 05d5bf078a45f6af8022bf06b398213d0bc2409d       --check-quantity ~/basf2/tracking/validation/results/current/TrackingValidation.root:hEfficiency       --script "validate_basf2 -s *_trackingEfficiency_*.py"

which will run through all commits from 5e066d31b391bb240cb2b381e67c5488681a8ec6 to 05d5bf078a45f6af8022bf06b398213d0bc2409d and execute validate_basf2 -s *_trackingEfficiency_*.py for each of them. Then it will extract the meanX, the meanY and the zeroSupressedMeanY of the TNtuple hEfficiency in the file ~/basf2/tracking/validation/results/current/tracking/TrackingValidation.root.

The options --script and --check-quantity can be given more than once.

usage: b2validation-bisect [-h] [--skip-compile] [--use-stash]
                           [--check-quantity CHECK_QUANTITY] [--script SCRIPT]
                           git_start_hash git_end_hash

Required Arguments

git_start_hash

Start git hash to check.

git_end_hash

End git hash to check.

options

--skip-compile

Do not trigger a compile, useful for saving time when setting up the quantity checks.

--use-stash

Use a git stash and a git stash pop before and after the git checkout.

--check-quantity

Check for a quantity in validation files

--script

Name of validation script to run after the compile and before the quantity check. Only this validation script and all scripts it depends on are executed.

25.2. Convenience Functions#

In order to simplify the generation of validation histograms in your basf2 steering file, you can use the following:

validation_tools.metadata.create_validation_histograms(path: Path, rootfile: str | PurePath, particlelist: str, variables_1d: List[Tuple] | None = None, variables_2d: List[Tuple] | None = None, description='') → None[source]#

Create histograms for all the variables and also label them to be useful in validation plots in one go. This is similar to the modularAnalysis.variablesToHistogram function but also sets the metadata correctly to be used by the validation

Parameters:

  • path (basf2.Path) – Path where to put the modules

  • rootfile (str or pathlib.PurePath) – Name of the output root file

  • particlelist (str) – Name of the particle list, can be empty for event dependent variables

  • variables_1d – List of 1D histogram definitions of the form var, bins, min, max, title, contact, description, check_for [, xlabel [, ylabel [, metaoptions]]]

  • variables_2d – List of 2D histogram definitions of the form var1, bins1, min1, max1, var2, bins2, min2, max2, title, contact, description, check_for [, xlabel [, ylabel [, metaoptions]]]

  • description – Common description/information of/about all plots in this ROOT file (will be displayed above the plots)

Warning

Sadly, there are two different ways to specify latex formulas.

  1. The ROOT-style using #: "Einstein-Pythagoras a^{2} + b^{2} = #frac{E}{m}"

    This style should be used for histogram title and labels, that is the title, xlabel and ylabel arguments

  2. The normal Latex style (with escaped backslashes or raw string literals): "Einstein-Pythagoras $a^2 + b^2 = \\frac{E}{m}$".

    This style should be used for all other fields like description, check_for

You can use the normal Latex style also for histogram title and descriptions but the PDF version of the plots will still be buggy and not show all formulas correctly.

You can also add metadata to an existing ROOT object using

validation_tools.metadata.validation_metadata_set(obj: TObject, title: str, contact: str, description: str, check: str, xlabel: str | None = None, ylabel: str | None = None, metaoptions='') → None[source]#

Set the validation metadata for a given object by setting the necessary values. This function can be used on any object supported by the Validation (histograms, profiles, ntuples)

Parameters:

  • obj – Instance of the object which should get the metadata

  • title (str) – Title to use for the object

  • contact (str) – Contact person, usually in the form “Name <email>”

  • description (str) – Text description what can be seen in the plot

  • check (str) – Text description what to look for in the validation for shifters to easily see if the distribution looks ok

  • xlabel (str) – If given try to set this as the label for the x axis

  • ylabel (str) – If given try to set this as the label for the y axis

  • metaoptions (str) – Metaoptions (additional options to influence the comparison between revisions, styling of the plot, etc.)

Warning

Different ways to specify LaTeX for different arguments: see create_validation_histograms

or add metadata to an already written ROOT file:

validation_tools.metadata.validation_metadata_update(rootfile: str | TFile | PurePath, name: str, *args, **argk) → None[source]#

This is a convenience helper for validation_metadata_set in case the objects have already been saved in a ROOT file before: It will open the file (or use an existing TFile), extract the object, add the metadata and save the new version to the file

Parameters:

  • rootfile (str, ROOT.TFile or pathlib.PurePath) – Name of the root file to open or an already open TFile instance

  • name (str) – Name of the object in the file

  • title (str) – Title to use for the object

  • contact (str) – Contact person, usually in the form “Name <email>”

  • description (str) – Text description what can be seen in the plot

  • check (str) – Text description what to look for in the validation for shifters to easily see if the distribution looks ok

  • xlabel (str) – If given try to set this as the label for the x axis

  • ylabel (str) – If given try to set this as the label for the y axis

  • metaoptions (str) – Metaoptions (additional options to influence the comparison between revisions, styling of the plot, etc.)

Warning

Different ways to specify LaTeX for different arguments: see create_validation_histograms

validation_tools.metadata.file_description_set(rootfile: TFile | str | PurePath, description: str) → None[source]#

Add file description validation metadata to a ROOT file.

Parameters:

  • rootfile (ROOT.TFile, str or pathlib.PurePath) – Name of the root file to open or an already open TFile instance

  • description (str) – Common description/information of/about all plots in this ROOT file (will be displayed above the plots)

Returns:

None

Contents