Tools for Physics Validation of the Software
Contents
24. Tools for Physics Validation of the Software#
The validation
package is used to validate the physics performance of
the software. As the software is constantly developed further, algorithms are
changed and we want to make sure that they don’t perform worse than before
(e.g. lower reconstruction efficiencies or slower runtime) and that they work
as intended.
To that end, every software package has a validation
subdirectory that
contains a number of simple steering files. The validation
package collects
these steering files and runs them. The resulting output *.root
files can then be compared
across software versions (“revisions”) to spot performance degradations and other problems.
To provide a simple user interface, the validation
package contains a small webserver,
that can also be run locally to show the comparison plots.
The validation framework is also run centrally on a nightly basis. The results are shown at https://validation.belle2.org/.
24.1. Command line utilities#
24.1.1. b2validation
: Run Physics Validation#
This program is used to run the full physics validation suite
usage: b2validation [-h] [-d] [-m {local}] [-i INTERVALS]
[-o OPTIONS [OPTIONS ...]] [-p PARALLEL]
[-pkg PACKAGES [PACKAGES ...]] [-s SELECT [SELECT ...]]
[-si SELECT_IGNORE_DEPENDENCIES [SELECT_IGNORE_DEPENDENCIES ...]]
[--send-mails]
[--send-mails-mode {full,incremental,automatic}] [-q]
[-t TAG] [--test] [--use-cache] [--view]
[--max-run-time MAX_RUN_TIME]
Optional Arguments
- -d, --dry
Perform a dry run, i.e. run the validation module without actually executing the steering files (for debugging purposes).
- -m, --mode
Possible choices: local
The mode which will be used for running the validation. Possible values: local. Default is ‘local’
- -i, --intervals
Comma seperated list of intervals for which to execute the validation scripts. Default is ‘nightly’
- -o, --options
One or more strings that will be passed to basf2 as arguments. Example: ‘-n 100’. Quotes are necessary!
- -p, --parallel
The maximum number of parallel processes to run the validation. Only used for local execution. Default is number of CPU cores.
- -pkg, --packages
The name(s) of one or multiple packages. Validation will be run only on these packages! E.g. -pkg analysis arich
- -s, --select
The file name(s) of one or more space separated validation scripts that should be executed exclusively. All dependent scripts will also be executed. E.g. -s ECL2D.C
- -si, --select-ignore-dependencies
The file name of one or more space separated validation scripts that should be executed exclusively. This will ignore all dependencies. This is useful if you modified a script that produces plots based on the output of its dependencies.
- --send-mails
Send email to the contact persons who have failed comparison plots. Mail is sent from b2soft@mail.desy.de via /usr/sbin/sendmail.
- --send-mails-mode
Possible choices: full, incremental, automatic
How to send mails: Full report, incremental report (new/changed warnings/failures only) or automatic (default; follow hard coded rule, e.g. full reports every Monday).
- -q, --quiet
Suppress the progress bar
- -t, --tag
The name that will be used for the current revision in the results folder. Default is ‘current’.
- --test
Execute validation in testing mode where only the validation scripts contained in the validation package are executed. During regular validation, these scripts are ignored.
- --use-cache
If validation scripts are marked as cacheable and their output files already exist, don’t execute these scripts again
- --view
Once the validation is finished, start the local web server and display the validation results in the system’s default browser.
- --max-run-time
By default, running scripts (that is, steering files executed bythe validation framework) are terminated after a certain time. Use this flag to change this setting by supplying the maximal run time in minutes. Value <=0 disables the run time upper limit entirely.
24.1.2. b2validation-plots
: Compare and plot a selection of revisions#
Select several different software versions (revisions) for which output files
were generated using b2validation
and generate comparison plots:
usage: b2validation-plots [-h] [-r [REVISIONS [REVISIONS ...]]] [-f]
[--popular]
Optional Arguments
- -r, --revisions
Takes a list of revisions (separated by spaces) and generates the plots for them
- -f, --force
Regenerates plots even if the requested combination exists already
- --popular
Creates several sets of plots for popular combinations of revisions.
24.1.3. b2validation-server
: Run Webserver Displaying the Physics Validation Results#
With b2validation-server
you can run a local webserver to display the
results of a physics validation performed with b2validation
.
usage: b2validation-server [-h] [-ip IP] [-p PORT] [-v] [--production]
Optional Arguments
- -ip, --ip
The IP address on which theserver starts. Default is ‘127.0.0.1’.
- -p, --port
The port number on which the server starts. Default is ‘8000’.
- -v, --view
Open validation website in the system’s default browser.
- --production
Run in production environment: no log/error output via website and no auto-reload
24.1.4. b2validation-bisect
: Find commits which introduced validation degradation#
Utility script to be used in conjunction with the git bisect functionality 1 to search for degradations in validation variables. To use it, you can set the last known good git commit and the first known bad commit and start the bisecting process. Git will then checkout commits between the good and the bad and perform a binary search to arrive at the first bad commit.
To give feed back to git, which is first bad commit, this script either returns the exit code 0 for good commit, code 125 to ignore the current commit or code 1 for a bad commit.
Information contained within .root files produced by the official validation scripts or your custom scripts can be used to distinguish between good and bad commits.
The syntax of the --check-quantity
option is the following: <root
filename>:<quantity>:<compare op>:<value>
quantity
is the name of a histogram in the given root file. For histograms, the quantity to check for can be given in square brackets, for regular histograms, this is one of this strings:mean_x
= mean of the histogramentries
= number of entries in all binsmean_y
= the mean of the bin contentmean_y_zero_suppressed
= the mean of all the bin contents > 0
compare op
can be>
or<
value
is the reference value as a float to compare to
As an example, to identify the commit which made the tracking efficiency drop below 0.93, the following snippet can be used:
git bisect good <last good release>
git bisect bad <first bad release>
git bisect run bisect_validation.py --script 13_trackingEfficiency_createPlots.py
--check-quantity "results/current/tracking/TrackingValidation.root:hEfficiency[mean_y]:>:0.93"
usage: b2validation-bisect [-h] [--skip-compile] [--report-compile-fail]
[--report-execution-fail]
[--check-quantity CHECK_QUANTITY]
[--check-failed-comparison]
[--max-failed-comparison MAX_FAILED_COMPARISON]
[--test-check] [--execute EXECUTE]
[--script SCRIPT] [--keep]
Optional Arguments
- --skip-compile
Do not trigger a compile, useful for saving time when setting up the quantity checks.
- --report-compile-fail
Report a bad revision to git if the code base did not compile. By default, failing compiles are ignored.
- --report-execution-fail
Report a bad revision to git if the provided script has an error code != 0. By default, this is reported to git.
- --check-quantity
Check for a quantity in validation files
- --check-failed-comparison
Check if one comparison to a reference plot failed
- --max-failed-comparison
Maximum number of comparisons allowed to fail
- --test-check
Just check for the quantity, but do not run any git or compile commands
- --execute
File to execute after the compile and before the quantity check
- --script
Name of validation script to run after the compile and before the quantity check. Only this validation script and all scripts it depends on are executed.
- --keep
Keep the result folder of each bisect step for later examination
24.1.5. b2validation-revisions
: Extract quantities from each revision#
Tool to do for each git commit in a specific range:
- check out the commit (configurable if you want to run git stash and git stash pop before and afterwards)
- run scons
or not (configurable)
- run a script (given on command line)
- extract the meanX
, the meanY
and the zeroSupressedMeanY
from variables from ROOT files (also configured on the commandline)
- write all variables together with their git hash and git log message into a pandas DataFrame and this dataframe into the results.pkl
file as a pickle stream.
For example if you want to examine some commits related to tracking, you can use:
b2validation-revisions 5e066d31b391bb240cb2b381e67c5488681a8ec6 05d5bf078a45f6af8022bf06b398213d0bc2409d --check-quantity ~/basf2/tracking/validation/results/current/TrackingValidation.root:hEfficiency --script "validate_basf2 -s *_trackingEfficiency_*.py"
which will run through all commits from
5e066d31b391bb240cb2b381e67c5488681a8ec6
to
05d5bf078a45f6af8022bf06b398213d0bc2409d
and execute validate_basf2 -s *_trackingEfficiency_*.py
for each of them.
Then it will extract the meanX
, the meanY
and
the zeroSupressedMeanY
of the TNtuple hEfficiency
in the file
~/basf2/tracking/validation/results/current/tracking/TrackingValidation.root
.
The options --script
and --check-quantity
can be given more than once.
usage: b2validation-bisect [-h] [--skip-compile] [--use-stash]
[--check-quantity CHECK_QUANTITY] [--script SCRIPT]
git_start_hash git_end_hash
Required Arguments
- git_start_hash
Start git hash to check.
- git_end_hash
End git hash to check.
Optional Arguments
- --skip-compile
Do not trigger a compile, useful for saving time when setting up the quantity checks.
- --use-stash
Use a git stash and a git stash pop before and after the git checkout.
- --check-quantity
Check for a quantity in validation files
- --script
Name of validation script to run after the compile and before the quantity check. Only this validation script and all scripts it depends on are executed.
24.2. Convenience Functions#
In order to simplify the generation of validation histograms in your basf2 steering file, you can use the following:
- validation_tools.metadata.create_validation_histograms(path: pybasf2.Path, rootfile: Union[str, pathlib.PurePath], particlelist: str, variables_1d: Optional[List[Tuple]] = None, variables_2d: Optional[List[Tuple]] = None, description='') None [source]#
Create histograms for all the variables and also label them to be useful in validation plots in one go. This is similar to the
modularAnalysis.variablesToHistogram
function but also sets the metadata correctly to be used by the validation- Parameters
path (basf2.Path) – Path where to put the modules
rootfile (str or pathlib.PurePath) – Name of the output root file
particlelist (str) – Name of the particle list, can be empty for event dependent variables
variables_1d – List of 1D histogram definitions of the form
var, bins, min, max, title, contact, description, check_for [, xlabel [, ylabel [, metaoptions]]]
variables_2d – List of 2D histogram definitions of the form
var1, bins1, min1, max1, var2, bins2, min2, max2, title, contact, description, check_for [, xlabel [, ylabel [, metaoptions]]]
description – Common description/information of/about all plots in this ROOT file (will be displayed above the plots)
Warning
Sadly, there are two different ways to specify latex formulas.
The ROOT-style using
#
:"Einstein-Pythagoras a^{2} + b^{2} = #frac{E}{m}"
This style should be used for histogram title and labels, that is the
title
,xlabel
andylabel
argumentsThe normal Latex style (with escaped backslashes or raw string literals):
"Einstein-Pythagoras $a^2 + b^2 = \\frac{E}{m}$"
.This style should be used for all other fields like
description
,check_for
You can use the normal Latex style also for histogram title and descriptions but the PDF version of the plots will still be buggy and not show all formulas correctly.
You can also add metadata to an existing ROOT object using
- validation_tools.metadata.validation_metadata_set(obj: cppyy.gbl.TObject, title: str, contact: str, description: str, check: str, xlabel: Optional[str] = None, ylabel: Optional[str] = None, metaoptions='') None [source]#
Set the validation metadata for a given object by setting the necessary values. This function can be used on any object supported by the Validation (histograms, profiles, ntuples)
- Parameters
obj – Instance of the object which should get the metadata
title (str) – Title to use for the object
contact (str) – Contact person, usually in the form “Name <email>”
description (str) – Text description what can be seen in the plot
check (str) – Text description what to look for in the validation for shifters to easily see if the distribution looks ok
xlabel (str) – If given try to set this as the label for the x axis
ylabel (str) – If given try to set this as the label for the y axis
metaoptions (str) – Metaoptions (additional options to influence the comparison between revisions, styling of the plot, etc.)
Warning
Different ways to specify LaTeX for different arguments: see
create_validation_histograms
or add metadata to an already written ROOT file:
- validation_tools.metadata.validation_metadata_update(rootfile: Union[str, cppyy.gbl.TFile, pathlib.PurePath], name: str, *args, **argk) None [source]#
This is a convenience helper for
validation_metadata_set
in case the objects have already been saved in a ROOT file before: It will open the file (or use an existing TFile), extract the object, add the metadata and save the new version to the file- Parameters
rootfile (str, ROOT.TFile or pathlib.PurePath) – Name of the root file to open or an already open TFile instance
name (str) – Name of the object in the file
title (str) – Title to use for the object
contact (str) – Contact person, usually in the form “Name <email>”
description (str) – Text description what can be seen in the plot
check (str) – Text description what to look for in the validation for shifters to easily see if the distribution looks ok
xlabel (str) – If given try to set this as the label for the x axis
ylabel (str) – If given try to set this as the label for the y axis
metaoptions (str) – Metaoptions (additional options to influence the comparison between revisions, styling of the plot, etc.)
Warning
Different ways to specify LaTeX for different arguments: see
create_validation_histograms
- validation_tools.metadata.file_description_set(rootfile: Union[cppyy.gbl.TFile, str, pathlib.PurePath], description: str) None [source]#
Add file description validation metdata to a ROOT file.
- Parameters
rootfile (ROOT.TFile, str or pathlib.PurePath) – Name of the root file to open or an already open TFile instance
description (str) – Common description/information of/about all plots in this ROOT file (will be displayed above the plots)
- Returns
None