b2conditionsdb: Conditions DB interface

4.6. b2conditionsdb: Conditions DB interface#

This tool provides a command line interface to all the tasks related to the Conditions database: manage globaltags and iovs as well as upload new payloads or download of existing payloads.

The usage of this tool is similar to git: there are sub commands like for example tag which groups all actions related to the management of globaltags. All the available commands are listed below.

While the read access to the conditions database is always allowed (e.g. for downloading globaltags and payloads), users need a valid JSON Web Token (JWT) to authenticate to the conditions database when using this tool for creating/manpipulating globaltags or uploading payloads. For practical purposes, it is only necessary to know that a JWT is a string containing encrypted information, and that string is stored in a file. More information about what a JWT is can be found on Wikipedia.

Warning

By default, users can only create globaltags whose name starts with user_<username>_ or temp_<username>_, where username is the B2MMS username.

The tool automatically queries the JWT issuing server (https://token.belle2.org) and gets a valid token by asking the B2MMS username and password. The issued “default” JWT has a validity of 1 hour; after it expires, a new JWT needs to be obtained for authenticating the conditions database. When retrieved by this tool, the JWT is stored locally in the file ${HOME}/b2cdb_${BELLE2_USER}.token.

Some selected users (e.g. the calibration managers) are granted a JWT with an extended validity (30 days) to allow smooth operations with some automated workflows. Such “extended” JWTs are issued by a different server (https://token.belle2.org/extended). B2MMS username and password are necessary for getting the extended JWT. The extended JWT differs from the default JWT only by its validity and can be obtained only by manually querying the alternative server. If queried via web browser, a file containing the extended JWT will be downloaded in case the user is granted it. The server can also be queried via command line using wget --user USERNAME --ask-password --no-check-certificate https://token.belle2.org/extended or curl -u USERNAME -k https://token.belle2.org/extended.

If the environment variable ${BELLE2_CDB_AUTH_TOKEN} is defined and points to a file containing a valid JWT, the b2conditionsdb tools use this token to authenticate with the CDB instead of querying the issuing server. This is useful when using an extended token. Please note that, in case the JWT to which ${BELLE2_CDB_AUTH_TOKEN} points is not valid anymore, the b2conditionsdb tools will attempt to get a new one and store it into ${BELLE2_CDB_AUTH_TOKEN}. If a new extended token is needed, it has to be manually obtained via https://token.belle2.org/extended and stored into ${BELLE2_CDB_AUTH_TOKEN}.

usage: b2conditionsdb [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                      [--auth-token AUTH_TOKEN]
                      {diff,download,dump,iov,iovs,legacydownload,tag,upload}
                      ...

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

4.6.1. Sub-command b2conditionsdb diff#

This command allows to compare two globaltags. It will show the changes in a format similar to a unified diff but by default it will not show any context, only the new or removed payloads. Added payloads are marked with a + in the first column, removed payloads with a -

If --full is given it will show all payloads, even the ones common to both globaltags. The differences can be limited to a given run and limited to a set of payloads names using --filter or --exclude. If the --regex option is supplied the search term will be interpreted as a python regular expression where the case is ignored.

Changed in version release-03-00-00: modified output structure and added --human-readable

Changed in version after: release-04-00-00 added parameter --checksums and --show-ids

b2conditionsdb diff [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                    [--auth-token AUTH_TOKEN] [--full] [--run N N]
                    [--human-readable] [--checksums] [--show-ids]
                    [--run-range FIRST_EXP FIRST_RUN FINAL_EXP FINAL_RUN]
                    [-f SEARCHTERM] [-e EXCLUDE] [-r]
                    TAGNAME1 TAGNAME2

Required Arguments

TAGNAME1

base for comparison

TAGNAME2

tagname to compare

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--full

If given print all iovs, also those which are the same in both tags

--run

exp and run numbers to limit showing iovs to a ones present in a given run

--human-readable

If given the iovs will be written in a more human friendly format. Also repeated payload names will be omitted to create a more readable listing.

--checksums

If given don’t show the revision number but the md5 checksum

--show-ids

If given also show the payload and iov ids for each iov

--run-range

Can be four numbers to limit the run range to be comparedOnly iovs overlapping, even partially, with this range will be considered.

-f, --filter

only payloads matching this pattern will be included. Pattern is case insensitive

-e, --exclude

payloads matching this pattern will be excluded. Pattern is case insensitive

-r, --regex

if given, –filter or –exclude options will be interpreted as a python regular expression (see https://docs.python.org/3/library/re.html)

4.6.2. Sub-command b2conditionsdb download#

This command allows to download the information from one or more globaltags from the central database to be used locally.

The command requires at least one tag name to download. It will check for existing payloads in the output directory and only download payloads which are not present or don’t have the expected checksum.

By default this script will create a local directory called conditions/ which contains a metadata.sqlite with all the payload information of all selected globaltags and sub directories containing all the payload files.

This can be changed by specifying a different name for the metadata file using the -o argument but the payloads will always be saved in sub directories in the same directory as the sqlite file.

Changed in version release-04-00-00: Previously this command was primarily intended to download payloads for one globaltag and optionally create a text file with payload information as well as download all necessary file. This has been changed and will now create a sqlite file containing the payload metadata. If you need the old behavior please use the command b2conditionsdb-legacydownload

b2conditionsdb download [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                        [--auth-token AUTH_TOKEN] [-o DATABASEFILE] [-f]
                        [--append] [--no-download | --only-download]
                        [--delete-extra-payloads] [--ignore-missing]
                        [-j NPROCESS] [--retries RETRIES]
                        [--tag-pattern | --tag-regex]
                        [TAGNAME ...]

Required Arguments

TAGNAME

globaltag to download

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

-o, --dbfile

Name of the database file to create (default: “conditions/metadata.sqlite”)

-f, --force

Don’t ask permission if the output database file exists

--append

Append to the existing database file if possible. Otherwise the content in the database file will be overwritten

--no-download

Don’t download any payloads, just fetch the metadata information

--only-download

Assume the metadata file is already filled, just make sure all payloads are downloaded

--delete-extra-payloads

if given this script will delete all extra files that follow the payload naming convention AB/{name}_r{revision}.root if they are not referenced in the database file.

--ignore-missing

Ignore missing globaltags and download all other tags

-j

Number of concurrent connections to use for file download (default: 1)

--retries

Number of retries on connection problems (default: 3)

--tag-pattern

if given, all globaltags which match the shell-style pattern TAGNAME will be downloaded: * stands for anything, ? stands for a single character.

--tag-regex

if given, all globaltags matching the regular expression given by TAGNAME will be downloaded (see https://docs.python.org/3/library/re.html).

4.6.3. Sub-command b2conditionsdb dump#

Added in version release-03-00-00.

This command will dump the payload contents stored in a given payload. One can either specify the payloadId (from a previous output of b2conditionsdb iov), the payload name and its revision in the central database, or directly specify a local database payload file.

Examples

Dump the content of a previously downloaded payload file:

$ b2conditionsdb dump -f centraldb/dbstore_BeamParameters_rev_59449.root

Dump the content of a payload by name and revision directly from the central database:

$ b2conditionsdb dump -r BeamParameters 59449

Dump the content of the payload by name which is valid in a given globaltag for a given experiment and run:

$ b2conditionsdb dump -g BeamParameters main_2021-08-04 0 0

Or directly by payload id from a previous call to b2conditionsdb iov:

$ b2conditionsdb dump -i 59685

Usage

Depending on whether you want to display a payload by its id in the database, its name and revision in the database or from a local file provide one of the arguments -i, -r, -f or -g

Changed in version after: release-04-00-00 added argument -r to directly dump a payload valid for a given run in a given globaltag

b2conditionsdb dump [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                    [--auth-token AUTH_TOKEN] ([-i PAYLOADID | -r NAME
                    REVISION | -f FILENAME | -g NAME GLOBALTAG EXP RUN)]
                    [--show-typenames] [--show-streamerinfo]

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

-i, --id

payload id to dump

-r, --revision

Name and revision of the payload to dump

-f, --file

Dump local payload file

-g, --valid

Dump the payload valid for the given exp, run number in the given globaltag

--show-typenames

If given show the type names of all classes. This makes output more crowded but can be helpful for complex objects.

--show-streamerinfo

If given show the StreamerInfo for the classes in the the payload file. This can be helpful to find out which version of a payload object is included and what are the members

4.6.4. Sub-command b2conditionsdb iov#

This command lists all IoVs defined in a given globaltag. The list can be limited to a given run and optionally searched using --filter or --exclude. If the --regex option is supplied the search term will be interpreted as a Python regular expression where the case is ignored.

Changed in version release-03-00-00: modified output structure and added --human-readable

Changed in version after: release-04-00-00 added parameter --checksums and --show-ids

Changed in version after: release-08-00-04 added parameter --run-range

b2conditionsdb iov [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                   [--auth-token AUTH_TOKEN] [--run N N] [--detail]
                   [--human-readable] [--checksums] [--show-ids]
                   [--run-range FIRST_EXP FIRST_RUN FINAL_EXP FINAL_RUN]
                   [-f SEARCHTERM] [-e EXCLUDE] [-r]
                   TAGNAME

Required Arguments

TAGNAME

globaltag for which the the IoVs should be listed

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--run

exp and run numbers to limit showing iovs to a ones present in a given run

--detail

if given show a detailed information for all IoVs including details of the payloads

--human-readable

If given the iovs will be written in a more human friendly format. Also repeated payload names will be omitted to create a more readable listing.

--checksums

If given don’t show the revision number but the md5 checksum

--show-ids

If given also show the payload and iov ids for each iov

--run-range

Can be four numbers to limit the run range to be shownOnly iovs overlapping, even partially, with this range will be shown.

-f, --filter

only payloads matching this pattern will be included. Pattern is case insensitive

-e, --exclude

payloads matching this pattern will be excluded. Pattern is case insensitive

-r, --regex

if given, –filter or –exclude options will be interpreted as a python regular expression (see https://docs.python.org/3/library/re.html)

4.6.5. Sub-command b2conditionsdb iovs#

This command allows to modify, delete or copy iovs of a globaltag. If no other command is given do nothing.

b2conditionsdb iovs [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                    [--auth-token AUTH_TOKEN]
                    {copy,delete,modify} ...

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

Sub-command b2conditionsdb iovs copy#

This command allows to copy the iovs from a globaltags to another one, optionally limiting the iovs to be copied to those of a specific payload, revision, IoV id or run range.

b2conditionsdb iovs copy [-h] [--debugging] [--help-full]
                         [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                         [--iov-id IOV_ID] [--iov-pattern IOV_PATTERN]
                         [--run-range FIRST_EXP FIRST_RUN FINAL_EXP FINAL_RUN]
                         [--fully-contained] [--set-run-range] [-f SEARCHTERM]
                         [-e EXCLUDE] [-r] [--revision revision] [--dry-run]
                         [--replace] [-j NPROCESS]
                         INPUT_TAGNAME OUTPUT_TAGNAME

Required Arguments

INPUT_TAGNAME

globaltag for which the the IoVs should be copied

OUTPUT_TAGNAME

globaltag to which the the IoVs should be copied

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--iov-id

IoVid of the iov to be considered

--iov-pattern

whitespace-separated string with pattern of the iov to be considered. Use * to mark the fields that should be ignored. Valid patterns are 0 0 -1 -1 (a very specific IoV), 0 * -1 -1 (any iov that starts in any run of exp 0 and ends exactly in exp -1, run -1), * * 3 45 (any Iov ending in exp 3, run 45, regardless from where it starts).

--run-range

Can be four numbers to limit the run range to be considered Only iovs overlapping, even partially, with this range will be considered.

--fully-contained

If given together with --run_range limit the list of payloads to the ones fully contained in the given run range

--set-run-range

If given together with --run_range modify the interval of validity of partially overlapping iovs to be fully contained in the given run range

-f, --filter

only payloads matching this pattern will be included. Pattern is case insensitive

-e, --exclude

payloads matching this pattern will be excluded. Pattern is case insensitive

-r, --regex

if given, –filter or –exclude options will be interpreted as a python regular expression (see https://docs.python.org/3/library/re.html)

--revision

Specify the revision of the payload to be removed

--dry-run

Don’t do anything, just print what would be done

--replace

Modify the iovs in the output tag to avoid overlaps

-j

Number of concurrent threads to use.

Sub-command b2conditionsdb iovs delete#

This command allows to delete the iovs from a globaltags, optionally limiting the iovs to be deleted to those of a specific payload, revision, IoVid or run range.

b2conditionsdb iovs delete [-h] [--debugging] [--help-full]
                           [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                           [--iov-id IOV_ID] [--iov-pattern IOV_PATTERN]
                           [--run-range FIRST_EXP FIRST_RUN FINAL_EXP FINAL_RUN]
                           [--fully-contained] [-f SEARCHTERM] [-e EXCLUDE]
                           [-r] [--revision revision] [--dry-run]
                           [-j NPROCESS]
                           INPUT_TAGNAME

Required Arguments

INPUT_TAGNAME

globaltag for which the the IoVs should be deleted

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--iov-id

IoVid of the iov to be considered

--iov-pattern

whitespace-separated string with pattern of the iov to be considered. Use * to mark the fields that should be ignored. Valid patterns are 0 0 -1 -1 (a very specific IoV), 0 * -1 -1 (any iov that starts in any run of exp 0 and ends exactly in exp -1, run -1), * * 3 45 (any Iov ending in exp 3, run 45, regardless from where it starts).

--run-range

Can be four numbers to limit the run range to be considered Only iovs overlapping, even partially, with this range will be considered.

--fully-contained

If given together with --run_range limit the list of payloads to the ones fully contained in the given run range

-f, --filter

only payloads matching this pattern will be included. Pattern is case insensitive

-e, --exclude

payloads matching this pattern will be excluded. Pattern is case insensitive

-r, --regex

if given, –filter or –exclude options will be interpreted as a python regular expression (see https://docs.python.org/3/library/re.html)

--revision

Specify the revision of the payload to be removed

--dry-run

Don’t do anything, just print what would be done

-j

Number of concurrent threads to use.

Sub-command b2conditionsdb iovs modify#

This command allows to modify the iovs from a globaltags, optionally limiting the iovs to be modified to those of a specific payload, revision, IoV id or run range.

b2conditionsdb iovs modify [-h] [--debugging] [--help-full]
                           [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                           [--iov-id IOV_ID] [--iov-pattern IOV_PATTERN]
                           [--run-range FIRST_EXP FIRST_RUN FINAL_EXP FINAL_RUN]
                           [--fully-contained] [-f SEARCHTERM] [-e EXCLUDE]
                           [-r] [--revision revision] [--dry-run]
                           [-j NPROCESS]
                           INPUT_TAGNAME NEW_IOV

Required Arguments

INPUT_TAGNAME

globaltag for which the the IoVs should be modified

NEW_IOV

New iov to be set to all considered iovs. It should be a string with 4 numbers separated by spaces. Use * to mark the fields that should not be modified. For example if 7 0 * * is given the iov (7, 1, 9, 42) will become (7 0 9 42).

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--iov-id

IoVid of the iov to be considered

--iov-pattern

whitespace-separated string with pattern of the iov to be considered. Use * to mark the fields that should be ignored. Valid patterns are 0 0 -1 -1 (a very specific IoV), 0 * -1 -1 (any iov that starts in any run of exp 0 and ends exactly in exp -1, run -1), * * 3 45 (any Iov ending in exp 3, run 45, regardless from where it starts).

--run-range

Can be four numbers to limit the run range to be considered Only iovs overlapping, even partially, with this range will be considered.

--fully-contained

If given together with --run_range limit the list of payloads to the ones fully contained in the given run range

-f, --filter

only payloads matching this pattern will be included. Pattern is case insensitive

-e, --exclude

payloads matching this pattern will be excluded. Pattern is case insensitive

-r, --regex

if given, –filter or –exclude options will be interpreted as a python regular expression (see https://docs.python.org/3/library/re.html)

--revision

Specify the revision of the payload to be removed

--dry-run

Don’t do anything, just print what would be done

-j

Number of concurrent threads to use.

4.6.6. Sub-command b2conditionsdb legacydownload#

This command allows to download a globaltag from the central database to be used locally, either as lookup directory for payloads or as a standalone local database if –create-dbfile is specified.

The command requires the TAGNAME to download and optionally an output directory which defaults to centraldb in the local working directory. It will check for existing payloads in the output directory and only download payloads which are not present or don’t have the expected checksum.

One can filter the payloads to be downloaded by payload name using the –filter, –exclude and –regex options.

Added in version release-04-00-00: This has been renamed from download and is kept for compatibility

Warning

Downloading a globaltag should be done in the new format creating sqlite database files. Please use this legacy tool only for downloading “small” globaltags or very few payloads.

b2conditionsdb legacydownload [-h] [--debugging] [--help-full]
                              [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                              [-c] [-f SEARCHTERM] [-e EXCLUDE] [-r]
                              [-j NPROCESS] [--retries RETRIES]
                              [--run-range FIRST_EXP FIRST_RUN FINAL_EXP FINAL_RUN]
                              [--tag-pattern | --tag-regex]
                              TAGNAME [DIR]

Required Arguments

TAGNAME

globaltag to download

DIR

directory to put the payloads into (default: “centraldb”)

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

-c, --create-dbfile

if given save information about all payloads in DIR/database.txt

-f, --filter

only payloads matching this pattern will be included. Pattern is case insensitive

-e, --exclude

payloads matching this pattern will be excluded. Pattern is case insensitive

-r, --regex

if given, –filter or –exclude options will be interpreted as a python regular expression (see https://docs.python.org/3/library/re.html)

-j

Number of concurrent connections to use for file download (default: 1)

--retries

Number of retries on connection problems (default: 3)

--run-range

Can be four numbers to limit the run range to be downloadedOnly iovs overlapping, even partially, with this range will be downloaded.

--tag-pattern

if given, all globaltags which match the shell-style pattern TAGNAME will be downloaded: * stands for anything, ? stands for a single character. If -c is given as well the database files will be DIR/TAGNAME.txt

--tag-regex

if given, all globaltags matching the regular expression given by TAGNAME will be downloaded (see https://docs.python.org/3/library/re.html). If -c is given as well the database files will be DIR/TAGNAME.txt

4.6.7. Sub-command b2conditionsdb tag#

This command allows to list, show, create modify or clone globaltags in the central database. If no other command is given it will list all tags as if b2conditionsdb tag show was given.

b2conditionsdb tag [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                   [--auth-token AUTH_TOKEN]
                   {clone,create,list,merge,modify,runningupdate,show,state}
                   ...

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

Sub-command b2conditionsdb tag clone#

This command allows to clone a given globaltag with a new name but still containing all the IoVs defined in the original globaltag.

b2conditionsdb tag clone [-h] [--debugging] [--help-full]
                         [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                         TAGNAME NEWNAME

Required Arguments

TAGNAME

globaltag to be cloned

NEWNAME

name of the cloned globaltag

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

Sub-command b2conditionsdb tag create#

This command creates a new globaltag in the database with the given name and description. The name can only contain alphanumeric characters and the characters +-_:.

Warning

By default, users can only create globaltags whose name starts with user_<username>_ or temp_<username>_, where username is the B2MMS username.

b2conditionsdb tag create [-h] [--debugging] [--help-full]
                          [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                          [-u USER]
                          TYPE TAGNAME DESCRIPTION

Required Arguments

TYPE

type of the globaltag to create, usually one of DEV or RELEASE

TAGNAME

name of the globaltag to create

DESCRIPTION

description of the globaltag

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

-u, --user

username who created the tag. If not given we will try to supply a useful default

Sub-command b2conditionsdb tag list#

This command allows to list all globaltags, optionally limiting the output to ones matching a given search term. By default invalidated globaltags will not be included in the list, to show them as well please add --with-invalid as option. Alternatively one can use --only-published to show only tags which have been published

If the --regex option is supplied the search term will be interpreted as a Python regular expression where the case is ignored.

b2conditionsdb tag list [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                        [--auth-token AUTH_TOKEN] [--detail]
                        [--with-invalid | --only-published] [-f SEARCHTERM]
                        [-e EXCLUDE] [-r]

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--detail

show detailed information for all tags instead of summary table

--with-invalid

if given also invalidated tags will be shown

--only-published

if given only published tags will be shown

-f, --filter

only tags matching this pattern will be included. Pattern is case insensitive

-e, --exclude

tags matching this pattern will be excluded. Pattern is case insensitive

-r, --regex

if given, –filter or –exclude options will be interpreted as a python regular expression (see https://docs.python.org/3/library/re.html)

Sub-command b2conditionsdb tag merge#

This command allows to merge a number of globaltags into a single globaltag. Payloads from later globaltags in the list of arguments are used to fill gaps present in earlier globaltags.

The result is equivalent to having multiple globaltags setup in the conditions access for basf2 (highest priority goes first).

Warning

The order of the globaltags is highest priority first, so payloads from globaltags earlier on the command line will be taken with before globaltags from later tags.

For each globaltag in the list we copy all payloads to the output globaltag if there is no payload of that name valid for the given interval of validity in any previous globaltags in the list.

If the payload overlaps partially with a payload from a previous globaltag in the list the interval of validity is shortened (and possibly split) to not overlap but to just fill the gaps.

For example:

Globaltag A contains

payload1, rev 2, valid from 1,0 to 1,10
payload1, rev 3, valid from 1,20 to 1,22
payload2, rev 1, valid from 1,0 to 1,-1

Globaltag B contains

payload1, rev 1, valid from 1,1 to 1,30
payload2, rev 2, valid from 0,1 to 1,20

Then running b2conditionsdb tag merge -o C A B, the output globaltag C after the merge will contain:

payload1, rev 2, valid from 1,0 to 1,10
payload1, rev 1, valid from 1,11 to 1,19
payload1, rev 3, valid from 1,20 to 1,22
payload1, rev 1, valid from 1,23 to 1,30
payload2, rev 2, valid from 0,1 to 0,-1
payload2, rev 1, valid from 1,0 to 1,-1

When finished, this command will print a table of payloads and their validity and from which globaltag they were taken. If --dry-run is given it will only print the list of payloads.

Optionally one can specify --run-range to limit the run range for which the merging should take place. In the example above, running with --run-range 1 0 1 21 the result would be

payload1, rev 2, valid from 1,0 to 1,10
payload1, rev 1, valid from 1,11 to 1,19
payload1, rev 3, valid from 1,20 to 1,21
payload2, rev 1, valid from 1,0 to 1,21

Added in version release-05-01-00.

b2conditionsdb tag merge [-h] [--debugging] [--help-full]
                         [--base-url BASE_URL] [--auth-token AUTH_TOKEN] -o
                         OUTPUT [--dry-run]
                         [--run-range FIRST_EXP FIRST_RUN FINAL_EXP FINAL_RUN]
                         [-j NPROCESS]
                         globaltag [globaltag ...]

Required Arguments

globaltag

name of the globaltag

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--dry-run

Don’t do anything, just print a table with the results

--run-range

Can be for numbers to limit the run range to putin the output globaltag: All iovs will be limited to be in this range.

-j

Number of concurrent threads to use for creating payloads into the output globaltag.

required named arguments

-o, --output

Name of the output globaltag

Sub-command b2conditionsdb tag modify#

This command allows to change the name or description of an existing globaltag. You can supply any combination of -n,-d,-t and only the given values will be changed

b2conditionsdb tag modify [-h] [--debugging] [--help-full]
                          [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                          [-n NAME] [-d DESCRIPTION] [-t TYPE] [-u USER]
                          [-s STATE]
                          TAGNAME

Required Arguments

TAGNAME

globaltag to modify

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

-n, --name

new name

-d, --description

new description

-t, --type

new type of the globaltag

-u, --user

username who created the tag. If not given we will try to supply a useful default

-s, --state

new globaltag state, see the command tag state for details

Sub-command b2conditionsdb tag runningupdate#

This command will calculate and apply the necessary updates to a running globaltag with a given staging globaltag

Running tags are defined as “immutable for existing data but conditions for newer runs may be added” and the only modification allowed is to add new payloads for new runs or close existing payloads to no longer be valid for new runs.

This command takes previously prepared and validated payloads in a staging globaltag and will then calculate which payloads to close and what to add to the running globaltag.

For this to work we require

  1. A running globaltag in the state “RUNNING”

  2. A (experiment, run) number from which run on the update should be valid. This run number needs to be

    1. bigger than the start of validity for all iovs in the running tag

    2. bigger than the end of validity for all closed iovs (i.e. not valid to infinity) in the running tag

  3. A staging globaltag with the new payloads in state “VALIDATED”

    1. payloads in the staging tag starting at (0,0) will be interpreted as starting at the first valid run for the update

    2. all other payloads need to start at or after the first valid run for the update.

    3. The globaltag needs to be gap and overlap free

    4. All payloads in the staging tag should have as last iov an open iov (i.e. valid to infinity) but this can be disabled.

The script will check all the above requirements and will then calculate the necessary operations to

  1. Add all payloads from the staging tag where a start validity of (0, 0) is replaced by the starting run for which this update should be valid.

  2. close all iovs for payloads in the running tags just before the corresponding iov of the same payload in the staging tag, so either at the first run for the update to be valid or later

  3. Optionally, make sure all payloads in the staging tag end in an open iov.

Examples

Running tag contains

payload1, rev 1, valid from 0,1 to 1,0
payload1, rev 2, valid from 1,1 to -1,-1
payload2, rev 1, valid from 0,1 to -1,-1
payload3, rev 1, valid from 0,1 to 1,0
payload4, rev 1, valid from 0,1 to -1,-1
payload5, rev 1, valid from 0,1 to -1,-1

Staging tag contains

payload1, rev 3, valid from 0,0 to 1,8
payload1, rev 4, valid from 1,9 to 1,20
payload2, rev 2, valid from 1,5 to 1,20
payload3, rev 2, valid from 0,0 to -1,-1
payload4, rev 1, valid from 0,0 to 1,20

Then running b2conditionsdb tag runningupdate running staging --run 1 2 --allow-closed, the running globaltag after the update will contain

payload1, rev 1, valid from 0,1 to 1,0
payload1, rev 2, valid from 1,1 to 1,1
payload1, rev 3, valid from 1,2 to 1,8
payload1, rev 4, valid from 1,9 to 1,20
payload2, rev 1, valid from 0,1 to 1,4
payload2, rev 2, valid from 1,5 to 1,20
payload3, rev 1, valid from 0,1 to 1,0
payload3, rev 2, valid from 1,2 to -1,-1
payload4, rev 1, valid from 0,1 to 1,20
payload5, rev 1, valid from 0,1 to -1,-1

Note that

  • the start of payload1 and payload3 in staging has been adjusted

  • payload2 in the running tag as been closed at 1,4, just before the validity from the staging tag

  • payload3 was already closed in the running tag so no change is performed. This might result in gaps but is intentional

  • payload4 was not closed at rim 1,2 because the staging tag had the same revision of the payload so the these were merged to one long validity.

  • payload5 was not closed as there was no update to it in the staging tag. If we would have run with --full-replacement it would have been closed.

  • if we would have chosen --run 1 1 the update would have failed because payload1, rev2 in running starts at 1,1 so we would have a conflict

  • if we would have chosen --run 1 6 the update would have failed because payload2 in the staging tag starts before this run

  • if we would have chosen to open the final iovs in staging by using --fix-closed, payload1, rev 4; payload2, rev 2 and payload4 rev 1 would be valid until -1,-1 after the running tag. In fact, payload 4 would not be changed at all.

b2conditionsdb tag runningupdate [-h] [--debugging] [--help-full]
                                 [--base-url BASE_URL]
                                 [--auth-token AUTH_TOKEN] -r EXP RUN
                                 [--allow-closed | --fix-closed | --simple-mode | --full-replacement]
                                 [--dry-run]
                                 running staging

Required Arguments

running

name of the running globaltag

staging

name of the staging globaltag

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--allow-closed

if given allow payloads in the staging tag to not be open, i.e. they don’t have to be open ended in the update. Useful to retire a payload by adding one last update

--fix-closed

if given automatically open the last iov for each payload in staging if it is closed.

--simple-mode

if given require the staging tag to solely consist of fully infinite validities: Only one iov per payload with a validity of (0,0,-1,-1)

--full-replacement

if given perform a full replacement and close all open iovs in the running tag not present in the staging tag. After such an update exactly the payloads in the staging tag will be valid after the given run. This allows for closed iovs in the staging tag as with --allow-closed

--dry-run

Only show the changes, don’t try to apply them

required named arguments

-r, --run

First experiment + run number for which the update should be valid. Two numbers separated by space

Sub-command b2conditionsdb tag show#

This command will show details for the given globaltags like name, description and number of payloads.

b2conditionsdb tag show [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                        [--auth-token AUTH_TOKEN]
                        TAGNAME [TAGNAME ...]

Required Arguments

TAGNAME

globaltags to show

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

Sub-command b2conditionsdb tag state#

This command changes the state of a globaltag to the given value.

Usually the valid states are

OPEN

Tag can be modified, payloads and iovs can be created and deleted. This is the default state for new or cloned globaltags and is not suitable for use in data analysis

Can be transitioned to TESTING, RUNNING

TESTING

Tag cannot be modified and is suitable for testing but can be reopened

Can be transitioned to VALIDATED, OPEN

VALIDATED

Tag cannot be modified and has been tested.

Can be transitioned to PUBLISHED, OPEN

PUBLISHED

Tag cannot be modified and is suitable for user analysis

Can only be transitioned to INVALID

RUNNING

Tag can only be modified by adding new runs, not modifying the payloads for existing runs.

INVALID:

Tag is invalid and should not be used for anything.

This state is end of life for a globaltag and cannot be transitioned to any other state.

Added in version release-04-00-00.

b2conditionsdb tag state [-h] [--debugging] [--help-full]
                         [--base-url BASE_URL] [--auth-token AUTH_TOKEN]
                         TAGNAME STATE

Required Arguments

TAGNAME

globaltag to be changed

STATE

new state for the globaltag

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

4.6.8. Sub-command b2conditionsdb upload#

This command allows uploading a local database which was created by basf2 to the central database. It assumes that the globaltag already exists so please create it before if necessary using ‘tag create’.

The command requires the tagname to upload the payloads to and a database.txt containing the payloads and their iovs. One can supply a directory where to look for the payloads, by default they are assumed to be in the same directory as the database text file.

b2conditionsdb upload [-h] [--debugging] [--help-full] [--base-url BASE_URL]
                      [--auth-token AUTH_TOKEN]
                      [--normalize | --normalize-name NORMALIZE_NAME]
                      [-j NPROCESS] [--retries RETRIES] [--ignore-existing]
                      TAGNAME PAYLOADSFILE

Required Arguments

TAGNAME

globaltag to use for iov creation

PAYLOADSFILE

Testing payload storage file containing list of iovs

options

--debugging

Enable debugging of http traffic

--base-url

URI for the base of the REST API, if not given a list of default locations is tried

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

--normalize

Normalize the payload files to have reproducible checksums. This option should only be used if the payload files were created with an older software version (before release-04)

--normalize-name

Set the file name in the root file metadata to the given value. This implicitly enables --normalize and should only be used if the payload files were created with an older software version (before release-04)

-j

Number of concurrent connections to use for database connection (default: 1)

--retries

Number of retries on connection problems (default: 3)

--ignore-existing

Don’t check if payloads or iovs already exist in database. Speeds up initialization as the list of existing payloads doesn’t need to be downloaded. Can be used on first upload but the script cannot resume an upload if this option is given

4.6.9. b2conditionsdb-extract: Extract payload for a run range and save as TTree#

Added in version release-03-00-00.

This tool reads the given conditions objects from the database and writes them to a TTree. Intra run dependencies are not supported.

For example the command

b2conditionsdb-extract --exp 5 --runs 1-3,12 BeamParameters

will create a root file with a Tree containing the beam parameters of the runs 1, 2, 3, and 12 of experiment 5. The branch BeamParameters_valid contains 1 if the beam parameters are available for the given run and 0 if not. The following command plots the x position of the beam spot after opening the conditions.root file in root:

conditions->Draw("BeamParameters.m_vertex.X():run", "BeamParameters_valid", "*")

usage: b2conditionsdb-extract [-h] [--output OUTPUT] [--tag TAG] --exp EXP
                              --runs RUNS
                              condition [condition ...]

Required Arguments

condition

payload identifier, if the type is not equal to the name it has to be appended after a /

options

--output, -o

output file containing the TTree of conditions

Default: “conditions.root”

--tag, -t

global tag

--exp, -e

experiment number

--runs

run numbers, ranges are given as X-Y, multiple runs or ranges are separated by comma

4.6.10. b2conditionsdb-recommend: Recommend a global tag to analyse a given file#

Added in version release-03-00-00.

This tool determines the recommended set of global tags.

usage: b2conditionsdb-recommend [-h] [--oneline] [input_file]

Required Arguments

input_file

Input file that should be processed. No input file means MC generation.

options

--oneline

Prints the recommended global tags as a one-liner string.

4.6.11. b2conditionsdb-request: Request inclusion of payloads in official global tag#

Added in version release-03-00-00.

Tool to request an update of a global tag.

The tool will ask for the information required by the global tag manager for the update or take them from the command line options.

usage: b2conditionsdb-request [-h]
                              [--task {main,new,modify,validation,online,prompt,data,mc,analysis}]
                              [--normalize | --normalize-name NORMALIZE_NAME]
                              [--release RELEASE] [--reason REASON]
                              [--details DETAILS | --details-file DETAILS_FILE]
                              [--issue ISSUE] [--password PASSWORD | --oauth]
                              [--confirm] [--auth-token AUTH_TOKEN]
                              payloads

Required Arguments

payloads

Name of the testing payload storage file

options

--task

Possible choices: main, new, modify, validation, online, prompt, data, mc, analysis

Type of global tag that should be updated.

Use new/modify if the request contains new/modified payload classes.

--normalize

Normalize the payload files to have reproducible checksums. This option should only be used if the payload files were created with an older software version (before release-04)

--normalize-name

Set the file name in the root file metadata to the given value. This implicitly enables --normalize and should only be used if the payload files were created with an older software version (before release-04)

--release

Required release version

--reason

Reason for the request

--details

Detailed instructions/information for the global tag manager

--details-file

File containing detailed instructions/information for the global tag manager

--issue

Identifier of an existing jira issue for the request

--password

Password to authenticate the request

--oauth

Authenticate via oauth. This is should only be used for automated updates running within the DAQ network.

--confirm

Confirm that the given information is correct and that the payloads should be uploaded

--auth-token

JSON Web Token necessary for authenticating to the conditions database. Useful only for debugging, since by the default the tool automatically gets a token for you by asking the B2MMS username and password. If the environment variable $BELLE2_CDB_AUTH_TOKEN points to a file with a valid token, such token is used (useful for automatic workflows).

Contents