conditions_db
Contents
5.6.5. conditions_db#
Python interface to the ConditionsDB
- class conditions_db.BearerAuth(token)[source]#
Simple class to present bearer token instead of username/password
- class conditions_db.ConditionsDB(base_url=None, max_connections=10, retries=3)[source]#
Class to interface conditions db REST interface
- BASE_URLS = ['http://belle2db.sdcc.bnl.gov/b2s/rest/']#
base url to the conditions db to be used if no custom url is given
- check_payloads(payloads, information='payloadId')[source]#
Check for the existence of payloads in the database.
- Parameters
- Returns
A dictionary with the payload identifiers (name, checksum) as keys and the requested information as values for all payloads which are already present in the database.
- create_iov(globalTagId, payloadId, firstExp, firstRun, finalExp, finalRun)[source]#
Create an iov.
- Parameters
globalTagId (int) – id of the globaltag, obtain with get_globalTagId()
payloadId (int) – id of the payload, obtain from create_payload() or get_payloads()
firstExp (int) – first experiment for which this iov is valid
firstRun (int) – first run for which this iov is valid
finalExp (int) – final experiment for which this iov is valid
finalRun (int) – final run for which this iov is valid
- Returns
payloadIovId of the created iov, None if creation was not successful
- get_all_iovs(globalTag, exp=None, run=None, message=None, run_range=None, fully_contained=False)[source]#
Return list of all payloads in the given globaltag where each element is a
PayloadInformation
instance- Parameters
gobalTag (str) – name of the globaltag
exp (int) – if given limit the list of payloads to the ones valid for the given exp,run combination
run (int) – if given limit the list of payloads to the ones valid for the given exp,run combination
message (str) – additional message to show when downloading the payload information. Will be directly appended to “Obtaining lists of iovs for globaltag {globalTag}”
run_range (tuple) – if given limit the list of payloads to the ones overlapping with the given run range, if
fully_contained (bool) – if True and the run_range is not None it limits the list of payloads to the ones fully contained in the given run range
Warning
Both, exp and run, need to be given at the same time. Just supplying an experiment or a run number will not work
- static get_base_urls(given_url)[source]#
Resolve the list of server urls. If a url is given just return it. Otherwise return servers listed in BELLE2_CONDB_SERVERLIST or the builtin defaults
- Parameters
given_url (str) – Explicit base_url. If this is not None it will be returned as is in a list of length 1
- Returns
a list of urls to try for database connectivity
- get_globalTagInfo(name)[source]#
Get the id of the globaltag with the given name. Returns either the id or None if the tag was not found
- get_globalTagType(name)[source]#
Get the dictionary describing the given globaltag type (currently one of DEV or RELEASE). Returns None if tag type was not found.
- get_globalTags()[source]#
Get a list of all globaltags. Returns a dictionary with the globaltag names and the corresponding ids in the database
- get_iovs(globalTagName, payloadName=None)[source]#
Return existing iovs for a given tag name. It returns a dictionary which maps (payloadId, first runId, final runId) to iovId
- get_payloads(global_tag=None)[source]#
Get a list of all defined payloads (for the given global_tag or by default for all). Returns a dictionary which maps (module, checksum) to the payload id.
- get_revisions(entries)[source]#
Get the revision numbers of payloads in the database.
- Parameters
entries (list) – A list of payload entries. Each entry must have the attributes module and checksum.
- Returns
True if successful.
- modify_iov(iovId, firstExp, firstRun, finalExp, finalRun)[source]#
Modify the validity range of a given iov
- request(method, url, message=None, *args, **argk)[source]#
Request function, similar to requests.request but adding the base_url
- Parameters
All other arguments will be forwarded to requests.request.
- set_authentication_token(token)[source]#
Set authentication token when talking to the database
- Parameters
token (str) – JWT to hand to the database. Will not be checked for validity here.
- staging_request(filename, normalize, data, password)[source]#
Upload a testing payload storage to a staging globaltag and create or update a jira issue
- Parameters
filename (str) – filename of the testing payload storage file that should be uploaded
normalize (Union[bool, str]) – if True the payload root files will be normalized to have the same checksum for the same content, if normalize is a string in addition the file name in the root file metadata will be set to it
data (dict) –
a dictionary with the information provided by the user:
task: category of globaltag, either main, online, prompt, data, mc, or analysis
tag: the globaltag name
request: type of request, either Update, New, or Modification. The latter two imply task == main because if new payload classes are introduced or payload classes are modified then they will first be included in the main globaltag. Here a synchronization of code and payload changes has to be managed. If new or modified payload classes should be included in other globaltags they must already be in a release.
pull-request: number of the pull request containing new or modified payload classes, only for request == New or Modified
backward-compatibility: description of what happens if the old payload is encountered by the updated code, only for request == Modified
forward-compatibility: description of what happens if a new payload is encountered by the existing code, only for request == Modified
release: the required release version
reason: the reason for the request
description: a detailed description for the globaltag manager
issue: identifier of an existing jira issue (optional)
user: name of the user
time: time stamp of the request
password – the password for access to jira or the access token and secret for oauth access
- Returns
True if the upload and jira issue creation/upload was successful
- upload(filename, global_tag, normalize=False, ignore_existing=False, nprocess=1, uploaded_entries=None)[source]#
Upload a testing payload storage to the conditions database.
- Parameters
filename (str) – filename of the testing payload storage file that should be uploaded
global_tage (str) – name of the globaltag to which the data should be uploaded
normalize (Union[bool, str]) – if True the payload root files will be normalized to have the same checksum for the same content, if normalize is a string in addition the file name in the root file metadata will be set to it
ignore_existing (bool) – if True do not upload payloads that already exist
nprocess (int) – maximal number of parallel uploads
uploaded_entries (list) – the list of successfully uploaded entries
- Returns
True if the upload was successful
- class conditions_db.PayloadInformation(payload_id, name, revision, checksum, payload_url, base_url, iov_id=None, iov=None)[source]#
Small container class to help compare payload information for efficient comparison between globaltags
- base_url#
base url
- checksum#
checksum of the payload
- classmethod from_json(payload, iov=None)[source]#
Set all internal members from the json information of the payload and the iov.
- iov#
interval of validity
- iov_id#
iov id in CDB, not used for comparisons
- name#
name of the payload
- payload_id#
payload id in CDB, not used for comparisons
- payload_url#
payload url
- revision#
revision, not used for comparisons
- property url#
Return the full url to the payload on the server
- conditions_db.enable_debugging()[source]#
Enable verbose output of python-requests to be able to debug http connections
- conditions_db.get_cdb_authentication_token(path=None)[source]#
Helper function for correctly retrieving the CDB authentication token (either via file or via issuing server).
- Parameters
path – Path to a file containing a CDB authentication token; if None, the function will use a default path (
${HOME}/b2cdb_${BELLE2_USER}.token
or/tmp/b2cdb_${BELLE2_USER}.token
) to look for a token.
- conditions_db.require_database_for_test(timeout=60, base_url=None)[source]#
Make sure that the database is available and skip the test if not.
This function should be called in test scripts if they are expected to fail if the database is down. It either returns when the database is ok or it will signal test_basf2 that the test should be skipped and exit
- conditions_db.set_cdb_authentication_token(cdb_instance, auth_token=None)[source]#
Helper function for setting the CDB authentication token.
- Parameters
cdb_instance – An instance of the
ConditionsDB
class.auth_token – A CDB authentication token: if
None
, it is automatically retrieved from the issuing server and then set
5.6.6. conditions_db.iov#
This module contains classes to work with validity intervals. There’s a class
for a single interval, IntervalOfValidity
and a class to manage a set of
validities, IoVSet
, which can be used to manipulate iov ranges
- class conditions_db.iov.IntervalOfValidity(*iov)[source]#
Interval of validity class to support set operations like union and intersection.
An interval of validity is a set of runs for which something is valid. An IntervalOfValidity consists of a
first
valid run and afinal
valid run.Warning
The
final
run is inclusive so the the validity is including the final run.Each run is identified by a experiment number and a run number. Accessing
first
orfinal
will return a tuple(experiment, run)
but the elements can also be accessed separately withfirst_exp
,first_exp
,final_exp
andfinal_run
.For
final
there’s a special case where either the run or both, the run and the experiment number are infinite. This means the validity extends to all values. If only the run number is infinite then it’s valid for all further runs in this experiment. If both are infinite the validity extends to everything.For simplicity
-1
can be passed in instead of infinity when creating objects.- static always()[source]#
Return an iov that is valid everywhere
>>> IntervalOfValidity.always() (0, 0, inf, inf)
- property final#
Return the final valid experiment,run
- property final_exp#
Return the final valid experiment
- property final_run#
Return the final valid run
- property first#
Return the first valid experiment,run
- property first_exp#
Return the first valid experiment
- property first_run#
Return the first valid run
- intersect(other)[source]#
Intersection with another iov.
Will return None if the payloads don’t overlap
>>> iov1 = IntervalOfValidity(1,0,2,5) >>> iov2 = IntervalOfValidity(2,0,2,-1) >>> iov3 = IntervalOfValidity(2,10,5,-1) >>> iov1.intersect(iov2) (2, 0, 2, 5) >>> iov2.intersect(iov3) (2, 10, 2, inf) >>> iov3.intersect(iov1) is None True
- property is_open#
Check whether the iov is valid until infinity
- subtract(other)[source]#
Return a new iov with the validity of the other removed. Will return None if everything is removed.
Warning
If the other iov is in the middle of the validity we will return a tuple of two new iovs
>>> iov1 = IntervalOfValidity(0,0,10,-1) >>> iov2 = IntervalOfValidity(5,0,5,-1) >>> iov1 - iov2 ((0, 0, 4, inf), (6, 0, 10, inf))
- property tuple#
Return the iov as a tuple with experiment/run numbers replaced with -1
This is mostly helpful where infinity is not supported and is how the intervals are represented in the database.
>>> a = IntervalOfValidity.always() >>> a (0, 0, inf, inf) >>> a.tuple (0, 0, -1, -1)
- union(other, allow_startone=False)[source]#
Return the union with another iov.
>>> iov1 = IntervalOfValidity(1,0,1,-1) >>> iov2 = IntervalOfValidity(2,0,2,-1) >>> iov3 = IntervalOfValidity(2,10,5,-1) >>> iov1.union(iov2) (1, 0, 2, inf) >>> iov2.union(iov3) (2, 0, 5, inf) >>> iov3.union(iov1) is None True
Warning
This method will return None if the iovs don’t overlap or connect to each other as no union can be formed.
- Parameters
other (IntervalOfValidity) – IoV to calculate the union with
allow_startone (bool) –
If True we will consider run 0 and run 1 the first run in an experiment. This means that if one of the iovs has un unlimited final run it can be joined with the other iov if the experiment number increases and the iov starts at run 0 and 1. If this is False just run 0 is considered the next run.
>>> iov1 = IntervalOfValidity(0,0,0,-1) >>> iov2 = IntervalOfValidity(1,1,1,-1) >>> iov1.union(iov2, False) is None True >>> iov1.union(iov2, True) (0, 0, 1, inf)
- class conditions_db.iov.IoVSet(iterable=None, *, allow_overlaps=False, allow_startone=False)[source]#
A set of iovs.
This class allows to combine iovs into a set. New iovs can be added with
add()
and will be combined with existing iovs if possible.The final, minimal number of iovs can be obtained with the
iovs
property>>> a = IoVSet() >>> a.add((0,0,0,2)) >>> a.add((0,3,0,5)) >>> a.add((0,8,0,9)) >>> a {(0, 0, 0, 5), (0, 8, 0, 9)}
- add(iov, allow_overlaps=None)[source]#
Add a new iov to the set.
The new iov be combined with existing iovs if possible. After the operation the set will contain the minimal amount of separate iovs possible to represent all added iovs
>>> a = IoVSet() >>> a.add((0, 0, 0, 2)) >>> a.add((0, 3, 0, 5)) >>> a.add((0, 8, 0, 9)) >>> a {(0, 0, 0, 5), (0, 8, 0, 9)} >>> a.add(IoVSet([(10, 0, 10, 1), (10, 2, 10, -1)])) >>> a {(0, 0, 0, 5), (0, 8, 0, 9), (10, 0, 10, inf)}
Be aware, by default it’s not possible to add overlapping iovs to the set. This can be changed either on construction or per
add
call usingallow_overlap
>>> a.add((0, 2, 0, 3)) Traceback (most recent call last): ... ValueError: Overlap between (0, 0, 0, 5) and (0, 2, 0, 3) >>> a.add((0, 2, 0, 3), allow_overlaps=True) >>> a {(0, 0, 0, 5), (0, 8, 0, 9), (10, 0, 10, inf)}
- Parameters
iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to add to this set
allow_overlaps (bool) – Can be used to override global overlap setting of this set to allow/restrict overlaps for a single insertion operation
Warning
This method modifies the set in place
- contains(iov)[source]#
Check if an iov is fully covered by the set
>>> a = IoVSet([(0,0,2,-1), (5,0,5,-1)]) >>> a.contains((0,0,1,-1)) True >>> a.contains(IntervalOfValidity(0,0,3,2)) False >>> a.contains(IoVSet([(0,1,1,23), (5,0,5,23)])) True >>> a.contains(IoVSet([(0,1,1,23), (5,0,6,23)])) False >>> a.contains((3,0,4,-1)) False
- Parameters
iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to be checked
- Returns
True if the full iov or all the iovs in the given set are fully present in this set
- property final#
Return the final run covered by this iov set
>>> a = IoVSet([(3,0,3,10), (10,11,10,23), (0,0,2,-1), (5,0,5,-1)]) >>> a.final (10, 23)
- property first#
Return the first run covered by this iov set
>>> a = IoVSet([(3,0,3,10), (10,11,10,23), (0,0,2,-1), (5,0,5,-1)]) >>> a.first (0, 0)
- property gaps#
Return the gaps in the set. Any area not covered between the first point of validity and the last
>>> a = IoVSet([(0,0,2,-1)]) >>> a.gaps {} >>> b = IoVSet([(0,0,2,-1), (5,0,5,-1)]) >>> b.gaps {(3, 0, 4, inf)} >>> c = IoVSet([(0,0,2,-1), (5,0,5,-1), (10,3,10,6)]) >>> c.gaps {(3, 0, 4, inf), (6, 0, 10, 2)}
- intersect(iov)[source]#
Intersect this set with another set and return a new set which is valid exactly where both sets have been valid before
>>> a = IoVSet() >>> a.add((0,0,10,-1)) >>> a.intersect((5,0,20,-1)) {(5, 0, 10, inf)} >>> a.intersect(IoVSet([(0,0,3,-1), (9,0,20,-1)])) {(0, 0, 3, inf), (9, 0, 10, inf)}
- Parameters
iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to intersect with this set
- property iovs#
Return the set of valid iovs
- overlaps(iov)[source]#
Check if the given iov overlaps with this set.
In contrast to
contains
this doesn’t require the given iov to be fully covered. It’s enough if the any run covered by the iov is also covered by this set.>>> a = IoVSet([(0,0,2,-1), (5,0,5,-1)]) >>> a.overlaps((0,0,1,-1)) True >>> a.overlaps(IntervalOfValidity(0,0,3,2)) True >>> a.overlaps(IoVSet([(0,1,1,23), (5,0,5,23)])) True >>> a.overlaps(IoVSet([(0,1,1,23), (5,0,6,23)])) True >>> a.overlaps((3,0,4,-1)) False
- Parameters
iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to be checked
- Returns
True if the iov or any of the iovs in the given set overlap with any iov in this set
- remove(iov)[source]#
Remove an iov or a set of iovs from this set
After this operation the set will not be valid for the given iov or set of iovs:
>>> a = IoVSet() >>> a.add((0,0,10,-1)) >>> a.remove((1,0,1,-1)) >>> a.remove((5,0,8,5)) >>> a {(0, 0, 0, inf), (2, 0, 4, inf), (8, 6, 10, inf)} >>> a.remove(IoVSet([(3,0,3,10), (3,11,3,-1)])) >>> a {(0, 0, 0, inf), (2, 0, 2, inf), (4, 0, 4, inf), (8, 6, 10, inf)}
- Parameters
iov (Union[IoVSet, IntervalOfValidity, tuple(int)]) – IoV or set of IoVs to remove from this set
Warning
This method modifies the set in place