Class to enable configuration of the conditions database access in C++ and python. More...

#include <Configuration.h>

Public Member Functions
	Configuration ()
	Initialize default values.

void	reset ()
	Reset to default values.

void	setInitialized (bool value)
	Set by the Database singleton upon initialization and cleanup.

Configuration of globaltags
These members are responsible to configure the list of user globaltags which will be used in addition or instead of the base globaltags.
void	appendGlobalTag (const std::string &globalTag)
	Append a globaltag.

void	prependGlobalTag (const std::string &globalTag)
	prepend a globaltag

void	setGlobalTags (const std::vector< std::string > &list)
	Set the list of globaltags.

void	setGlobalTagsPy (const boost::python::list &globalTags)
	Set the list of globaltags from python.

std::vector< std::string >	getGlobalTags ()
	Get the list of user globaltags.

boost::python::list	getGlobalTagsPy ()
	Get the list of user globaltags as python version.

std::vector< std::string >	getDefaultGlobalTags () const
	Get the std::vector of default globaltags.

boost::python::tuple	getDefaultGlobalTagsPy () const
	Get the tuple of default globaltags as python version.

void	setInputGlobaltags (const std::vector< std::string > &inputTags)
	To be called by input modules with the tags to be added from input files.

void	setInputMetadata (const std::vector< FileMetaData > &inputMetadata)
	To be called by input modules with the list of all input FileMetaData.

std::vector< std::string >	getBaseTags () const
	Get the base globaltags to be used in addition to user globaltags.

void	overrideGlobalTags ()
	Enable globaltag override: If this is called once than overrideEnabled() will return true and getFinalListOfTags() will just return getGlobalTags()

void	overrideGlobalTagsPy (const boost::python::list &globalTags)
	Enable globaltag override and set the list of user globaltags in one go.

bool	overrideEnabled () const
	Check if override is enabled by previous calls to overrideGlobalTags()

void	disableGlobalTagReplay ()
	Disable global tag replay.

std::vector< std::string >	getFinalListOfTags ()
	Get the final list of globaltags to be used for processing.

Testing Payload Configuration These members control where to look
for temporary testing payloads. Each entry in the list of locations should be a filename of a textfile containing payload information. All payload files need to be in the same directory as the text file. Entries are highest priority first: Payloads found by earlier entries will take precedence over later entries. Payloads found in these text files. take precedence over payloads from globaltags. Warning This causes non reproducible results and is only for testing purposes.
void	appendTestingPayloadLocation (const std::string &filename)
	Add a local text file with testing payloads.

void	prependTestingPayloadLocation (const std::string &filename)
	Prepend a local text file with testing payloads to the list.

void	setTestingPayloadLocations (const std::vector< std::string > &list)
	Set the list of local text files to look for testing payloads.

void	setTestingPayloadLocationsPy (const boost::python::list &list)
	Set the list of text files containing test payloads in python.

std::vector< std::string >	getTestingPayloadLocations ()
	Get the list of testing payload locations.

boost::python::list	getTestingPayloadLocationsPy ()
	Get the list of text files containing test payloads in python.

Configure Metadata providers
These members are used to configure metadata providers: Where to look for payload information given the list of globaltags. Each entry in the list should be an URI or filename to a central REST server or a sqlite file containing a previously downloaded dump. The first entry in the list is used unless there is a problem reaching the server/reading the file in which case we use the next entry in the list as failover.
void	appendMetadataProvider (const std::string &provider)
	Append a metadata provider to the list.

void	prependMetadataProvider (const std::string &provider)
	Prepend a metadata provider to the list.

void	setMetadataProviders (const std::vector< std::string > &list)
	Set the list of metadata providers.

void	setMetadataProvidersPy (const boost::python::list &list)
	Set the list of metadata providers in python.

std::vector< std::string >	getMetadataProviders ()
	Get the list of metadata providers.

boost::python::list	getMetadataProvidersPy ()
	Get the list of metadata providers in python.

std::string	getDefaultRemoteMetadataProviderServer ()
	Get the default server URL for the remote metadata provider.

std::string	getDefaultLocalMetadataProviderPath ()
	Get the default local path for the local metadata provider.

Payload Location Configuration
Configure where to look for payload files. This should be a list of directories and or http(s) urls where the files containing the payload content can be found. Each location will be tried in turn and if the payload is not found in any of them we try to download it from the official server.
void	appendPayloadLocation (const std::string &location)
	Append a payload to the list of locations.

void	prependPayloadLocation (const std::string &location)
	Prepend a payload to the list of locations.

void	setPayloadLocations (const std::vector< std::string > &list)
	Set the list of payload locations.

void	setPayloadLocationsPy (const boost::python::list &list)
	Set the list of payload locations in python.

std::vector< std::string >	getPayloadLocations ()
	Get the list of payload locations.

boost::python::list	getPayloadLocationsPy ()
	Get the list og payload locations in python.

Expert Settings
These members are for changing some expert settings which should not be necessary for most users. These functions are exported to python using one `exper_settings()` function so no separate signatures for python are necessary.
void	setNewPayloadLocation (const std::string &filename)
	Set the file where to save newly created payload information.

std::string	getNewPayloadLocation () const
	Get the filename where to save newly created payload information.

void	setDownloadCacheDirectory (const std::string &directory)
	Set the directory where to place downloaded payloads.

std::string	getDownloadCacheDirectory () const
	Get the directory where to place downloaded payloads.

void	setDownloadLockTimeout (size_t timeout)
	Set the timeout we try to lock a file in the download cache directory for downloading.

size_t	getDownloadLockTimeout () const
	Get the timeout we try to lock a file in the download cache directory for downloading.

void	setUsableTagStates (const std::set< std::string > &states)
	Set the set of usable globaltag states to be allowed for processing.

const std::set< std::string > &	getUsableTagStates () const
	Get the set of usable globaltag states allowed to be used for processing.

void	setGlobaltagCallbackPy (const boost::python::object &obj)
	Set a callback function from python which will be called when processing starts and should return the final list of globaltags to be used.

Static Public Member Functions
static Configuration &	getInstance ()
	Get a reference to the instance which will be used when the Database is initialized.

static void	exposePythonAPI ()
	expose this class to python

Private Member Functions
void	ensureEditable () const
	Check whether the configuration object can be edited or if the database has been initialized already.

Static Private Member Functions
template<class T>
static void	fillFromEnv (T &target, const std::string &envName, const std::string &defaultValue)
	Fill a target object from a list of environment variables.

Private Attributes
bool	m_overrideEnabled {false}
	is the globaltag override enabled?

std::optional< std::vector< std::string > >	m_inputGlobaltags
	the list of globaltags from all the input files to be used in addition to the user globaltags

std::vector< FileMetaData >	m_inputMetadata
	the file metadata of all input files if globaltag replay is requested by input module

CppOrPyList	m_globalTags
	the list with all user globaltags

CppOrPyList	m_testingPayloadLocations
	the files with testing payloads to use during processing

CppOrPyList	m_metadataProviders
	the list with all the metadata providers

CppOrPyList	m_payloadLocations
	the list with all the payload locations

std::string	m_defaultRemoteMetadataProviderServer {"http://belle2db.sdcc.bnl.gov/b2s/rest/"}
	default server URL for the remote metadata provider

std::string	m_defaultLocalMetadataProviderPath {"/cvmfs/belle.cern.ch/conditions"}
	default local path for the local metadata provider

std::string	m_newPayloadFile {"localdb/database.txt"}
	the file to put the newly created payload information

std::string	m_downloadCacheDirectory {""}
	the directory to put newly downloaded payloads

size_t	m_downloadLockTimeout {120}
	the timeout when trying to lock files in the download directory

std::set< std::string >	m_usableTagStates {"TESTING", "VALIDATED", "PUBLISHED", "RUNNING"}
	the tag states accepted for processing

std::optional< boost::python::object >	m_callback
	the callback function to determine the final final list of globaltags

bool	m_databaseInitialized {false}
	bool indicating whether the database has been initialized, in which case any changes to the configuration object would not be reflected by the database setup so we refuse them

Detailed Description

Class to enable configuration of the conditions database access in C++ and python.

This class contains all settings relevant for the usage of the conditions database.

Mostly this class is just a container for user settings without much intelligence. The only thing it does is construct the final list of globaltags to be used from input data and user settings, maybe via a user callback, in getFinalListOfTags().

the exposePythonAPI() function contains detailed documentation to most of the members in this class for the python interface and details all of the settings.

Definition at line 61 of file Configuration.h.

Constructor & Destructor Documentation

◆ Configuration()

Configuration ( )

Initialize default values.

Definition at line 97 of file Configuration.cc.

  {
    // Backwards compatibility with the existing BELLE2_CONDB_GLOBALTAG
    // environment variable: If it is set disable replay
    if (EnvironmentVariables::isSet("BELLE2_CONDB_GLOBALTAG")) {
      fillFromEnv(m_globalTags, "BELLE2_CONDB_GLOBALTAG", "");
      overrideGlobalTags();
    }
    const std::string serverList = EnvironmentVariables::get("BELLE2_CONDB_SERVERLIST", "");
    // The list of the metadata providers we are going to query:
    const std::string metatadaProviders = serverList + " " + // First, the list of servers provided via env. variable
                                          m_defaultLocalMetadataProviderPath + "/database.sqlite" + " " +  // Then the default local provider (CVMFS)
                                          m_defaultRemoteMetadataProviderServer;  // And finally, the default remote provider (BNL)
    fillFromEnv(m_metadataProviders, "BELLE2_CONDB_METADATA", metatadaProviders);
    fillFromEnv(m_payloadLocations, "BELLE2_CONDB_PAYLOADS", m_defaultLocalMetadataProviderPath);
  }

Member Function Documentation

◆ appendGlobalTag()

void appendGlobalTag ( const std::string & globalTag )

inline

Append a globaltag.

Definition at line 78 of file Configuration.h.

78{ ensureEditable(); m_globalTags.append(globalTag); }

◆ appendMetadataProvider()

void appendMetadataProvider ( const std::string & provider )

inline

Append a metadata provider to the list.

Definition at line 222 of file Configuration.h.

222{ ensureEditable(); m_metadataProviders.append(provider); }

◆ appendPayloadLocation()

void appendPayloadLocation ( const std::string & location )

inline

Append a payload to the list of locations.

Definition at line 250 of file Configuration.h.

250{ ensureEditable(); m_payloadLocations.append(location); }

◆ appendTestingPayloadLocation()

void appendTestingPayloadLocation ( const std::string & filename )

inline

Add a local text file with testing payloads.

Definition at line 194 of file Configuration.h.

194{ ensureEditable(); m_testingPayloadLocations.append(filename); }

◆ disableGlobalTagReplay()

void disableGlobalTagReplay ( )

inline

Disable global tag replay.

This restores the previous behavior of just using the default globaltag and is just a shortcut for calling overrideGlobalTags() and appending each tag in getDefaultGlobalTags() tho the list of globaltags using appendGlobalTag()

Definition at line 152 of file Configuration.h.

    {
      overrideGlobalTags();
      for (const auto& tag : getDefaultGlobalTags()) appendGlobalTag(tag);
    }

◆ ensureEditable()

void ensureEditable ( ) const

inlineprivate

Check whether the configuration object can be edited or if the database has been initialized already.

Definition at line 316 of file Configuration.h.

    {
      if (m_databaseInitialized)
        throw std::runtime_error("Database already initialized, please reset before changing the configuration object");
    }

◆ exposePythonAPI()

void exposePythonAPI ( )

static

expose this class to python

Definition at line 376 of file Configuration.cc.

  {
    //don't show c++ signature in python doc to keep it simple
    py::docstring_options options(true, false, false);
 
    void (Configuration::*overrideGTFlag)() = &Configuration::overrideGlobalTags;
    void (Configuration::*overrideGTList)(const py::list&) = &Configuration::overrideGlobalTagsPy;
    py::object expert = raw_function(expertSettings);
    py::class_<Configuration>("ConditionsConfiguration", R"DOC(
This class contains all configurations for the conditions database service
 
* which globaltags to use
* where to look for payload information
* where to find the actual payload files
* which temporary testing payloads to use
 
But for most users the only thing they should need to care about is to set the
list of additional `globaltags` to use.
)DOC")
    .add_property("override_enabled", &Configuration::overrideEnabled, R"DOC(
Indicator whether or not the override of globaltags is enabled. If true then
globaltags present in input files  will be ignored and only the ones given in
`globaltags` will be considered.
)DOC")
    .def("reset", &Configuration::reset, R"DOC(reset()
 
Reset the conditions database configuration to its original state.
)DOC")
    .add_property("default_globaltags", &Configuration::getDefaultGlobalTagsPy, R"DOC(
A tuple containing the default globaltags to be used if events are generated without an input file.
)DOC")
    .add_property("globaltags", &Configuration::getGlobalTagsPy, &Configuration::setGlobalTagsPy, R"DOC(
List of globaltags to be used. These globaltags will be the ones with highest
priority but by default the globaltags used to create the input files or the
default globaltag will also be used.
 
The priority of the globaltags in this list is highest first. So the first in
the list will be checked first and all other globaltags will only be checked for
payloads not found so far.
 
Warning:
    By default this list contains the globaltags to be used **in addition** to
    the ones from the input file or the default one if no input file is present.
    If this is not desirable you need to call `override_globaltags()` to disable
    any addition or modification of this list.
)DOC")
    .def("append_globaltag", &Configuration::appendGlobalTag, py::args("name"), R"DOC(append_globaltag(name)
 
Append a globaltag to the end of the `globaltags` list. That means it will be
the lowest priority of all tags in the list.
)DOC")
    .def("prepend_globaltag", &Configuration::prependGlobalTag, py::args("name"), R"DOC(prepend_globaltag(name)
 
Add a globaltag to the beginning of the `globaltags` list. That means it will be
the highest priority of all tags in the list.
)DOC")
    .def("override_globaltags", overrideGTFlag)
    .def("override_globaltags", overrideGTList, py::args("globaltags"), R"DOC(override_globaltags(list=None)
 
Enable globaltag override. This disables all modification of the globaltag list at the beginning of processing:
 
* the default globaltag or the input file globaltags will be ignored.
* any callback set with `set_globaltag_callback` will be ignored.
* the list of `globaltags` will be used exactly as it is.
 
Parameters:
    list (list(str) or None) if given this list will replace the current content of `globaltags`
 
Warning:
    it's still possible to modify `globaltags` after this call.
)DOC")
    .def("disable_globaltag_replay", &Configuration::disableGlobalTagReplay, R"DOC(disable_globaltag_replay()
 
Disable global tag replay and revert to the old behavior that the default
globaltag will be used if no other globaltags are specified.
 
This is a shortcut to just calling
 
    >>> conditions.override_globaltags()
    >>> conditions.globaltags += list(conditions.default_globaltags)
 
)DOC")
    .def("append_testing_payloads", &Configuration::appendTestingPayloadLocation, py::args("filename"), R"DOC(append_testing_payloads(filename)
 
Append a text file containing local test payloads to the end of the list of
`testing_payloads`. This will mean they will have lower priority than payloads
in previously defined text files but still higher priority than globaltags.
 
Parameters:
    filename (str): file containing a local definition of payloads and their
        intervals of validity for testing
 
Warning:
    This functionality is strictly for testing purposes. Using local payloads
    leads to results which cannot be reproduced by anyone else and thus cannot
    be published.
)DOC")
    .def("prepend_testing_payloads", &Configuration::prependTestingPayloadLocation, py::args("filename"), R"DOC(prepend_testing_payloads(filename)
 
Insert a text file containing local test payloads in the beginning of the list
of `testing_payloads`. This will mean they will have higher priority than payloads in
previously defined text files as well as higher priority than globaltags.
 
Parameters:
    filename (str): file containing a local definition of payloads and their
        intervals of validity for testing
 
Warning:
    This functionality is strictly for testing purposes. Using local payloads
    leads to results which cannot be reproduced by anyone else and thus cannot
    be published.
)DOC")
    .add_property("testing_payloads", &Configuration::getTestingPayloadLocationsPy, &Configuration::setTestingPayloadLocationsPy, R"DOC(
List of text files to look for local testing payloads. Each entry should be a
text file containing local payloads and their intervals of validity to be used
for testing.
 
Payloads found in these files and valid for the current run will have a higher
priority than any of the `globaltags`. If a valid payload is present in multiple
files the first one in the list will have higher priority.
 
Warning:
    This functionality is strictly for testing purposes. Using local payloads
    leads to results which cannot be reproduced by anyone else and thus cannot
    be published.
)DOC")
    .add_property("metadata_providers", &Configuration::getMetadataProvidersPy, &Configuration::setMetadataProvidersPy, R"DOC(
List of metadata providers to use when looking for payload metadata. There are currently two supported providers:
 
1. Central metadata provider to look for payloads in the central conditions database.
   This provider is used for any entry in this list which starts with ``http(s)://``.
   The URL should point to the top level of the REST api endpoints on the server
 
2. Local metadata provider to look for payloads in a local SQLite snapshot taken
   from the central server. This provider will be assumed for any entry in this
   list not starting with a protocol specifier or if the protocol is given as ``file://``
 
This list should rarely need to be changed. The only exception is for users who
want to be able to use the software without internet connection after they
downloaded a snapshot of the necessary globaltags with ``b2conditionsdb download``
to point to this location.
)DOC")
    .add_property("default_metadata_provider_server", &Configuration::getDefaultRemoteMetadataProviderServer, R"DOC(
URL of the default central metadata provider to look for payloads in the
conditions database.
)DOC")
    .add_property("payload_locations", &Configuration::getPayloadLocationsPy, &Configuration::setPayloadLocationsPy, R"DOC(
List of payload locations to search for payloads which have been found by any of
the configured `metadata_providers`. This can be a local directory or a
``http(s)://`` url pointing to the payload directory on a server.
 
For remote locations starting with ``http(s)://`` we assume that the layout of
the payloads on the server is the same as on the main payload server:
The combination of given location and the relative url in the payload metadata
field ``payloadUrl`` should point to the correct payload on the server.
 
For local directories, two layouts are supported and will be auto detected:
 
flat
    All payloads are in the same directory without any substructure with the name
    ``dbstore_{name}_rev_{revision}.root``
hashed
    All payloads are stored in subdirectories in the form ``AB/{name}_r{revision}.root``
    where ``A`` and ``B`` are the first two characters of the md5 checksum of the
    payload file.
 
Example:
  Given ``payload_locations = ["payload_dir/", "http://server.com/payloads"]``
  the framework would look for a payload with name ``BeamParameters`` in revision
  ``45`` (and checksum ``a34ce5...``) in the following places
 
 
  1. ``payload_dir/a3/BeamParameters_r45.root``
  2. ``payload_dir/dbstore_BeamParameters_rev_45.root``
  3. ``http://server.com/payloads/dbstore/BeamParameters/dbstore_BeamParameters_rev_45.root``
     given the usual pattern of the ``payloadUrl`` metadata. But this could be
     changed on the central servers so mirrors should not depend on this convention
     but copy the actual structure of the central server.
 
If the payload cannot be found in any of the given locations the framework will
always attempt to download it directly from the central server and put it in a
local cache directory.
)DOC")
    .def("expert_settings", expert, R"DOC(expert_settings(**kwargs)
 
Set some additional settings for the conditions database.
 
You can supply any combination of keyword-only arguments defined below. The
function will return a dictionary containing all current settings.
 
    >>> conditions.expert_settings(connection_timeout=5, max_retries=1)
    {'save_payloads': 'localdb/database.txt',
     'download_cache_location': '',
     'download_lock_timeout': 120,
     'usable_globaltag_states': {'PUBLISHED', 'RUNNING', 'TESTING', 'VALIDATED'},
     'connection_timeout': 5,
     'stalled_timeout': 60,
     'max_retries': 1,
     'backoff_factor': 5}

◆ fillFromEnv()

template<class T>

static void fillFromEnv	(	T &	target,
		const std::string &	envName,
		const std::string &	defaultValue )

inlinestaticprivate

Fill a target object from a list of environment variables.

Definition at line 310 of file Configuration.h.

    {
      const auto values = EnvironmentVariables::getOrCreateList(envName, defaultValue);
      for (const std::string& v : values) target.append(v);
    }

◆ getBaseTags()

std::vector< std::string > getBaseTags ( ) const

Get the base globaltags to be used in addition to user globaltags.

If no input file is present, the result of getBaseTags() is identical to the list returned by getDefaultGlobalTags()
If one or more input files are present, the input module is responsible to call setInputGlobalTags(). If setInputGlobalTags() has been called then getBaseTags() returns the list of globaltags by splitting the value passed from the input modules at the commas.

Definition at line 214 of file Configuration.cc.

  {
    // return the list of base tags to be used: Either the default tag
    // or the list of globaltags from the input files
    if (not m_inputGlobaltags) return getDefaultGlobalTags();
    return *m_inputGlobaltags;
  }

◆ getDefaultGlobalTags()

std::vector< std::string > getDefaultGlobalTags ( ) const

Get the std::vector of default globaltags.

Definition at line 122 of file Configuration.cc.

  {
    // currently the default globaltag can be overwritten by environment variable
    // so keep that
    return EnvironmentVariables::getOrCreateList("BELLE2_CONDB_GLOBALTAG", CURRENT_DEFAULT_TAG);
  }

◆ getDefaultGlobalTagsPy()

py::tuple getDefaultGlobalTagsPy ( ) const

Get the tuple of default globaltags as python version.

Definition at line 129 of file Configuration.cc.

  {
    // same as above but as a python tuple ...
    py::list list;
    fillFromEnv(list, "BELLE2_CONDB_GLOBALTAG", CURRENT_DEFAULT_TAG);
    return py::tuple(list);
  }

◆ getDefaultLocalMetadataProviderPath()

std::string getDefaultLocalMetadataProviderPath ( )

inline

Get the default local path for the local metadata provider.

Definition at line 236 of file Configuration.h.

236{ return m_defaultLocalMetadataProviderPath; }

◆ getDefaultRemoteMetadataProviderServer()

std::string getDefaultRemoteMetadataProviderServer ( )

inline

Get the default server URL for the remote metadata provider.

Definition at line 234 of file Configuration.h.

234{ return m_defaultRemoteMetadataProviderServer; }

◆ getDownloadCacheDirectory()

std::string getDownloadCacheDirectory ( ) const

inline

Get the directory where to place downloaded payloads.

Empty string is shorthand to put them in a folder basf2-conditions in the temp dir

Definition at line 284 of file Configuration.h.

284{ return m_downloadCacheDirectory; }

◆ getDownloadLockTimeout()

size_t getDownloadLockTimeout ( ) const

inline

Get the timeout we try to lock a file in the download cache directory for downloading.

Definition at line 289 of file Configuration.h.

289{ return m_downloadLockTimeout; }

◆ getFinalListOfTags()

std::vector< std::string > getFinalListOfTags ( )

Get the final list of globaltags to be used for processing.

if overrideEnabled() is false then processing will use getGlobalTags() and getBaseTags() (in this order) as the list of globaltags to be used for processing. if getBaseTags() is empty an error will be raised and processing will be aborted.
if overrideEnabled() is true (by calling overrideGlobalTags()) then only the list from getGlobaltags() will be used for processing.

See also: setGlobaltagCallbackPy() to set a callback function from python to further customize this behavior

Warning: This function is only to be called by the conditions database service when processing starts. Calling this function from user code might abort processing if called at the wrong time or without properly initializing the settings.

Definition at line 222 of file Configuration.cc.

  {
    if (m_overrideEnabled) {
      B2INFO("Global tag override is in effect: input globaltags and default globaltag will be ignored");
      return m_globalTags.ensureCpp();
    }
 
    auto baseList = getBaseTags();
    if (m_callback) {
      // Create a dictionary of keyword arguments for the callback
      py::dict arguments;
      // we want a python list of the base tags
      {
        py::list baseListPy;
        for (const auto& tag : baseList) baseListPy.append(tag);
        arguments["base_tags"] = baseListPy;
      }
      // and set the user tags from our list.
      arguments["user_tags"] = m_globalTags.ensurePy();
      // and prepare list of metadata. It's None when no replay has been
      // requested which should mean that we generate events
      arguments["metadata"] = py::object();
      // otherwise it's a list of file metadata instances
      if (m_inputGlobaltags) {
        py::list metaDataList;
        for (const auto& m : m_inputMetadata) metaDataList.append(createROOTObjectPyCopy(m));
        arguments["metadata"] = metaDataList;
      }
      // arguments ready, call callback function, python will handle the exceptions
      py::object retval = (*m_callback)(*py::tuple(), **arguments);
      // If the return value is not None it should be an iterable
      // containing the final tag list
      if (retval != py::object()) {
        return extractStringList(retval);
      }
      // callback returned None so fall back to default
    }
    // Default tag replay ... bail if list of globaltags is empty
    if (baseList.empty()) {
      if (m_inputGlobaltags) {
        B2FATAL(R"(No baseline globaltags available.
    The input files you selected don't have compatible globaltags or an empty
    globaltag setting. As such globaltag configuration cannot be determined
    automatically.
 
    If you really sure that it is a good idea to process these files together
    you have to manually override the list of globaltags:
 
    >>> basf2.conditions.override_globaltags()
)");
      }else{
        B2FATAL(R"(No default globaltags available.
    There is no default globaltag available for processing. This usually means
    you set the environment variable BELLE2_CONDB_GLOBALTAG to an empty value.
 
    As this is unlikely to work for even the most basic functionality this is not
    directly supported anymore. If you really want to disable any access to the
    conditions database please configure this explicitly
 
    >>> basf2.conditions.metadata_providers = []
    >>> basf2.conditions.override_globaltags([])
)");
      }
    }
    // We have base tags and possibly user tags, so return both
    std::vector finalList = m_globalTags.ensureCpp();
    for (const auto& tag : baseList) { finalList.emplace_back(tag); }
    return finalList;
  }

◆ getGlobalTags()

std::vector< std::string > getGlobalTags ( )

inline

Get the list of user globaltags.

Definition at line 86 of file Configuration.h.

86{ return m_globalTags.ensureCpp(); }

◆ getGlobalTagsPy()

boost::python::list getGlobalTagsPy ( )

inline

Get the list of user globaltags as python version.

Definition at line 88 of file Configuration.h.

88{ return m_globalTags.ensurePy(); }

◆ getInstance()

Configuration & getInstance ( )

static

Get a reference to the instance which will be used when the Database is initialized.

Definition at line 91 of file Configuration.cc.

  {
    static Configuration instance;
    return instance;
  }

◆ getMetadataProviders()

std::vector< std::string > getMetadataProviders ( )

inline

Get the list of metadata providers.

Definition at line 230 of file Configuration.h.

230{ return m_metadataProviders.ensureCpp(); }

◆ getMetadataProvidersPy()

boost::python::list getMetadataProvidersPy ( )

inline

Get the list of metadata providers in python.

Definition at line 232 of file Configuration.h.

232{ return m_metadataProviders.ensurePy(); }

◆ getNewPayloadLocation()

std::string getNewPayloadLocation ( ) const

inline

Get the filename where to save newly created payload information.

Definition at line 277 of file Configuration.h.

277{ return m_newPayloadFile; }

◆ getPayloadLocations()

std::vector< std::string > getPayloadLocations ( )

inline

Get the list of payload locations.

Definition at line 258 of file Configuration.h.

258{ return m_payloadLocations.ensureCpp(); }

◆ getPayloadLocationsPy()

boost::python::list getPayloadLocationsPy ( )

inline

Get the list og payload locations in python.

Definition at line 260 of file Configuration.h.

260{ return m_payloadLocations.ensurePy(); }

◆ getTestingPayloadLocations()

std::vector< std::string > getTestingPayloadLocations ( )

inline

Get the list of testing payload locations.

Definition at line 202 of file Configuration.h.

202{ return m_testingPayloadLocations.ensureCpp(); }

◆ getTestingPayloadLocationsPy()

boost::python::list getTestingPayloadLocationsPy ( )

inline

Get the list of text files containing test payloads in python.

Definition at line 204 of file Configuration.h.

204{ return m_testingPayloadLocations.ensurePy(); }

◆ getUsableTagStates()

const std::set< std::string > & getUsableTagStates ( ) const

inline

Get the set of usable globaltag states allowed to be used for processing.

Definition at line 295 of file Configuration.h.

295{ return m_usableTagStates; }

◆ overrideEnabled()

bool overrideEnabled ( ) const

inline

Check if override is enabled by previous calls to overrideGlobalTags()

Definition at line 143 of file Configuration.h.

143{ return m_overrideEnabled; }

◆ overrideGlobalTags()

void overrideGlobalTags ( )

inline

Enable globaltag override: If this is called once than overrideEnabled() will return true and getFinalListOfTags() will just return getGlobalTags()

Definition at line 139 of file Configuration.h.

139{ ensureEditable(); m_overrideEnabled = true; }

◆ overrideGlobalTagsPy()

void overrideGlobalTagsPy ( const boost::python::list & globalTags )

Enable globaltag override and set the list of user globaltags in one go.

Definition at line 370 of file Configuration.cc.

  {
    setGlobalTagsPy(globalTags);
    m_overrideEnabled = true;
  }

◆ prependGlobalTag()

void prependGlobalTag ( const std::string & globalTag )

inline

prepend a globaltag

Definition at line 80 of file Configuration.h.

80{ ensureEditable(); m_globalTags.prepend(globalTag); }

◆ prependMetadataProvider()

void prependMetadataProvider ( const std::string & provider )

inline

Prepend a metadata provider to the list.

Definition at line 224 of file Configuration.h.

224{ ensureEditable(); m_metadataProviders.prepend(provider); }

◆ prependPayloadLocation()

void prependPayloadLocation ( const std::string & location )

inline

Prepend a payload to the list of locations.

Definition at line 252 of file Configuration.h.

252{ ensureEditable(); m_payloadLocations.prepend(location); }

◆ prependTestingPayloadLocation()

void prependTestingPayloadLocation ( const std::string & filename )

inline

Prepend a local text file with testing payloads to the list.

Definition at line 196 of file Configuration.h.

196{ ensureEditable(); m_testingPayloadLocations.prepend(filename); }

◆ reset()

void reset ( )

Reset to default values.

Definition at line 114 of file Configuration.cc.

  {
    if (m_databaseInitialized) {
      Database::Instance().reset(true);
    }
    *this = Configuration();
  }

◆ setDownloadCacheDirectory()

void setDownloadCacheDirectory ( const std::string & directory )

inline

Set the directory where to place downloaded payloads.

Empty string is shorthand to put them in a folder basf2-conditions in the temp dir

Definition at line 281 of file Configuration.h.

281{ ensureEditable(); m_downloadCacheDirectory = directory; }

◆ setDownloadLockTimeout()

void setDownloadLockTimeout ( size_t timeout )

inline

Set the timeout we try to lock a file in the download cache directory for downloading.

Definition at line 287 of file Configuration.h.

287{ ensureEditable(); m_downloadLockTimeout = timeout; }

◆ setGlobaltagCallbackPy()

void setGlobaltagCallbackPy ( const boost::python::object & obj )

inline

Set a callback function from python which will be called when processing starts and should return the final list of globaltags to be used.

See the python documentation for more details

Definition at line 300 of file Configuration.h.

300{ ensureEditable(); m_callback = obj; }

◆ setGlobalTags()

void setGlobalTags ( const std::vector< std::string > & list )

inline

Set the list of globaltags.

Definition at line 82 of file Configuration.h.

82{ ensureEditable(); m_globalTags.ensureCpp() = list; }

◆ setGlobalTagsPy()

void setGlobalTagsPy ( const boost::python::list & globalTags )

inline

Set the list of globaltags from python.

Definition at line 84 of file Configuration.h.

84{ ensureEditable(); m_globalTags.shallowCopy(globalTags); }

◆ setInitialized()

void setInitialized ( bool value )

inline

Set by the Database singleton upon initialization and cleanup.

Definition at line 305 of file Configuration.h.

305{ m_databaseInitialized = value; }

◆ setInputGlobaltags()

void setInputGlobaltags ( const std::vector< std::string > & inputTags )

inline

To be called by input modules with the tags to be added from input files.

This method is to be called by input modules which know the list of globaltags to be used but cannot provide FileMetaData instances for these input files.

In case an empty list is put in this means global tag replay is not possible for these input files and the user will need to manually supply all tags and use the override flag to disable any attempt at tag replay.

See also: setInputMetadata()

Definition at line 108 of file Configuration.h.

    {
      ensureEditable();
      m_inputGlobaltags = inputTags;
    }

◆ setInputMetadata()

void setInputMetadata ( const std::vector< FileMetaData > & inputMetadata )

To be called by input modules with the list of all input FileMetaData.

This method is to be called by input modules which have a full list of FileMetaData instances for all input files. It will then check if all instances have a compatible globaltag setting and call setInputGlobalTags accordingly.

See also: setInputGlobaltags()

Definition at line 137 of file Configuration.cc.

  {
    ensureEditable();
    m_inputMetadata = inputMetadata;
    // make sure the list of globaltags to be used is created but empty
    m_inputGlobaltags.emplace();
    // now check for compatibility: make sure all metadata have the same globaltag
    // setting. Unless we don't have metadata ...
    if (inputMetadata.empty()) return;
 
    std::optional<std::string> inputGlobaltags;
    for (const auto& metadata : inputMetadata) {
      if (!inputGlobaltags) {
        inputGlobaltags = metadata.getDatabaseGlobalTag();
      } else {
        if (inputGlobaltags != metadata.getDatabaseGlobalTag()) {
          B2WARNING("Input files metadata contain incompatible globaltag settings, globaltag replay not possible");
          // no need to set anything
          return;
        }
      }
    }
    // if it's still set and empty we have an empty input list ... warn specifically.
    if (inputGlobaltags and inputGlobaltags->empty()) {
      B2WARNING("Input files metadata all have empty globaltag setting, globaltag replay not possible");
      return;
    }
    // set the list of globaltags from the string containing the globaltags
    boost::split(*m_inputGlobaltags, *inputGlobaltags, boost::is_any_of(","));
 
    // HACK: So, we successfully set the input globaltags from the input file,
    // however we also decided that we want to add new payloads for
    // boost, invariant mass, beam spot, collision axis in CMS.
    // So if the release is older than when these features were introduced
    // or if files were produced before specific date, extra GTs are appended.
    // The appended GTs contain only the possible missing info and are added
    // with lowest priority.
    // If the files actually had all payloads these legacy payloads will never
    // be used as they have lowest priority.
    // Otherwise this should enable running over old files.
    //
    // TODO: Once we're sure all files being used contain all payloads remove this.
 
    std::optional<std::string> relMin, dateMin;
 
    for (const auto& metadata : inputMetadata) {
      // get oldest release
      std::string rel = metadata.getRelease().substr(0, 10);
      if (std::regex_match(rel, std::regex("release-[0-9][0-9]"))) {
        if (!relMin) relMin = rel;
        relMin = min(*relMin, rel);
      }
 
      // get oldest production date
      std::string date = metadata.getDate().substr(0, 10);
      if (std::regex_match(date, std::regex("[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"))) {
        if (!dateMin) dateMin = date;
        dateMin = min(*dateMin, date);
      }
    }
 
    // add IP GT if rel older than rel04 or for old files
    if ((relMin && relMin < "release-04") ||
        (!relMin && (!dateMin || dateMin < "2019-12-31"))) {
      B2DEBUG(30, "Enabling legacy IP information globaltag in tag replay");
      m_inputGlobaltags->emplace_back("Legacy_IP_Information");
    }
 
    // add CollisionAxisCMS GT if rel older than rel08 or for old files
    if ((relMin && relMin < "release-08") ||
        (!relMin && (!dateMin || dateMin < "2023-08-31"))) {
      B2DEBUG(30, "Enabling legacy CollsionAxisCMS globaltag in tag replay");
      m_inputGlobaltags->emplace_back("Legacy_CollisionAxisCMS");
    }
    // END TODO/HACK
  }

◆ setMetadataProviders()

void setMetadataProviders ( const std::vector< std::string > & list )

inline

Set the list of metadata providers.

Definition at line 226 of file Configuration.h.

226{ ensureEditable(); m_metadataProviders.ensureCpp() = list; }

◆ setMetadataProvidersPy()

void setMetadataProvidersPy ( const boost::python::list & list )

inline

Set the list of metadata providers in python.

Definition at line 228 of file Configuration.h.

228{ ensureEditable(); m_metadataProviders.shallowCopy(list); }

◆ setNewPayloadLocation()

void setNewPayloadLocation ( const std::string & filename )

inline

Set the file where to save newly created payload information.

Definition at line 275 of file Configuration.h.

275{ ensureEditable(); m_newPayloadFile = filename; }

◆ setPayloadLocations()

void setPayloadLocations ( const std::vector< std::string > & list )

inline

Set the list of payload locations.

Definition at line 254 of file Configuration.h.

254{ ensureEditable(); m_payloadLocations.ensureCpp() = list; }

◆ setPayloadLocationsPy()

void setPayloadLocationsPy ( const boost::python::list & list )

inline

Set the list of payload locations in python.

Definition at line 256 of file Configuration.h.

256{ ensureEditable(); m_payloadLocations.shallowCopy(list); }

◆ setTestingPayloadLocations()

void setTestingPayloadLocations ( const std::vector< std::string > & list )

inline

Set the list of local text files to look for testing payloads.

Definition at line 198 of file Configuration.h.

198{ ensureEditable(); m_testingPayloadLocations.ensureCpp() = list;}

◆ setTestingPayloadLocationsPy()

void setTestingPayloadLocationsPy ( const boost::python::list & list )

inline

Set the list of text files containing test payloads in python.

Definition at line 200 of file Configuration.h.

200{ ensureEditable(); m_testingPayloadLocations.shallowCopy(list); }

◆ setUsableTagStates()

void setUsableTagStates ( const std::set< std::string > & states )

inline

Set the set of usable globaltag states to be allowed for processing.

The state INVALID will always be ignored and not permitted

Definition at line 293 of file Configuration.h.

293{ ensureEditable(); m_usableTagStates = states; }

Member Data Documentation

◆ m_callback

std::optional<boost::python::object> m_callback

private

the callback function to determine the final final list of globaltags

Definition at line 349 of file Configuration.h.

◆ m_databaseInitialized

bool m_databaseInitialized {false}

private

bool indicating whether the database has been initialized, in which case any changes to the configuration object would not be reflected by the database setup so we refuse them

Definition at line 352 of file Configuration.h.

352{false};

◆ m_defaultLocalMetadataProviderPath

std::string m_defaultLocalMetadataProviderPath {"/cvmfs/belle.cern.ch/conditions"}

private

default local path for the local metadata provider

Definition at line 339 of file Configuration.h.

339{"/cvmfs/belle.cern.ch/conditions"};

◆ m_defaultRemoteMetadataProviderServer

std::string m_defaultRemoteMetadataProviderServer {"http://belle2db.sdcc.bnl.gov/b2s/rest/"}

private

default server URL for the remote metadata provider

Definition at line 337 of file Configuration.h.

337{"http://belle2db.sdcc.bnl.gov/b2s/rest/"};

◆ m_downloadCacheDirectory

std::string m_downloadCacheDirectory {""}

private

the directory to put newly downloaded payloads

Definition at line 343 of file Configuration.h.

343{""};

◆ m_downloadLockTimeout

size_t m_downloadLockTimeout {120}

private

the timeout when trying to lock files in the download directory

Definition at line 345 of file Configuration.h.

345{120};

◆ m_globalTags

CppOrPyList m_globalTags

private

the list with all user globaltags

Definition at line 329 of file Configuration.h.

◆ m_inputGlobaltags

std::optional<std::vector<std::string> > m_inputGlobaltags

private

the list of globaltags from all the input files to be used in addition to the user globaltags

Definition at line 325 of file Configuration.h.

◆ m_inputMetadata

std::vector<FileMetaData> m_inputMetadata

private

the file metadata of all input files if globaltag replay is requested by input module

Definition at line 327 of file Configuration.h.

◆ m_metadataProviders

CppOrPyList m_metadataProviders

private

the list with all the metadata providers

Definition at line 333 of file Configuration.h.

◆ m_newPayloadFile

std::string m_newPayloadFile {"localdb/database.txt"}

private

the file to put the newly created payload information

Definition at line 341 of file Configuration.h.

341{"localdb/database.txt"};

◆ m_overrideEnabled

bool m_overrideEnabled {false}

private

is the globaltag override enabled?

Definition at line 322 of file Configuration.h.

322{false};

◆ m_payloadLocations

CppOrPyList m_payloadLocations

private

the list with all the payload locations

Definition at line 335 of file Configuration.h.

◆ m_testingPayloadLocations

CppOrPyList m_testingPayloadLocations

private

the files with testing payloads to use during processing

Definition at line 331 of file Configuration.h.

◆ m_usableTagStates

std::set<std::string> m_usableTagStates {"TESTING", "VALIDATED", "PUBLISHED", "RUNNING"}

private

the tag states accepted for processing

Definition at line 347 of file Configuration.h.

347{"TESTING", "VALIDATED", "PUBLISHED", "RUNNING"};

The documentation for this class was generated from the following files:

framework/database/include/Configuration.h
framework/database/src/Configuration.cc

Public Member Functions

Static Public Member Functions

Private Member Functions

Static Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Configuration()

Member Function Documentation

◆ appendGlobalTag()

◆ appendMetadataProvider()

◆ appendPayloadLocation()

◆ appendTestingPayloadLocation()

◆ disableGlobalTagReplay()

◆ ensureEditable()

◆ exposePythonAPI()

◆ fillFromEnv()

◆ getBaseTags()

◆ getDefaultGlobalTags()

◆ getDefaultGlobalTagsPy()

◆ getDefaultLocalMetadataProviderPath()

◆ getDefaultRemoteMetadataProviderServer()

◆ getDownloadCacheDirectory()

◆ getDownloadLockTimeout()

◆ getFinalListOfTags()

◆ getGlobalTags()

◆ getGlobalTagsPy()

◆ getInstance()

◆ getMetadataProviders()

◆ getMetadataProvidersPy()

◆ getNewPayloadLocation()

◆ getPayloadLocations()

◆ getPayloadLocationsPy()

◆ getTestingPayloadLocations()

◆ getTestingPayloadLocationsPy()

◆ getUsableTagStates()

◆ overrideEnabled()

◆ overrideGlobalTags()

◆ overrideGlobalTagsPy()

◆ prependGlobalTag()

◆ prependMetadataProvider()

◆ prependPayloadLocation()

◆ prependTestingPayloadLocation()

◆ reset()

◆ setDownloadCacheDirectory()

◆ setDownloadLockTimeout()

◆ setGlobaltagCallbackPy()

◆ setGlobalTags()

◆ setGlobalTagsPy()

◆ setInitialized()

◆ setInputGlobaltags()

◆ setInputMetadata()

◆ setMetadataProviders()

◆ setMetadataProvidersPy()

◆ setNewPayloadLocation()

◆ setPayloadLocations()

◆ setPayloadLocationsPy()

◆ setTestingPayloadLocations()

◆ setTestingPayloadLocationsPy()

◆ setUsableTagStates()

Member Data Documentation

◆ m_callback

◆ m_databaseInitialized

◆ m_defaultLocalMetadataProviderPath

◆ m_defaultRemoteMetadataProviderServer

◆ m_downloadCacheDirectory

◆ m_downloadLockTimeout

◆ m_globalTags

◆ m_inputGlobaltags

◆ m_inputMetadata

◆ m_metadataProviders

◆ m_newPayloadFile

◆ m_overrideEnabled

◆ m_payloadLocations

◆ m_testingPayloadLocations

◆ m_usableTagStates