Belle II Software light-2406-ragdoll
LocalMetadataProvider Class Reference

Class to obtain metadata of all valid payloads from a local SQLite file instead of the central server. More...

#include <LocalMetadataProvider.h>

Inheritance diagram for LocalMetadataProvider:
Collaboration diagram for LocalMetadataProvider:

Public Types

using PayloadMap = std::unordered_map< std::string, PayloadMetadata >
 Type for the internal list of payloads.
 

Public Member Functions

 LocalMetadataProvider (std::string filename, const std::set< std::string > &usableTagStates)
 Construct with the name of the sqlite file.
 
 LocalMetadataProvider (const LocalMetadataProvider &)=delete
 copy constructor
 
 LocalMetadataProvider (LocalMetadataProvider &&)=delete
 move constructor
 
LocalMetadataProvideroperator= (const LocalMetadataProvider &)=delete
 assignment operator
 
LocalMetadataProvideroperator= (LocalMetadataProvider &&)=delete
 move assignment operator
 
virtual ~LocalMetadataProvider ()=default
 default destructor
 
std::string getGlobaltagStatus (const std::string &globaltag) override
 Return the status of the given globaltag.
 
bool updatePayloads (const std::string &globaltag, int exp, int run) override
 Update the list of existing payloads from a given globaltag, exp and run combination.
 
bool setTags (const std::vector< std::string > &tags)
 Set the list of globaltag names to be considered for payloads.
 
bool getPayloads (int exp, int run, std::vector< PayloadMetadata > &info)
 Update the information in the vector of metadata instances with the actual values.
 
std::set< std::string > getUsableTagStates ()
 Get the valid tag states when checking globaltag status.
 
void setUsableTagStates (const std::set< std::string > &states)
 Set the valid tag states for this provider when checking globaltag status.
 

Protected Member Functions

void addPayload (PayloadMetadata &&payload, const std::string &messagePrefix="")
 Add a payload information to the internal list.
 
void printInfoMessage (const std::string &provider)
 Print an INFO message about the used metadata provider.
 

Protected Attributes

std::vector< std::string > m_tags
 List of globaltags to consider.
 
std::unordered_map< std::string, PayloadMetadataCachem_cache
 Map of globaltag->known metadata.
 
PayloadMapm_payloads {nullptr}
 Map of known payloads for current conditions.
 
std::set< std::string > m_usableTagStates {"TESTING", "VALIDATED", "RUNNING", "PUBLISHED"}
 Set of global tag states to consider valid (except for 'INVALID' which is always considered invalid)
 

Private Attributes

std::string m_filename
 Filename for the sqlite database file.
 
sqlite::Connection m_connection
 SQLite connection object.
 
sqlite::SimpleStatement< std::string > m_globaltagStatus
 SQLite statement to return the status of a global tag.
 
sqlite::ObjectStatement< PayloadMetadata, std::string, std::string, std::string, std::string, std::string, int, int, int, int, int > m_selectPayloads
 SQLite statement to return all payloads for a given globaltag, exp, run.
 

Detailed Description

Class to obtain metadata of all valid payloads from a local SQLite file instead of the central server.

This is intended as a fallback in case the official server is not reachable or a way to download the conditions for use without internet connection.

In contrast to a LocalDatabase this class just provides local access to payload information previously downloaded from the central server. There's no method nor is it intended to put custom payloads for testing into such a downloaded file. It is always assumed that the content in these files is identical to the central database server

The requirements for the sqlite file are

  • there needs to be a table/view which allows to check for the global tag status
    SELECT globalTagStatus FROM globaltags WHERE globalTagName=?
  • there needs to be a table/view which allows to check for payloads like so
    SELECT
    payloadName, globalTagName, payloadUrl, baseUrl, checksum,
    firstExp, firstRun, finalExp, finalRun, revision
    FROM iov_payloads
    WHERE globalTagName=:globaltag AND firstExp<=:exp AND firstRun<=:run AND
    (finalExp<0 OR (finalRun<0 AND finalExp>=:exp) OR (finalExp>=:exp AND finalRun>=:run))

Definition at line 46 of file LocalMetadataProvider.h.

Member Typedef Documentation

◆ PayloadMap

using PayloadMap = std::unordered_map<std::string, PayloadMetadata>
inherited

Type for the internal list of payloads.

Definition at line 30 of file MetadataProvider.h.

Constructor & Destructor Documentation

◆ LocalMetadataProvider()

LocalMetadataProvider ( std::string  filename,
const std::set< std::string > &  usableTagStates 
)

Construct with the name of the sqlite file.

Definition at line 15 of file LocalMetadataProvider.cc.

15 :
16 MetadataProvider(usableTagStates), m_filename{std::move(filename)}, m_connection{m_filename},
17 m_globaltagStatus{m_connection, "SELECT globalTagStatus FROM globaltags WHERE globalTagName=:globaltag", true},
19 SELECT
20 payloadName, globalTagName, payloadUrl, baseUrl, checksum,
21 firstExp, firstRun, finalExp, finalRun, revision
22 FROM iov_payloads
23 WHERE globalTagName=:globaltag AND
24 ((firstExp==:exp AND firstRun<=:run) OR firstExp<:exp) AND
25 (finalExp<0 OR (finalRun<0 AND finalExp>=:exp) OR finalExp>:exp OR (finalExp==:exp AND finalRun>=:run));
26 )SQL", true}
27 {
29 }
sqlite::Connection m_connection
SQLite connection object.
sqlite::ObjectStatement< PayloadMetadata, std::string, std::string, std::string, std::string, std::string, int, int, int, int, int > m_selectPayloads
SQLite statement to return all payloads for a given globaltag, exp, run.
sqlite::SimpleStatement< std::string > m_globaltagStatus
SQLite statement to return the status of a global tag.
std::string m_filename
Filename for the sqlite database file.
MetadataProvider()=default
Default constructible.
void printInfoMessage(const std::string &provider)
Print an INFO message about the used metadata provider.

Member Function Documentation

◆ addPayload()

void addPayload ( PayloadMetadata &&  payload,
const std::string &  messagePrefix = "" 
)
protectedinherited

Add a payload information to the internal list.

This should be called by implementations during updatePayloads() for each payload found.

Parameters
payloadpayload information filled from the globaltag
messagePrefixa message prefix to be shown for possible log messages to indicate the correct metadata provider

Definition at line 91 of file MetadataProvider.cc.

92 {
93 const auto [it, inserted] = m_payloads->emplace(payload.name, payload);
94 if (!inserted) {
95 auto& existing = it->second;
96 if (existing.revision < payload.revision) {
97 std::swap(existing, payload);
98 }
99 B2DEBUG(36, messagePrefix << (messagePrefix.empty() ? "" : ": ") << "Found duplicate payload. Discarding one of them"
100 << LogVar("globaltag", existing.globaltag)
101 << LogVar("name", existing.name)
102 << LogVar("revision", existing.revision)
103 << LogVar("checksum", existing.checksum)
104 << LogVar("discarded revision", payload.revision));
105 } else {
106 B2DEBUG(37, messagePrefix << (messagePrefix.empty() ? "" : ": ") << "Found payload"
107 << LogVar("globaltag", payload.globaltag)
108 << LogVar("name", payload.name)
109 << LogVar("revision", payload.revision)
110 << LogVar("checksum", payload.checksum));
111 }
112 }
PayloadMap * m_payloads
Map of known payloads for current conditions.
Class to store variables with their name which were sent to the logging service.

◆ getGlobaltagStatus()

std::string getGlobaltagStatus ( const std::string &  globaltag)
overridevirtual

Return the status of the given globaltag.

Parameters
globaltagname of the globaltag
Returns
the string representation of the global tag status (like 'OPEN', 'PUBLISHED', 'INVALID') or an empty string if the tag does not exist.

Implements MetadataProvider.

Definition at line 31 of file LocalMetadataProvider.cc.

32 {
33 try {
34 if (m_globaltagStatus.execute(globaltag).step()) {
35 return m_globaltagStatus.getRow();
36 }
37 B2ERROR("Local Database: Global tag does not exist"
38 << LogVar("database", m_filename) << LogVar("globaltag", globaltag));
39 } catch (std::exception& e) {
40 B2ERROR("Local Database: Error checking globaltag"
41 << LogVar("database", m_filename) << LogVar("globaltag", globaltag) << LogVar("error", e.what()));
42 }
43 return "";
44 }

◆ getPayloads()

bool getPayloads ( int  exp,
int  run,
std::vector< PayloadMetadata > &  info 
)
inherited

Update the information in the vector of metadata instances with the actual values.

The input is a list of metadata instances and the metadata provider will try to modify them in place and add all the missing information it can find.

  • It will try to find iov, revision, checksum, url for each entry which doesn't have a valid revision number (revision=0)
  • It will ignore any entries with a valid revision number (revision>0)
  • any payload which cannot be found in any globaltag will raise an error as long as required=true
  • the modified list can still contain payloads without a valid revision some payloads could not be found.

This function will call updatePayloads() if necessary to update the list of known payloads for a given run but it will try to cache these results so incrementally asking for new payloads as they get requested is perfectly fine.

updatePayloads() will only called for globaltags if we still need to find payload information. If all payloads are found no further queries will be made so adding additional globaltags that are not used at the end is not expensive.

Parameters
expthe experiment number
runthe run number
[in,out]infoa list of metadata instances which we try to find the missing metadata for.
Returns
true if all required payloads could be found, false if at least one payload which still didn't have a valid revision number and is required could not be found.

Definition at line 36 of file MetadataProvider.cc.

37 {
38 if (info.empty()) return true;
39 for (const auto& tag : m_tags) {
40 // Check whether we already know the payloads for this exp,run ...
41 bool present{false};
42 std::tie(present, m_payloads) = m_cache[tag].get(exp, run);
43 // Apparently not, try to get them
44 if (!present && !updatePayloads(tag, exp, run)) {
45 // Error obtaining payloads means something is wrong (server/network/file problem).
46 // In this case we want to fall back to next metadata provider ...
47 throw std::runtime_error("Problem updating metadata");
48 }
49 // And once we got them let's update the info for all requested payloads
50 const auto& existing = *m_payloads;
51 // and check if we found all payloads we're looking for.
52 // Warning: **DO NOT** use `std::all_of` here which looks better but stops
53 // iteration on the first missing payload instead of going through all the
54 // remaining as well.
55 const size_t found = std::count_if(info.begin(), info.end(), [&existing](auto & payload) {
56 // already filled by previous gt or has a filename from testing payloads ... so don't do anything
57 if (payload.revision > 0 or !payload.filename.empty()) return true;
58 // otherwise look for the payload in the list of existing payloads for this run
59 if (auto&& it = existing.find(payload.name); it != existing.end()) {
60 payload.update(it->second);
61 B2DEBUG(35, "Found requested payload metadata"
62 << LogVar("globaltag", payload.globaltag)
63 << LogVar("name", payload.name)
64 << LogVar("revision", payload.revision)
65 << LogVar("checksum", payload.checksum));
66 return true;
67 }
68 // Not found, fine, not a big problem yet, there can be multiple global tags
69 return false;
70 });
71 // If we found all we don't need to check the next global tag at all ...
72 if (found == info.size()) break;
73 }
74 // Check that all payloads could be found (unless they're optional)
75 // Again, we don't use std::all_of here because then it stops early and we
76 // don't get errors for all missing payloads. But at least it would still be correct.
77 const int missing = std::count_if(info.begin(), info.end(), [this, exp, run](const auto & p) {
78 if (p.revision == 0 and p.filename.empty() and p.required) {
79 B2ERROR("Cannot find payload in any of the configured global tags"
80 << LogVar("name", p.name)
81 << LogVar("globaltags", boost::algorithm::join(m_tags, ", "))
82 << LogVar("experiment", exp) << LogVar("run", run));
83 return true;
84 }
85 return false;
86 });
87 // Only happy without missing payloads
88 return missing == 0;
89 }
std::unordered_map< std::string, PayloadMetadataCache > m_cache
Map of globaltag->known metadata.
std::vector< std::string > m_tags
List of globaltags to consider.
virtual bool updatePayloads(const std::string &globaltag, int exp, int run)=0
Update the list of existing payloads from a given globaltag, exp and run combination.

◆ getUsableTagStates()

std::set< std::string > getUsableTagStates ( )
inlineinherited

Get the valid tag states when checking globaltag status.

Definition at line 122 of file MetadataProvider.h.

122{ return m_usableTagStates; }
std::set< std::string > m_usableTagStates
Set of global tag states to consider valid (except for 'INVALID' which is always considered invalid)

◆ printInfoMessage()

void printInfoMessage ( const std::string &  provider = "")
protectedinherited

Print an INFO message about the used metadata provider.

Parameters
providerthe actual metadata provider used for the job

Definition at line 114 of file MetadataProvider.cc.

115 {
116 B2INFO("Conditions Database: found working metadata provider"
117 << LogVar("provider", provider));
118 }

◆ setTags()

bool setTags ( const std::vector< std::string > &  tags)
inherited

Set the list of globaltag names to be considered for payloads.

This should be called before requesting any payloads but after calling setValidTagStates().

Returns
true if all globaltags can be found and are in a valid state

Definition at line 15 of file MetadataProvider.cc.

16 {
17 m_tags = tags;
18 const size_t validTags = std::count_if(m_tags.begin(), m_tags.end(), [this](const auto & name) {
19 // dispatch obtaining the state to the implementation
20 auto status = getGlobaltagStatus(name);
21 // empty status: Unspecified error already dealt with
22 if (status.empty()) return false;
23 // otherwise check for valid states
24 if (m_usableTagStates.count(status) == 0) {
25 B2ERROR("The globaltag has a status which is not permitted for use. This is for your own protection"
26 << LogVar("globaltag", name)
27 << LogVar("status", status)
28 << LogVar("allowed states", boost::algorithm::join(m_usableTagStates, ", ")));
29 return false;
30 }
31 return true;
32 });
33 return validTags == tags.size();
34 }

◆ setUsableTagStates()

void setUsableTagStates ( const std::set< std::string > &  states)
inlineinherited

Set the valid tag states for this provider when checking globaltag status.

Should be called before setTags() if necessary.

Warning
The state "INVALID" will never be accepted and removed from the given set if present

Definition at line 128 of file MetadataProvider.h.

129 {
130 m_usableTagStates = states;
131 m_usableTagStates.erase("INVALID");
132 }

◆ updatePayloads()

bool updatePayloads ( const std::string &  globaltag,
int  exp,
int  run 
)
overridevirtual

Update the list of existing payloads from a given globaltag, exp and run combination.

Calls addPayload() for each payload it finds

Returns
true on success, false on failure

Implements MetadataProvider.

Definition at line 46 of file LocalMetadataProvider.cc.

47 {
48 try {
49 for (auto&& row : m_selectPayloads.execute(globaltag, exp, run)) {
50 addPayload(std::move(row), "Local Database");
51 }
52 } catch (std::exception& e) {
53 B2ERROR("Local Database: Error obtaining payloads"
54 << LogVar("database", m_filename) << LogVar("globaltag", globaltag) << LogVar("error", e.what()));
55 return false;
56 }
57 return true;
58 }
void addPayload(PayloadMetadata &&payload, const std::string &messagePrefix="")
Add a payload information to the internal list.

Member Data Documentation

◆ m_cache

std::unordered_map<std::string, PayloadMetadataCache> m_cache
protectedinherited

Map of globaltag->known metadata.

Definition at line 172 of file MetadataProvider.h.

◆ m_connection

sqlite::Connection m_connection
private

SQLite connection object.

Definition at line 78 of file LocalMetadataProvider.h.

◆ m_filename

std::string m_filename
private

Filename for the sqlite database file.

Definition at line 76 of file LocalMetadataProvider.h.

◆ m_globaltagStatus

sqlite::SimpleStatement<std::string> m_globaltagStatus
private

SQLite statement to return the status of a global tag.

Definition at line 80 of file LocalMetadataProvider.h.

◆ m_payloads

PayloadMap* m_payloads {nullptr}
protectedinherited

Map of known payloads for current conditions.

Definition at line 174 of file MetadataProvider.h.

◆ m_selectPayloads

sqlite::ObjectStatement<PayloadMetadata, std::string, std::string, std::string, std::string, std::string, int, int, int, int, int> m_selectPayloads
private

SQLite statement to return all payloads for a given globaltag, exp, run.

Definition at line 84 of file LocalMetadataProvider.h.

◆ m_tags

std::vector<std::string> m_tags
protectedinherited

List of globaltags to consider.

Definition at line 170 of file MetadataProvider.h.

◆ m_usableTagStates

std::set<std::string> m_usableTagStates {"TESTING", "VALIDATED", "RUNNING", "PUBLISHED"}
protectedinherited

Set of global tag states to consider valid (except for 'INVALID' which is always considered invalid)

Definition at line 176 of file MetadataProvider.h.


The documentation for this class was generated from the following files: