Class to obtain payload metadata from the HSF central server via REST requests. More...

#include <HSFCentralMetadataProvider.h>

Inheritance diagram for HSFCentralMetadataProvider:

Collaboration diagram for HSFCentralMetadataProvider:

[legend]

Public Types
using	PayloadMap = std::unordered_map<std::string, PayloadMetadata>
	Type for the internal list of payloads.

Public Member Functions
	HSFCentralMetadataProvider (const std::set< std::string > &usableTagStates)
	Create using the default HSF central base URL and given usable tag states.

	HSFCentralMetadataProvider (std::string baseUrl, const std::set< std::string > &usableTagStates)
	Create using a custom base URL for the HSF central server.

virtual	~HSFCentralMetadataProvider ()=default
	default destructor

std::string	getGlobaltagStatus (const std::string &name) override
	Check the status of a given globaltag.

bool	updatePayloads (const std::string &globaltag, int exp, int run) override
	Update the list of known payloads for the given globaltag/exp/run.

bool	setTags (const std::vector< std::string > &tags)
	Set the list of globaltag names to be considered for payloads.

bool	getPayloads (int exp, int run, std::vector< PayloadMetadata > &info)
	Update the information in the vector of metadata instances with the actual values.

std::set< std::string >	getUsableTagStates ()
	Get the valid tag states when checking globaltag status.

void	setUsableTagStates (const std::set< std::string > &states)
	Set the valid tag states for this provider when checking globaltag status.

Protected Member Functions
void	addPayload (PayloadMetadata &&payload, const std::string &messagePrefix="")
	Add a payload information to the internal list.

void	printInfoMessage (const std::string &provider)
	Print an INFO message about the used metadata provider.

Protected Attributes
std::vector< std::string >	m_tags
	List of globaltags to consider.

std::unordered_map< std::string, PayloadMetadataCache >	m_cache
	Map of globaltag->known metadata.

PayloadMap *	m_payloads {nullptr}
	Map of known payloads for current conditions.

std::set< std::string >	m_usableTagStates {"TESTING", "VALIDATED", "RUNNING", "PUBLISHED"}
	Set of global tag states to consider valid (except for 'INVALID' which is always considered invalid)

Private Member Functions
nlohmann::json	get (const std::string &url)
	Download a given relative url (the baseUrl will be prepended) and return the json description.

Private Attributes
Downloader &	m_downloader {Downloader::getDefaultInstance()}
	Reference to the downloader instance for convenience.

std::string	m_baseUrl
	base url of the server

std::string	m_payloadBaseUrl
	base url of the payload server

Static Private Attributes
static constexpr long long	m_maxIoV = 9223372036854775807
	Max IoV value, representing infinite end of validity.

Detailed Description

Class to obtain payload metadata from the HSF central server via REST requests.

This class connects to the Belle II HSF central server instead of the main Belle II conditions database server. It provides the same functionality as CentralMetadataProvider but uses a different base URL and a different API.

Definition at line 21 of file HSFCentralMetadataProvider.h.

Member Typedef Documentation

◆ PayloadMap

using PayloadMap = std::unordered_map<std::string, PayloadMetadata>

inherited

Type for the internal list of payloads.

Definition at line 30 of file MetadataProvider.h.

Constructor & Destructor Documentation

◆ HSFCentralMetadataProvider()

HSFCentralMetadataProvider	(	std::string	baseUrl,
		const std::set< std::string > &	usableTagStates )

Create using a custom base URL for the HSF central server.

Definition at line 19 of file HSFCentralMetadataProvider.cc.

                                                                                                                   :
    MetadataProvider(usableTagStates), m_baseUrl(std::move(baseUrl))
  {
    // We want to be sure on construction that the server is working. So let's
    // just check the list of valid states
    const auto result = get("/gtstatus");
    // check list of valid states
    auto validStates = getUsableTagStates();
    std::string invalidStates = "";
    for (const auto& info : result) {
      const std::string status = info.at("name");
      if (not validStates.erase(status)) {
        if (!invalidStates.empty()) invalidStates += ", ";
        invalidStates += status;
      }
    }
    B2DEBUG(31, "Infinite IoV value is set to " << m_maxIoV);
    printInfoMessage(m_baseUrl);
    B2DEBUG(31, "Conditions Database: unusable globaltag states: " << invalidStates);
    for (const auto& status : validStates) {
      B2DEBUG(31, "Conditions Database: status marked as usable for global tags is not known to the database"
              << LogVar("status", status));
 
    }
    const auto payloadUrl = get("user_settings/CDB_PAYLOAD_BASE_URL");
    m_payloadBaseUrl = payloadUrl.at("CDB_PAYLOAD_BASE_URL");
    B2DEBUG(31, "Fetched payload location" << LogVar("URL", m_payloadBaseUrl));
  }

Member Function Documentation

◆ addPayload()

void addPayload	(	PayloadMetadata &&	payload,
		const std::string &	messagePrefix = "" )

protectedinherited

Add a payload information to the internal list.

This should be called by implementations during updatePayloads() for each payload found.

Parameters

payload	payload information filled from the globaltag
messagePrefix	a message prefix to be shown for possible log messages to indicate the correct metadata provider

Definition at line 91 of file MetadataProvider.cc.

  {
    const auto [it, inserted] = m_payloads->emplace(payload.name, payload);
    if (!inserted) {
      auto& existing = it->second;
      if (existing.revision < payload.revision) {
        std::swap(existing, payload);
      }
      B2DEBUG(36, messagePrefix << (messagePrefix.empty() ? "" : ": ") << "Found duplicate payload. Discarding one of them"
              << LogVar("globaltag", existing.globaltag)
              << LogVar("name", existing.name)
              << LogVar("revision", existing.revision)
              << LogVar("checksum", existing.checksum)
              << LogVar("discarded revision", payload.revision));
    } else {
      B2DEBUG(37, messagePrefix << (messagePrefix.empty() ? "" : ": ") << "Found payload"
              << LogVar("globaltag", payload.globaltag)
              << LogVar("name", payload.name)
              << LogVar("revision", payload.revision)
              << LogVar("checksum", payload.checksum));
    }
  }

◆ get()

json get ( const std::string & url )

private

Download a given relative url (the baseUrl will be prepended) and return the json description.

Will raise exceptions on error

Definition at line 48 of file HSFCentralMetadataProvider.cc.

  {
    std::stringstream stream;
    const auto fullUrl = m_downloader.joinWithSlash(m_baseUrl, url);
    m_downloader.download(fullUrl, stream);
    stream.clear();
    stream.seekg(0, std::ios::beg);
    return json::parse(stream);
  }

◆ getGlobaltagStatus()

std::string getGlobaltagStatus ( const std::string & name )

overridevirtual

Check the status of a given globaltag.

Returns: the status of the globaltag (like "OPEN", "INVALID", "PUBLISHED"). Returns empty string on any errors or if the tag doesn't exist.

Implements MetadataProvider.

Definition at line 58 of file HSFCentralMetadataProvider.cc.

  {
    auto escaped = m_downloader.escapeString(name);
    const std::string url = "/globalTag/" + escaped;
    try {
      const auto gtinfo = get(url);
      return gtinfo.at("status").at("name");
    } catch (std::runtime_error& e) {
      B2WARNING("Conditions Database: Cannot download information on global tag. "
                "Either the database is overloaded or the selected globaltag doesn't exist"
                << LogVar("server url", m_baseUrl) << LogVar("globaltag", name) << LogVar("query", url));
    } catch (std::exception& e) {
      B2WARNING("Conditions Database: Problem determining global tag status"
                << LogVar("server url", m_baseUrl) << LogVar("globaltag", name) << LogVar("query", url) << LogVar("error", e.what()));
    }
    return "";
  }

◆ getPayloads()

bool getPayloads	(	int	exp,
		int	run,
		std::vector< PayloadMetadata > &	info )

inherited

Update the information in the vector of metadata instances with the actual values.

The input is a list of metadata instances and the metadata provider will try to modify them in place and add all the missing information it can find.

It will try to find iov, revision, checksum, url for each entry which doesn't have a valid revision number (revision=0)
It will ignore any entries with a valid revision number (revision>0)
any payload which cannot be found in any globaltag will raise an error as long as required=true
the modified list can still contain payloads without a valid revision some payloads could not be found.

This function will call updatePayloads() if necessary to update the list of known payloads for a given run but it will try to cache these results so incrementally asking for new payloads as they get requested is perfectly fine.

updatePayloads() will only called for globaltags if we still need to find payload information. If all payloads are found no further queries will be made so adding additional globaltags that are not used at the end is not expensive.

Parameters

	exp	the experiment number
	run	the run number
[in,out]	info	a list of metadata instances which we try to find the missing metadata for.

Returns: true if all required payloads could be found, false if at least one payload which still didn't have a valid revision number and is required could not be found.

Definition at line 36 of file MetadataProvider.cc.

  {
    if (info.empty()) return true;
    for (const auto& tag : m_tags) {
      // Check whether we already know the payloads for this exp,run ...
      bool present{false};
      std::tie(present, m_payloads) = m_cache[tag].get(exp, run);
      // Apparently not, try to get them
      if (!present && !updatePayloads(tag, exp, run)) {
        // Error obtaining payloads means something is wrong (server/network/file problem).
        // In this case we want to fall back to next metadata provider ...
        throw std::runtime_error("Problem updating metadata");
      }
      // And once we got them let's update the info for all requested payloads
      const auto& existing = *m_payloads;
      // and check if we found all payloads we're looking for.
      // Warning: **DO NOT** use `std::all_of` here which looks better but stops
      // iteration on the first missing payload instead of going through all the
      // remaining as well.
      const size_t found = std::count_if(info.begin(), info.end(), [&existing](auto & payload) {
        // already filled by previous gt or has a filename from testing payloads ... so don't do anything
        if (payload.revision > 0 or !payload.filename.empty()) return true;
        // otherwise look for the payload in the list of existing payloads for this run
        if (auto&& it = existing.find(payload.name); it != existing.end()) {
          payload.update(it->second);
          B2DEBUG(35, "Found requested payload metadata"
                  << LogVar("globaltag", payload.globaltag)
                  << LogVar("name", payload.name)
                  << LogVar("revision", payload.revision)
                  << LogVar("checksum", payload.checksum));
          return true;
        }
        // Not found, fine, not a big problem yet, there can be multiple global tags
        return false;
      });
      // If we found all we don't need to check the next global tag at all ...
      if (found == info.size()) break;
    }
    // Check that all payloads could be found (unless they're optional)
    // Again, we don't use std::all_of here because then it stops early and we
    // don't get errors for all missing payloads. But at least it would still be correct.
    const int missing = std::count_if(info.begin(), info.end(), [this, exp, run](const auto & p) {
      if (p.revision == 0 and p.filename.empty() and p.required) {
        B2ERROR("Cannot find payload in any of the configured global tags"
                << LogVar("name", p.name)
                << LogVar("globaltags", boost::algorithm::join(m_tags, ", "))
                << LogVar("experiment", exp) << LogVar("run", run));
        return true;
      }
      return false;
    });
    // Only happy without missing payloads
    return missing == 0;
  }

◆ getUsableTagStates()

std::set< std::string > getUsableTagStates ( )

inlineinherited

Get the valid tag states when checking globaltag status.

Definition at line 122 of file MetadataProvider.h.

122{ return m_usableTagStates; }

◆ printInfoMessage()

void printInfoMessage ( const std::string & provider = "" )

protectedinherited

Print an INFO message about the used metadata provider.

Parameters

provider the actual metadata provider used for the job

Definition at line 114 of file MetadataProvider.cc.

  {
    B2INFO("Conditions Database: found working metadata provider"
           << LogVar("provider", provider));
  }

◆ setTags()

bool setTags ( const std::vector< std::string > & tags )

inherited

Set the list of globaltag names to be considered for payloads.

This should be called before requesting any payloads but after calling setValidTagStates().

Returns: true if all globaltags can be found and are in a valid state

Definition at line 15 of file MetadataProvider.cc.

  {
    m_tags = tags;
    const size_t validTags = std::count_if(m_tags.begin(), m_tags.end(), [this](const auto & name) {
      // dispatch obtaining the state to the implementation
      auto status = getGlobaltagStatus(name);
      // empty status: Unspecified error already dealt with
      if (status.empty()) return false;
      // otherwise check for valid states
      if (m_usableTagStates.count(status) == 0) {
        B2ERROR("The globaltag has a status which is not permitted for use. This is for your own protection"
                << LogVar("globaltag", name)
                << LogVar("status", status)
                << LogVar("allowed states", boost::algorithm::join(m_usableTagStates, ", ")));
        return false;
      }
      return true;
    });
    return validTags == tags.size();
  }

◆ setUsableTagStates()

void setUsableTagStates ( const std::set< std::string > & states )

inlineinherited

Set the valid tag states for this provider when checking globaltag status.

Should be called before setTags() if necessary.

Warning: The state "INVALID" will never be accepted and removed from the given set if present

Definition at line 128 of file MetadataProvider.h.

    {
      m_usableTagStates = states;
      m_usableTagStates.erase("INVALID");
    }

◆ updatePayloads()

bool updatePayloads	(	const std::string &	globaltag,
		int	exp,
		int	run )

overridevirtual

Update the list of known payloads for the given globaltag/exp/run.

This basically just calls addPayload() for each payload it finds in the globaltag for the given exp/run.

Returns: true on success, false on any error

Implements MetadataProvider.

Definition at line 76 of file HSFCentralMetadataProvider.cc.

  {
    auto escaped = m_downloader.escapeString(globaltag);
    const std::string url =  "payloadiovs/?gtName=" + escaped +
                             "&majorIOV=" + std::to_string(exp) +
                             "&minorIOV=" + std::to_string(run) +
                             "&shape=dict";
    try {
      const auto payloads = get(url);
      if (!payloads.is_array()) throw std::runtime_error("expected array");
      for (const auto& payload : payloads) {
        if (!payload.is_object()) throw std::runtime_error("expected payload object");
 
        const long long int experimentHighFromServer = payload.at("major_iov_end");
        const long long int runHighFromServer = payload.at("minor_iov_end");
 
        // The infinite IoV is represented by m_maxIoV; in basf2 we replace it for -1 for historical reasons
        const long long int experimentHigh = (experimentHighFromServer == m_maxIoV) ? -1 : experimentHighFromServer;
        const long long int runHigh = (runHighFromServer == m_maxIoV) ? -1 : runHighFromServer;
 
        // Check if the current (exp, run) falls into the payload IoV:
        // if yes, let's keep the payload, otherwise skip it.
        const IntervalOfValidity iov{payload.at("major_iov"), payload.at("minor_iov"), static_cast<int>(experimentHigh), static_cast<int>(runHigh)};
        if (iov.contains(exp, run)) {
          addPayload(PayloadMetadata(
                       payload.at("payload_type_name"),
                       globaltag,
                       payload.at("payload_url"),
                       m_payloadBaseUrl,
                       payload.at("checksum"),
                       iov.getExperimentLow(),
                       iov.getRunLow(),
                       iov.getExperimentHigh(),
                       iov.getRunHigh(),
                       convertString<unsigned long int>(payload.at("revision"))
                     ));
          B2DEBUG(31, "Conditions Database: added payload from new central server"
                  << LogVar("Payload type name", payload.at("payload_type_name"))
                  << LogVar("url", payload.at("payload_url"))
                  << LogVar("checksum", payload.at("checksum")));
        }
      }
 
      B2DEBUG(31, "Conditions Database: fetched " << payloads.size() << " payloads for globaltag"
              << LogVar("globaltag", globaltag) << LogVar("exp", exp) << LogVar("run", run));
    } catch (std::exception& e) {
      B2WARNING("Conditions Database: Problem while fetching the list of payloads"
                << LogVar("globaltag", globaltag) << LogVar("server url", m_baseUrl) << LogVar("query", url) << LogVar("error", e.what()));
      return false;
    }
    return true;
  }

Member Data Documentation

◆ m_baseUrl

std::string m_baseUrl

private

base url of the server

Definition at line 54 of file HSFCentralMetadataProvider.h.

◆ m_cache

std::unordered_map<std::string, PayloadMetadataCache> m_cache

protectedinherited

Map of globaltag->known metadata.

Definition at line 172 of file MetadataProvider.h.

◆ m_downloader

Downloader& m_downloader {Downloader::getDefaultInstance()}

private

Reference to the downloader instance for convenience.

Definition at line 51 of file HSFCentralMetadataProvider.h.

51{Downloader::getDefaultInstance()};

◆ m_maxIoV

long long m_maxIoV = 9223372036854775807

staticconstexprprivate

Max IoV value, representing infinite end of validity.

Definition at line 60 of file HSFCentralMetadataProvider.h.

◆ m_payloadBaseUrl

std::string m_payloadBaseUrl

private

base url of the payload server

Definition at line 57 of file HSFCentralMetadataProvider.h.

◆ m_payloads

PayloadMap* m_payloads {nullptr}

protectedinherited

Map of known payloads for current conditions.

Definition at line 174 of file MetadataProvider.h.

174{nullptr};

◆ m_tags

std::vector<std::string> m_tags

protectedinherited

List of globaltags to consider.

Definition at line 170 of file MetadataProvider.h.

◆ m_usableTagStates

std::set<std::string> m_usableTagStates {"TESTING", "VALIDATED", "RUNNING", "PUBLISHED"}

protectedinherited

Set of global tag states to consider valid (except for 'INVALID' which is always considered invalid)

Definition at line 176 of file MetadataProvider.h.

176{"TESTING", "VALIDATED", "RUNNING", "PUBLISHED"};

The documentation for this class was generated from the following files:

framework/database/include/HSFCentralMetadataProvider.h
framework/database/src/HSFCentralMetadataProvider.cc

Public Types

Public Member Functions

Protected Member Functions

Protected Attributes

Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Member Typedef Documentation

◆ PayloadMap

Constructor & Destructor Documentation

◆ HSFCentralMetadataProvider()

Member Function Documentation

◆ addPayload()

◆ get()

◆ getGlobaltagStatus()

◆ getPayloads()

◆ getUsableTagStates()

◆ printInfoMessage()

◆ setTags()

◆ setUsableTagStates()

◆ updatePayloads()

Member Data Documentation

◆ m_baseUrl

◆ m_cache

◆ m_downloader

◆ m_maxIoV

◆ m_payloadBaseUrl

◆ m_payloads

◆ m_tags

◆ m_usableTagStates