PayloadProvider Class Reference

Class to find payload files in a list of locations. More...

#include <PayloadProvider.h>

Collaboration diagram for PayloadProvider:

Classes
struct	PayloadLocation
	Simple struct to represent a lookup location. More...

Public Types
enum class	EDirectoryLayout {   c_flat ,   c_hashed }
	Enumeration of different directory layouts. More...

Public Member Functions
	PayloadProvider (const std::vector< std::string > &locations, const std::string &cachedir="", int timeout=60)
	Constructor for a given list of locations and optionally the location where downloaded payloads should be cached Each location can either be a path to a directory or a http/https url of a server where the payloads can be downloaded (starting with http(s)://). More...
bool	find (PayloadMetadata &meta)
	Try to find a payload, return true on success, false if it cannot be found. More...

Private Member Functions
bool	getLocalFile (const PayloadLocation &loc, PayloadMetadata &meta) const
	Look for a payload in the local directory location, set the filename member of the metadata instance and return true on succes.
bool	getRemoteFile (const PayloadLocation &loc, PayloadMetadata &meta)
	Look for a payload on a remote server and download if possible, set the filename member of the metadata instance and return true on succes.
bool	getTemporaryFile (const std::string &url, PayloadMetadata &meta, bool silentOnMissing)
	Try to download url into a temporary file, if successful set the filename member of the metadata and return true. More...
std::string	getFilename (EDirectoryLayout structure, const PayloadMetadata &payload) const
	Return the filename of a payload to look for given a directory structure and some metadata.

Private Attributes
std::vector< PayloadLocation >	m_locations
	List of configured lookup locations: The first one will always be the cache directory and the last one will always be fallback url included in the payload metadata.
PayloadLocation	m_cacheDir
	Location of the cache directory where/how we want to store downloaded payloads.
Downloader &	m_downloader {Downloader::getDefaultInstance()}
	Instance to the database file downloading instance.
std::unordered_map< std::string, std::unique_ptr< FileSystem::TemporaryFile > >	m_temporaryFiles
	Map of all active temporary files we downloaded and keep around until they can be closed.
int	m_timeout
	Timeout to wait for a write look when trying to download payloads.

Detailed Description

Class to find payload files in a list of locations.

Definition at line 20 of file PayloadProvider.h.

Member Enumeration Documentation

EDirectoryLayout

enum EDirectoryLayout

strong

Enumeration of different directory layouts.

Enumerator
c_flat	Flat directory containing the payloads in the form dbstore_{NAME}_rev_{REVISION}.root
c_hashed	Hashed directory structure containing the payloads in the form AB/{NAME}_r{REVISION}.root where A and B are the first to characters of the md5 checksum of the payload file.

Definition at line 32 of file PayloadProvider.h.

                                {
      c_flat,
      c_hashed,
    };

Constructor & Destructor Documentation

PayloadProvider()

PayloadProvider ( const std::vector< std::string > &  locations, const std::string &  cachedir = "", int  timeout = 60  )

explicit

Constructor for a given list of locations and optionally the location where downloaded payloads should be cached Each location can either be a path to a directory or a http/https url of a server where the payloads can be downloaded (starting with http(s)://).

For remote locations starting with http(s) we require that the payloads follow the same layout as on the central server. That is the location plus the value returned by the metadata for payloadUrl should point to the correct file.

For local locations we support two different layouts which will be auto detected and can be mixed:

• "flat": All payloads in one directory named dbstore_{NAME}_rev_{REVISION}.root
• "hashed": Payloads in subdirectories named /AB/{NAME}_r{REVISION}.root where A and B are the first two characters of the md5 checksum of the payload file.

If cachedir is empty a default value of $TMPDIR/basf2-conditions is assumed. Downloaded payloads will be placed in the cachedir using the hashed directory structure.

Definition at line 21 of file PayloadProvider.cc.

                                                                                                               : m_timeout{timeout}
  {
    // check whether we have a cache directory ... otherwise use default
    if (!cacheDir.empty()) {
      m_cacheDir = {fs::absolute(cacheDir).string(), false};
    } else {
      m_cacheDir = {fs::absolute(fs::temp_directory_path() / "basf2-conditions").string(), false};
    }
    m_locations.reserve(locations.size() + 2); //cache location + all configured + central server
    // always look in cache directory first
    m_locations.emplace_back(m_cacheDir);
    B2DEBUG(33, "Added payload cache location" << LogVar("path", m_cacheDir.base));
    // and then in the other directories as specified
    for (auto path : locations) {
      boost::algorithm::trim(path);
      if (path.empty()) {
        B2FATAL("Found empty payload location in configuration. "
                "Please make sure that the conditions database settings are correct");
      }
      bool remote = false;
      if (auto pos = path.find("://"); pos != std::string::npos) {
        //found a protocol: if file remove, otherwise keep as is and set as remote ...
        auto protocol = path.substr(0, pos);
        boost::algorithm::to_lower(protocol);
        if (protocol == "file") {
          path = path.substr(pos + 3);
        } else if (protocol == "http" or protocol == "https") {
          remote = true;
        } else {
          B2ERROR("Unknown protocol, only supported protocols for payload download are file, http, https" << LogVar("protocol", protocol));
          continue;
        }
      }
      // Also make sure files are absolute
      if (!remote) path = fs::absolute(path).string();
      // And then add it to the list
      B2DEBUG(33, "Added payload search location" << LogVar(remote ? "url" : "path", path));
      m_locations.emplace_back(PayloadLocation{path, remote});
    }
    // and as as last resort always go to the central server
    m_locations.emplace_back(PayloadLocation{"", true});
  }

Member Function Documentation

find()

bool find

(

PayloadMetadata &

meta

)

Try to find a payload, return true on success, false if it cannot be found.

This will go through all configured payloads locations and if it can find the payload will set the filename member of the metadata instance and return true.

Definition at line 64 of file PayloadProvider.cc.

getTemporaryFile()

bool getTemporaryFile ( const std::string &  url, PayloadMetadata &  meta, bool  silentOnMissing  )

private

Try to download url into a temporary file, if successful set the filename member of the metadata and return true.

Otherwise return false. If silentOnMissing is true a 404 error will not be treated as worthy of a message

Definition at line 184 of file PayloadProvider.cc.

---

The documentation for this class was generated from the following files:

• framework/database/include/PayloadProvider.h
• framework/database/src/PayloadProvider.cc