PayloadProvider Class Reference

Class to find payload files in a list of locations. More...

#include <PayloadProvider.h>

Collaboration diagram for PayloadProvider:

Classes
struct	PayloadLocation
	Simple struct to represent a lookup location. More...

Public Types
enum class	EDirectoryLayout {   c_flat ,   c_hashed }
	Enumeration of different directory layouts. More...

Public Member Functions
	PayloadProvider (const std::vector< std::string > &locations, const std::string &cachedir="", int timeout=60)
	Constructor for a given list of locations and optionally the location where downloaded payloads should be cached Each location can either be a path to a directory or a http/https url of a server where the payloads can be downloaded (starting with http(s)://).
bool	find (PayloadMetadata &meta)
	Try to find a payload, return true on success, false if it cannot be found.

Private Member Functions
bool	getLocalFile (const PayloadLocation &loc, PayloadMetadata &meta) const
	Look for a payload in the local directory location, set the filename member of the metadata instance and return true on succes.
bool	getRemoteFile (const PayloadLocation &loc, PayloadMetadata &meta)
	Look for a payload on a remote server and download if possible, set the filename member of the metadata instance and return true on succes.
bool	getTemporaryFile (const std::string &url, PayloadMetadata &meta, bool silentOnMissing)
	Try to download url into a temporary file, if successful set the filename member of the metadata and return true.
std::string	getFilename (EDirectoryLayout structure, const PayloadMetadata &payload) const
	Return the filename of a payload to look for given a directory structure and some metadata.

Private Attributes
std::vector< PayloadLocation >	m_locations
	List of configured lookup locations: The first one will always be the cache directory and the last one will always be fallback url included in the payload metadata.
PayloadLocation	m_cacheDir
	Location of the cache directory where/how we want to store downloaded payloads.
Downloader &	m_downloader {Downloader::getDefaultInstance()}
	Instance to the database file downloading instance.
std::unordered_map< std::string, std::unique_ptr< FileSystem::TemporaryFile > >	m_temporaryFiles
	Map of all active temporary files we downloaded and keep around until they can be closed.
int	m_timeout
	Timeout to wait for a write look when trying to download payloads.

Detailed Description

Class to find payload files in a list of locations.

Definition at line 20 of file PayloadProvider.h.

Member Enumeration Documentation

EDirectoryLayout

enum class EDirectoryLayout

strong

Enumeration of different directory layouts.

Enumerator
c_flat	Flat directory containing the payloads in the form dbstore_{NAME}_rev_{REVISION}.root
c_hashed	Hashed directory structure containing the payloads in the form AB/{NAME}_r{REVISION}.root where A and B are the first to characters of the md5 checksum of the payload file.

Definition at line 32 of file PayloadProvider.h.

                                {
      c_flat,
      c_hashed,
    };

Constructor & Destructor Documentation

PayloadProvider()

PayloadProvider ( const std::vector< std::string > &  locations, const std::string &  cachedir = "", int  timeout = 60  )

explicit

Constructor for a given list of locations and optionally the location where downloaded payloads should be cached Each location can either be a path to a directory or a http/https url of a server where the payloads can be downloaded (starting with http(s)://).

For remote locations starting with http(s) we require that the payloads follow the same layout as on the central server. That is the location plus the value returned by the metadata for payloadUrl should point to the correct file.

For local locations we support two different layouts which will be auto detected and can be mixed:

• "flat": All payloads in one directory named dbstore_{NAME}_rev_{REVISION}.root
• "hashed": Payloads in subdirectories named /AB/{NAME}_r{REVISION}.root where A and B are the first two characters of the md5 checksum of the payload file.

If cachedir is empty a default value of $TMPDIR/basf2-conditions is assumed. Downloaded payloads will be placed in the cachedir using the hashed directory structure.

Definition at line 21 of file PayloadProvider.cc.

                                                                                                               : m_timeout{timeout}
  {
    // check whether we have a cache directory ... otherwise use default
    if (!cacheDir.empty()) {
      m_cacheDir = {fs::absolute(cacheDir).string(), false};
    } else {
      m_cacheDir = {fs::absolute(fs::temp_directory_path() / "basf2-conditions").string(), false};
    }
    m_locations.reserve(locations.size() + 2); //cache location + all configured + central server
    // always look in cache directory first
    m_locations.emplace_back(m_cacheDir);
    B2DEBUG(33, "Added payload cache location" << LogVar("path", m_cacheDir.base));
    // and then in the other directories as specified
    for (auto path : locations) {
      boost::algorithm::trim(path);
      if (path.empty()) {
        B2FATAL("Found empty payload location in configuration. "
                "Please make sure that the conditions database settings are correct");
      }
      bool remote = false;
      if (auto pos = path.find("://"); pos != std::string::npos) {
        //found a protocol: if file remove, otherwise keep as is and set as remote ...
        auto protocol = path.substr(0, pos);
        boost::algorithm::to_lower(protocol);
        if (protocol == "file") {
          path = path.substr(pos + 3);
        } else if (protocol == "http" or protocol == "https") {
          remote = true;
        } else {
          B2ERROR("Unknown protocol, only supported protocols for payload download are file, http, https" << LogVar("protocol", protocol));
          continue;
        }
      }
      // Also make sure files are absolute
      if (!remote) path = fs::absolute(path).string();
      // And then add it to the list
      B2DEBUG(33, "Added payload search location" << LogVar(remote ? "url" : "path", path));
      m_locations.emplace_back(PayloadLocation{path, remote});
    }
    // and as as last resort always go to the central server
    m_locations.emplace_back(PayloadLocation{"", true});
  }

Member Function Documentation

find()

bool find

(

PayloadMetadata &

meta

)

Try to find a payload, return true on success, false if it cannot be found.

This will go through all configured payloads locations and if it can find the payload will set the filename member of the metadata instance and return true.

Definition at line 64 of file PayloadProvider.cc.

  {
    // Check all locations for the file ... but dispatch to the correct member function
    return std::any_of(m_locations.begin(), m_locations.end(), [this, &metadata](const auto & loc) {
      return loc.isRemote ? getRemoteFile(loc, metadata) : getLocalFile(loc, metadata);
    });
  }

getFilename()

std::string getFilename ( EDirectoryLayout  structure, const PayloadMetadata &  payload  ) const

private

Return the filename of a payload to look for given a directory structure and some metadata.

Definition at line 168 of file PayloadProvider.cc.

  {
    fs::path path("");
    switch (structure) {
      case EDirectoryLayout::c_hashed:
        path /= payload.checksum.substr(0, 2);
        path /= payload.name + "_r" + std::to_string(payload.revision) + ".root";
        break;
      case EDirectoryLayout::c_flat:
        path /= "dbstore_" + payload.name + "_rev_" + std::to_string(payload.revision) + ".root";
        break;
    };
    return path.string();
  }

getLocalFile()

bool getLocalFile ( const PayloadLocation &  loc, PayloadMetadata &  meta  ) const

private

Look for a payload in the local directory location, set the filename member of the metadata instance and return true on succes.

Definition at line 72 of file PayloadProvider.cc.

  {
    // look in both flat and hashed directory structures.
    for (EDirectoryLayout structure : {EDirectoryLayout::c_hashed, EDirectoryLayout::c_flat}) {
      auto fullPath = fs::path(loc.base) / getFilename(structure, metadata);
      // No such file? nothing to do
      if (!fs::exists(fullPath)) continue;
      // Otherwise check the md5
      B2DEBUG(36, "Checking checksum for payload file" << LogVar("name", metadata.name) << LogVar("local dir", loc.base)
              << LogVar("revision", metadata.revision) << LogVar("filename", fullPath) << LogVar("checksum", metadata.checksum));
      const auto actual = FileSystem::calculateMD5(fullPath.string());
      if (actual == metadata.checksum) {
        metadata.filename = fullPath.string();
        B2DEBUG(37, "Found matching payload file");
        return true;
      }
      B2DEBUG(37, "Checksum doesn't match, continue with next");
    }
    return false;
  }

getRemoteFile()

bool getRemoteFile ( const PayloadLocation &  loc, PayloadMetadata &  meta  )

private

Look for a payload on a remote server and download if possible, set the filename member of the metadata instance and return true on succes.

Definition at line 93 of file PayloadProvider.cc.

  {
    // we want to download payloads in a hashed directory structure to keep amount of payloads per directory to a managable level
    const auto local = fs::path(m_cacheDir.base) / getFilename(EDirectoryLayout::c_hashed, metadata);
    // empty location: use the central server supplied baseUrl from payload metadata
    const bool fallback = loc.base.empty();
    const auto base = fallback ? metadata.baseUrl : loc.base;
    // but we assume that on servers we have logical directory structure: value of payloadUrl is taken without modification
    const auto url = m_downloader.joinWithSlash(base, metadata.payloadUrl);
    // If anything fails we might want to go to temporary file and this happens at multiple places so lets use exception for that
    try {
      // now we need to make the directories to the file
      try {
        // Make sure that we create directories writable for all users
        auto oldUmask = umask(0);
        ScopeGuard umaskGuard([oldUmask] {umask(oldUmask);});
        fs::create_directories(local.parent_path());
      } catch (fs::filesystem_error& e) {
        B2WARNING("Cannot create local payload directory" << LogVar("directory", local.parent_path())
                  << LogVar("error", e.code().message()));
        throw; //rethrow existing exception
      }
      // ok, directory exists, now we need a write lock on the file to avoid race conditions
      Belle2::FileSystem::Lock writelock(local.string());
      B2DEBUG(37, "Attempting to lock payload file for writing ..." << LogVar("filename", local));
      // if we cannot get one the folder/file might be write protected or
      // download by another process takes to long.  So let's download into
      // temporary file.
      if (!writelock.lock(m_timeout, true)) {
        throw std::runtime_error("write lock failed");
      }
      // Ok we have the write lock, check if we can open the file for writing which should be a guaranteed success but who knows
      // so bail if that fails.
      std::fstream localStream(local.string().c_str(), std::ios::binary | std::ios::in | std::ios::out);
      B2DEBUG(37, "Got write lock, check for file access ...");
      if (!localStream.good()) {
        B2ERROR("Cannot open file for writing" << LogVar("filename", local) << LogVar("error", strerror(errno)));
        throw std::runtime_error("cannot open file for writing????");
      }
      // and make sure it's readable for all
      fs::permissions(local, fs::perms::all &
                      ~(fs::perms::owner_exec | fs::perms::group_exec | fs::perms::others_exec));
      // File is open. Someone might have downloaded the file
      // while we waited, check md5sum again.
      B2DEBUG(37, "Ok, check digest in case another process downloaded already...");
      if (not m_downloader.verifyChecksum(localStream, metadata.checksum)) {
        // we have lock and it's broken so download the file
        B2DEBUG(37, "Still not good, download now ...");
        try {
          if (not m_downloader.download(url, localStream, not fallback)) {
            // if this returns false we have a 404: no need to try again
            B2DEBUG(37, "Payload not found ... trying next source");
            return false;
          }
          // ok, download was fine, check checksum again
          if (not m_downloader.verifyChecksum(localStream, metadata.checksum)) {
            B2WARNING("Conditions Database: checksum mismatch after download. Trying once more in a temporary file"
                      << LogVar("name", metadata.name) << LogVar("revision", metadata.revision) << LogVar("filename", local.string()));
            throw std::runtime_error("checksum mismatch");
          }
        } catch (std::exception& e) {
          B2ERROR("Conditions Database: failure downloading url" << LogVar("url", url) << LogVar("error", e.what()));
          throw; // rethrow existing exception
        }
        B2DEBUG(37, "Download of payload successful");
      }
      // found in cache or downloaded, fine
      metadata.filename = local.string();
      return true;
    } catch (std::exception&) {
      // errors are already shown, just try again
      return getTemporaryFile(url, metadata, not fallback);
    }
  }

getTemporaryFile()

bool getTemporaryFile ( const std::string &  url, PayloadMetadata &  meta, bool  silentOnMissing  )

private

Try to download url into a temporary file, if successful set the filename member of the metadata and return true.

Otherwise return false. If silentOnMissing is true a 404 error will not be treated as worthy of a message

Definition at line 184 of file PayloadProvider.cc.

  {
    if (auto&& it = m_temporaryFiles.find(metadata.checksum); it != m_temporaryFiles.end()) {
      metadata.filename = it->second->getName();
      return true;
    }
    const auto openmode = std::ios_base::binary | std::ios_base::in | std::ios_base::out | std::ios_base::trunc;
    std::unique_ptr<FileSystem::TemporaryFile> tmpfile(new FileSystem::TemporaryFile(openmode));
    B2DEBUG(37, "Trying to download into temporary file " << tmpfile->getName());
    try {
      if (not m_downloader.download(url, *tmpfile, silentOnMissing)) return false;
      if (not m_downloader.verifyChecksum(*tmpfile, metadata.checksum)) return false;
    } catch (std::exception& e) {
      B2ERROR("Conditions Database: failure downloading url" << LogVar("url", url) << LogVar("error", e.what()));
      return false;
    }
    metadata.filename = tmpfile->getName();
    m_temporaryFiles[metadata.checksum] = std::move(tmpfile);
    return true;
  }

Member Data Documentation

m_cacheDir

PayloadLocation m_cacheDir

private

Location of the cache directory where/how we want to store downloaded payloads.

Definition at line 93 of file PayloadProvider.h.

m_downloader

Downloader& m_downloader {Downloader::getDefaultInstance()}

private

Instance to the database file downloading instance.

Definition at line 95 of file PayloadProvider.h.

m_locations

std::vector<PayloadLocation> m_locations

private

List of configured lookup locations: The first one will always be the cache directory and the last one will always be fallback url included in the payload metadata.

Definition at line 91 of file PayloadProvider.h.

m_temporaryFiles

std::unordered_map<std::string, std::unique_ptr<FileSystem::TemporaryFile> > m_temporaryFiles

private

Map of all active temporary files we downloaded and keep around until they can be closed.

Definition at line 97 of file PayloadProvider.h.

m_timeout

int m_timeout

private

Timeout to wait for a write look when trying to download payloads.

Definition at line 99 of file PayloadProvider.h.

---

The documentation for this class was generated from the following files:

• framework/database/include/PayloadProvider.h
• framework/database/src/PayloadProvider.cc