Class to store and retrieve temporary payloads. More...

#include <TestingPayloadStorage.h>

Public Member Functions
	TestingPayloadStorage (const std::string &filename)
	Create a new instance to work on a given filename.

bool	get (const EventMetaData &event, PayloadMetadata &info)
	Try to fill the PayloadMetaData for the given EventMetaData, return true on success, false if no machting payload could be found.

bool	storeData (const std::string &name, TObject *object, const IntervalOfValidity &iov)
	Store a TObject instance as a payload with given name and interval of validity.

bool	storePayload (const std::string &name, const std::string &fileName, const IntervalOfValidity &iov)
	Store an existing file as payload with given name and interval of validity.

void	reset ()
	Reset the list of known payloads.

Private Member Functions
void	read ()
	Read the given storage file, done lazily on first access to get() after construction or call to reset()

bool	writePayload (const std::string &fileName, const std::string &name, const TObject *object)
	Write a payload file from the given object and name.

bool	store (const std::string &name, const IntervalOfValidity &iov, const std::string &source, const std::function< bool(const std::string &)> &writer)
	Try to store a new payload with the given name and interval of validity.

Static Private Member Functions
static std::string	payloadFilename (const std::string &path, const std::string &name, const std::string &revision)
	Build the filename for a new payload with a given name and revision in a directory.

Private Attributes
std::unordered_map< std::string, std::vector< std::tuple< std::string, IntervalOfValidity > > >	m_payloads
	Map of known payloads to a list of known revisions and their interval of validity.

std::string	m_filename
	Storage file where to look for payloads.

std::string	m_absoluteFilename
	Storage file where to look for payloads converted to an absolute path to be robust against directory changes.

std::string	m_payloadDir
	Directory containing the storage file as absolute file name.

bool	m_initialized {false}
	Remember whether we read the file already.

Detailed Description

Class to store and retrieve temporary payloads.

Temporary payloads are stored in a directory where we create a root files and a plain text file with name, revision and iov for each of these files. This is not very safe but ideal for testing and manual adjustments before final validation and upload to the cental server.

The interface is very basic: We create an instance with a filename and then we either get() payload information for a given event and payload metadata struct or we store a given payload either from an object (storeData()) or from an existing file (storePayload()).

Both call the underlying store() which tries to be robust against multiple processes trying to create payloads at the same time.

Definition at line 39 of file TestingPayloadStorage.h.

Constructor & Destructor Documentation

◆ TestingPayloadStorage()

TestingPayloadStorage ( const std::string & filename )

explicit

Create a new instance to work on a given filename.

Definition at line 26 of file TestingPayloadStorage.cc.

                                                                       :
    m_filename{filename}, m_absoluteFilename{fs::absolute(m_filename).string()},
    m_payloadDir{fs::path(m_absoluteFilename).parent_path().string()}
  {}

Member Function Documentation

◆ get()

bool get	(	const EventMetaData &	event,
		PayloadMetadata &	info
	)

Try to fill the PayloadMetaData for the given EventMetaData, return true on success, false if no machting payload could be found.

Definition at line 31 of file TestingPayloadStorage.cc.

  {
    // Read database file on first access
    if (!m_initialized) read();
    m_initialized = true;
    // And then let's look for a payload with the name
    auto it = m_payloads.find(info.name);
    if (it == m_payloads.end()) return false;
    bool found{false};
    // and check all of them for what is the highest revision containing the event
    for (const auto& [revision, iov] : it->second) {
      if (iov.contains(event)) {
        info.revision = 0;
        info.globaltag = "temp://" + m_absoluteFilename;
        info.baseUrl = "";
        info.payloadUrl = "";
        info.filename = payloadFilename(m_payloadDir, info.name, revision);
        info.iov = iov;
        if (!FileSystem::fileExists(info.filename)) {
          B2FATAL("Could not find payload file specified in testing payload storage" << LogVar("storage filen", m_filename)
                  << LogVar("name", info.name) << LogVar("local revision", info.revision)
                  << LogVar("payload filename", info.filename));
        }
        info.checksum = FileSystem::calculateMD5(info.filename);
        found = true;
        break;
      }
    }
    if (found) {
      B2WARNING("Conditions: Temporary testing payload used for processing" << LogVar("storage file", m_filename)
                << LogVar("name", info.name) << LogVar("revision", info.revision) << LogVar("iov", info.iov));
    }
    // Did we find something? did we?
    return found;
  }

◆ payloadFilename()

std::string payloadFilename	(	const std::string &	path,
		const std::string &	name,
		const std::string &	revision
	)

staticprivate

Build the filename for a new payload with a given name and revision in a directory.

Definition at line 125 of file TestingPayloadStorage.cc.

  {
    std::stringstream result;
    if (!path.empty()) result << path << '/';
    result << "dbstore_" << name << "_rev_" << revision << ".root";
    return result.str();
  }

◆ read()

void read ( )

private

Read the given storage file, done lazily on first access to get() after construction or call to reset()

Definition at line 67 of file TestingPayloadStorage.cc.

  {
    m_payloads.clear();
    if (!fs::is_regular_file(m_absoluteFilename)) {
      B2WARNING("Given testing payload storage file doesn't exist or is not a regular file" << LogVar("storage file", m_filename));
      return;
    }
    // read and parse the database content
    std::ifstream file(m_absoluteFilename.c_str());
    if (!file.is_open()) {
      B2FATAL("Opening of testing payload storage file failed" << LogVar("storage file", m_filename) << LogVar("error", strerror(errno)));
    }
    int lineno{0};
    try {
      while (!file.eof()) {
        // read the file line by line
        std::string line;
        std::getline(file, line);
        ++lineno;
        // and remove comments from the line
        size_t commentChar = line.find('#');
        if (commentChar != std::string::npos) {
          line = line.substr(0, commentChar);
        }
        // trim whitespace on each side
        boost::algorithm::trim(line);
        // if nothing is left skip the line
        if (line.empty()) continue;
        // otherwise read name, revision and iov from the line
        std::string name;
        std::string revision;
        IntervalOfValidity iov;
        try {
          std::stringstream(line) >> name >> revision >> iov;
        } catch (std::runtime_error& e) {
          throw std::runtime_error("line must be of the form 'dbstore/<payloadname> <revision> <firstExp>,<firstRun>,<finalExp>,<finalRun>'");
        }
        // parse name
        size_t pos = name.find('/');
        if (pos == std::string::npos) {
          throw std::runtime_error("payload name must be of the form dbstore/<payloadname>");
        }
        std::string module = name.substr(pos + 1, name.length());
        // and add to map of payloads
        B2DEBUG(39, "Found testing payload" << LogVar("storage file", m_filename) << LogVar("name", module)
                << LogVar("revision/md5", revision) << LogVar("iov", iov));
        m_payloads[module].emplace_back(revision, iov);
      }
    } catch (std::exception& e) {
      B2FATAL("Problem reading testing payloads storage" << LogVar("storage file", m_filename)
              << LogVar("line", lineno) << LogVar("error", e.what()));
    }
    // and reverse all the payloads so the last ones in the file have highest priority
    for (auto& payload : m_payloads) {
      std::reverse(payload.second.begin(), payload.second.end());
    }
  }

◆ reset()

void reset ( )

inline

Reset the list of known payloads.

Will trigger re-reading the file on next access to get()

Definition at line 55 of file TestingPayloadStorage.h.

55{ m_initialized = false; }

◆ store()

bool store	(	const std::string &	name,
		const IntervalOfValidity &	iov,
		const std::string &	source,
		const std::function< bool(const std::string &)> &	writer
	)

private

Try to store a new payload with the given name and interval of validity.

This function first tries to find the next free revision and then call the writer function to write the payload to the given filename. If the writer function returns success the payload is added to the storage file

Parameters

name	payload name
iov	iov for the payload
source	source filename to use. If this is empty there is no file yet and it has to be created first
writer	callback function to create a file. Will be called with a destination filename if the source parameter was emty

Definition at line 156 of file TestingPayloadStorage.cc.

  {
    if (iov.empty()) {
      B2ERROR("IoV is empty, refusing to store object in testing payload storage"
              "Please provide a valid experiment/run range for the data, for example "
              "using IntervalOfValidity::always() to store data which is always valid"
              << LogVar("name", name));
      return false;
    }
 
    if (!fs::exists(m_payloadDir)) {
      fs::create_directories(m_payloadDir);
    }
 
    // create a temporary file if we don't have a source file yet
    fs::path sourcefile{source};
    int length = m_payloadDir.length();
    char* temporaryFileName = new char[length + 16];
    std::strcpy(temporaryFileName, m_payloadDir.c_str());
    std::strcpy(temporaryFileName + length, "/payload_XXXXXX");
    if (source.empty()) {
      while (true) {
        int fileDescriptor = mkstemp(temporaryFileName);
        if ((fileDescriptor == -1) && (errno != EINTR)) {
          B2ERROR("Cannot create payload file:" << strerror(errno));
          delete[] temporaryFileName;
          return false;
        }
        if (fileDescriptor > 0) {
          sourcefile = temporaryFileName;
          close(fileDescriptor);
          break;
        }
        B2DEBUG(35, "first try to create tempfile failed, trying again");
      }
      if (!writer(sourcefile.string())) return false;
    }
    delete[] temporaryFileName;
    // If we created a temporary file we want to delete it again so we'd like to
    // use a scope guard to do so. However we need it in this scope so we need
    // to create one in any case and release it if we didn't create a temporary
    // file
    ScopeGuard delete_srcfile([&sourcefile] {fs::remove(sourcefile);});
    if (!source.empty()) delete_srcfile.release();
 
    std::string md5 = FileSystem::calculateMD5(sourcefile.string());
 
    // Ok, now we have the file and it's md5 sum so let's get a write lock to the database file
    // to avoid race conditions when creating files and writing the info in the text file.
    FileSystem::Lock lock(m_absoluteFilename);
    if (!lock.lock()) {
      B2ERROR("Locking of testing payload storage file failed, cannot create payload"
              << LogVar("storage file", m_filename));
      return false;
    }
    std::ofstream file(m_absoluteFilename.c_str(), std::ios::app);
    if (!file.is_open()) {
      B2ERROR("Could not open testing payload storage file for writing" << LogVar("storage file", m_filename));
    }
 
    // So let's try renaming our temporary payload file to final destination
    // We start with a 5 digit hash and expand if there's a collision
    std::string revision;
    bool found = false;
    for (int i = 6; i <= 32; ++i) {
      revision = md5.substr(0, i);
      auto filename = payloadFilename(m_payloadDir, name, revision);
      if (FileSystem::fileExists(filename)) {
        if (md5 != FileSystem::calculateMD5(filename)) continue;
      } else {
        if (source.empty()) {
          fs::rename(sourcefile, filename);
          delete_srcfile.release();
        } else {
          fs::copy_file(source, filename, fs::copy_options::overwrite_existing);
        }
      }
      found = true;
      break;
    }
    if (!found) {
      B2ERROR("Cannot create payload file: checksum mismatch for existing files");
      return false;
    }
    // Ok, add to the text file
    file << "dbstore/" << name << " " << revision << " " << iov << std::endl;
    B2DEBUG(32, "Storing testing payload" << LogVar("storage file", m_filename) << LogVar("name", name)
            << LogVar("local revision", revision) << LogVar("iov", iov));
    // And make sure we reread the file on next request to payloads
    m_initialized = false;
    return true;
  }

◆ storeData()

bool storeData	(	const std::string &	name,
		TObject *	object,
		const IntervalOfValidity &	iov
	)

Store a TObject instance as a payload with given name and interval of validity.

This will create a new ROOT file containing the object in the directory of the storage file

Definition at line 134 of file TestingPayloadStorage.cc.

  {
    return store(name, iov, "", [this, &object, &name](const std::string & filename) {
      return writePayload(filename, name, object);
    });
  }

◆ storePayload()

bool storePayload	(	const std::string &	name,
		const std::string &	fileName,
		const IntervalOfValidity &	iov
	)

Store an existing file as payload with given name and interval of validity.

This will create a copy of the file with a different name in the directory of the storage file

Definition at line 141 of file TestingPayloadStorage.cc.

  {
    // resolve all symbolic links to make sure we point to the real file
    fs::path resolved = fs::canonical(fileName);
    if (not fs::is_regular_file(resolved)) {
      B2ERROR("Problem creating testing payload: Given payload storage file doesn't exist" << LogVar("storage file", fileName));
      return false;
    }
    return store(name, iov, resolved.string(), [&resolved](const std::string & destination) {
      // copy payload file to payload directory and rename it to follow the file name convention
      fs::copy_file(resolved, destination, fs::copy_options::overwrite_existing);
      return true;
    });
  }

◆ writePayload()

bool writePayload	(	const std::string &	fileName,
		const std::string &	name,
		const TObject *	object
	)

private

Write a payload file from the given object and name.

Will create new root file containing object under the Key name with the name fileName

Definition at line 250 of file TestingPayloadStorage.cc.

  {
    // Save the current gDirectory
    TDirectory::TContext saveDir;
    // And create a reproducible TFile: one that has the same checksum every time it's created as long as the content is the same
    std::unique_ptr<TFile> file{TFile::Open((fileName + "?reproducible=PayloadFile").c_str(), "RECREATE")};
    if (!file || !file->IsOpen()) {
      B2ERROR("Could not open payload file for writing." << LogVar("filename", m_filename));
      return false;
    }
    // Write the payload
    object->Write(name.c_str(), TObject::kSingleKey);
    // Done, let's go
    file->Close();
    return true;
  }

Member Data Documentation

◆ m_absoluteFilename

std::string m_absoluteFilename

private

Storage file where to look for payloads converted to an absolute path to be robust against directory changes.

Definition at line 64 of file TestingPayloadStorage.h.

◆ m_filename

std::string m_filename

private

Storage file where to look for payloads.

This is the logical file name as given by the user

Definition at line 62 of file TestingPayloadStorage.h.

◆ m_initialized

bool m_initialized {false}

private

Remember whether we read the file already.

Definition at line 70 of file TestingPayloadStorage.h.

◆ m_payloadDir

std::string m_payloadDir

private

Directory containing the storage file as absolute file name.

Will try to create it if it doesn't exist and storeData() or storePayload() is called

Definition at line 68 of file TestingPayloadStorage.h.

◆ m_payloads

std::unordered_map<std::string, std::vector<std::tuple<std::string, IntervalOfValidity> > > m_payloads

private

Map of known payloads to a list of known revisions and their interval of validity.

Definition at line 60 of file TestingPayloadStorage.h.

The documentation for this class was generated from the following files:

framework/database/include/TestingPayloadStorage.h
framework/database/src/TestingPayloadStorage.cc

Public Member Functions

Private Member Functions

Static Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ TestingPayloadStorage()

Member Function Documentation

◆ get()

◆ payloadFilename()

◆ read()

◆ reset()

◆ store()

◆ storeData()

◆ storePayload()

◆ writePayload()

Member Data Documentation

◆ m_absoluteFilename

◆ m_filename

◆ m_initialized

◆ m_payloadDir

◆ m_payloads