TestingPayloadStorage Class Reference

Class to store and retrieve temporary payloads. More...

#include <TestingPayloadStorage.h>

Collaboration diagram for TestingPayloadStorage:

Public Member Functions
	TestingPayloadStorage (const std::string &filename)
	Create a new instance to work on a given filename.
bool	get (const EventMetaData &event, PayloadMetadata &info)
	Try to fill the PayloadMetaData for the given EventMetaData, return true on success, false if no machting payload could be found.
bool	storeData (const std::string &name, TObject *object, const IntervalOfValidity &iov)
	Store a TObject instance as a payload with given name and interval of validity. More...
bool	storePayload (const std::string &name, const std::string &fileName, const IntervalOfValidity &iov)
	Store an existing file as payload with given name and interval of validity. More...
void	reset ()
	Reset the list of known payloads. More...

Private Member Functions
void	read ()
	Read the given storage file, done lazily on first access to get() after construction or call to reset()
bool	writePayload (const std::string &fileName, const std::string &name, const TObject *object)
	Write a payload file from the given object and name. More...
bool	store (const std::string &name, const IntervalOfValidity &iov, const std::string &source, const std::function< bool(const std::string &)> &writer)
	Try to store a new payload with the given name and interval of validity. More...

Static Private Member Functions
static std::string	payloadFilename (const std::string &path, const std::string &name, const std::string &revision)
	Build the filename for a new payload with a given name and revision in a directory.

Private Attributes
std::unordered_map< std::string, std::vector< std::tuple< std::string, IntervalOfValidity > > >	m_payloads
	Map of known payloads to a list of known revisions and their interval of validity.
std::string	m_filename
	Storage file where to look for payloads. More...
std::string	m_absoluteFilename
	Storage file where to look for payloads converted to an absolute path to be robust against directory changes.
std::string	m_payloadDir
	Directory containing the storage file as absolute file name. More...
bool	m_initialized {false}
	Remember whether we read the file already.

Detailed Description

Class to store and retrieve temporary payloads.

Temporary payloads are stored in a directory where we create a root files and a plain text file with name, revision and iov for each of these files. This is not very safe but ideal for testing and manual adjustments before final validation and upload to the cental server.

The interface is very basic: We create an instance with a filename and then we either get() payload information for a given event and payload metadata struct or we store a given payload either from an object (storeData()) or from an existing file (storePayload()).

Both call the underlying store() which tries to be robust against multiple processes trying to create payloads at the same time.

Definition at line 39 of file TestingPayloadStorage.h.

Member Function Documentation

reset()

void reset ( )

inline

Reset the list of known payloads.

Will trigger re-reading the file on next access to get()

Definition at line 55 of file TestingPayloadStorage.h.

{ m_initialized = false; }

store()

bool store ( const std::string &  name, const IntervalOfValidity &  iov, const std::string &  source, const std::function< bool(const std::string &)> &  writer  )

private

Try to store a new payload with the given name and interval of validity.

This function first tries to find the next free revision and then call the writer function to write the payload to the given filename. If the writer function returns success the payload is added to the storage file

Parameters

name	payload name
iov	iov for the payload
source	source filename to use. If this is empty there is no file yet and it has to be created first
writer	callback function to create a file. Will be called with a destination filename if the source parameter was emty

Definition at line 155 of file TestingPayloadStorage.cc.

  {
    if (iov.empty()) {
      B2ERROR("IoV is empty, refusing to store object in testing payload storage"
              "Please provide a valid experiment/run range for the data, for example "
              "using IntervalOfValidity::always() to store data which is always valid"
              << LogVar("name", name));
      return false;
    }
 
    if (!fs::exists(m_payloadDir)) {
      fs::create_directories(m_payloadDir);
    }
 
    // create a temporary file if we don't have a source file yet
    fs::path sourcefile{source};
    if (source.empty()) {
      while (true) {
        sourcefile = fs::path(m_payloadDir) / fs::unique_path();
        auto fd = open(sourcefile.c_str(), O_CREAT | O_EXCL);
        if (fd >= 0) {
          close(fd);
          break;
        }
        if (errno != EEXIST && errno != EINTR) {
          B2ERROR("Cannot create payload file:" << strerror(errno));
          return false;
        }
        B2DEBUG(35, "first try to create tempfile failed, trying again");
      }
      if (!writer(sourcefile.string())) return false;
    }
    // If we created a temporary file we want to delete it again so we'd like to
    // use a scope guard to do so. However we need it in this scope so we need
    // to create one in any case and release it if we didn't create a temporary
    // file
    ScopeGuard delete_srcfile([&sourcefile] {fs::remove(sourcefile);});
    if (!source.empty()) delete_srcfile.release();
 
    std::string md5 = FileSystem::calculateMD5(sourcefile.string());
 
    // Ok, now we have the file and it's md5 sum so let's get a write lock to the database file
    // to avoid race conditions when creating files and writing the info in the text file.
    FileSystem::Lock lock(m_absoluteFilename);
    if (!lock.lock()) {
      B2ERROR("Locking of testing payload storage file failed, cannot create payload"
              << LogVar("storage file", m_filename));
      return false;
    }
    std::ofstream file(m_absoluteFilename.c_str(), std::ios::app);
    if (!file.is_open()) {
      B2ERROR("Could not open testing payload storage file for writing" << LogVar("storage file", m_filename));
    }
 
    // So let's try renaming our temporary payload file to final destination
    // We start with a 5 digit hash and expand if there's a collision
    std::string revision;
    bool found = false;
    for (int i = 6; i <= 32; ++i) {
      revision = md5.substr(0, i);
      auto filename = payloadFilename(m_payloadDir, name, revision);
      if (FileSystem::fileExists(filename)) {
        if (md5 != FileSystem::calculateMD5(filename)) continue;
      } else {
        if (source.empty()) {
          fs::rename(sourcefile, filename);
          delete_srcfile.release();
        } else {
          fs::copy_file(source, filename, fs::copy_option::overwrite_if_exists);
        }
      }
      found = true;
      break;
    }
    if (!found) {
      B2ERROR("Cannot create payload file: checksum mistmatch for existing files");
      return false;
    }
    // Ok, add to the text file
    file << "dbstore/" << name << " " << revision << " " << iov << std::endl;
    B2DEBUG(32, "Storing testing payload" << LogVar("storage file", m_filename) << LogVar("name", name)
            << LogVar("local revision", revision) << LogVar("iov", iov));
    // And make sure we reread the file on next request to payloads
    m_initialized = false;
    return true;
  }

storeData()

bool storeData	(	const std::string &	name,
		TObject *	object,
		const IntervalOfValidity &	iov
	)

Store a TObject instance as a payload with given name and interval of validity.

This will create a new ROOT file containing the object in the directory of the storage file

Definition at line 133 of file TestingPayloadStorage.cc.

storePayload()

bool storePayload	(	const std::string &	name,
		const std::string &	fileName,
		const IntervalOfValidity &	iov
	)

Store an existing file as payload with given name and interval of validity.

This will create a copy of the file with a different name in the directory of the storage file

Definition at line 140 of file TestingPayloadStorage.cc.

writePayload()

bool writePayload ( const std::string &  fileName, const std::string &  name, const TObject *  object  )

private

Write a payload file from the given object and name.

Will create new root file containing object under the Key name with the name fileName

Definition at line 243 of file TestingPayloadStorage.cc.

Member Data Documentation

m_filename

std::string m_filename

private

Storage file where to look for payloads.

This is the logical file name as given by the user

Definition at line 62 of file TestingPayloadStorage.h.

m_payloadDir

std::string m_payloadDir

private

Directory containing the storage file as absolute file name.

Will try to create it if it doesn't exist and storeData() or storePayload() is called

Definition at line 68 of file TestingPayloadStorage.h.

---

The documentation for this class was generated from the following files:

• framework/database/include/TestingPayloadStorage.h
• framework/database/src/TestingPayloadStorage.cc