Simple class to encapsulate libcurl as used by the ConditionsDatabase. More...

#include <Downloader.h>

Collaboration diagram for Downloader:

[legend]

Public Member Functions
	Downloader ()=default
	Create a new payload downloader.

	~Downloader ()
	Destructor.

bool	startSession ()
	Start a new curl session if none is active at the moment.

void	finishSession ()
	Finish an existing curl session if any is active at the moment.

ScopeGuard	ensureSession ()
	Make sure there's an active session and return a ScopeGuard object that closes the session on destruction in case a new session was created;.

unsigned int	getConnectionTimeout () const
	Get the timeout to wait for connections in seconds, 0 means the built in curl default.

unsigned int	getStalledTimeout () const
	Get the timeout to wait for stalled connections (<10KB/s), 0 means no timeout.

unsigned int	getMaxRetries () const
	Get the number of retries to perform when downloading fails with HTTP response code >=500, 0 means no retries.

unsigned int	getBackoffFactor () const
	Get the backoff factor for retries in seconds.

void	setConnectionTimeout (unsigned int timeout)
	Set the timeout to wait for connections in seconds, 0 means built in curl default.

void	setStalledTimeout (unsigned int timeout)
	Set the timeout to wait for stalled connections (<10KB/s), 0 disables timeout.

void	setMaxRetries (unsigned int retries)
	Set the number of retries to perform when downloading fails with HTTP response code >=500, 0 disables retry.

void	setBackoffFactor (unsigned int factor)
	Set the backoff factor for retries in seconds.

bool	download (const std::string &url, std::ostream &stream, bool silentOnMissing=false)
	get an url and save the content to stream This function raises exceptions when there are any problems

bool	verifyChecksum (std::istream &input, const std::string &checksum)
	check the digest of a stream

std::string	escapeString (const std::string &text)
	Escape a string to make it safe to be used in web requests.

std::string	joinWithSlash (const std::string &base, const std::string &second)
	Join two strings and make sure that there is exactly one '/' between them.

Static Public Member Functions
static Downloader &	getDefaultInstance ()
	Return the default instance.

Private Member Functions
void	initializeRandomGeneratorSeed ()
	Initialize the seed of the internal random number generator.

Static Private Member Functions
static std::string	calculateChecksum (std::istream &input)
	calculate the digest/checksum on a given string.

Private Attributes
std::unique_ptr< CurlSession >	m_session
	curl session handle

unsigned int	m_connectionTimeout {60}
	Timeout to wait for connections in seconds.

unsigned int	m_stalledTimeout {60}
	Timeout to wait for stalled connections (<10KB/s)

unsigned int	m_maxRetries {5}
	Number of retries to perform when downloading fails with HTTP response code >=300.

unsigned int	m_backoffFactor {3}
	Backoff factor for retries in seconds.

std::unique_ptr< std::mt19937 >	m_rnd {std::make_unique<std::mt19937>()}
	This is a special exception in basf2 where an instance of gRandom is NOT used: since this class interacts with the Conditions Database, it might alter the state of the random number generator in case of connection troubles, loosing our capability to fully reproduce the results.

std::unique_ptr< std::uniform_real_distribution< double > >	m_rndDistribution {std::make_unique<std::uniform_real_distribution<double>>()}
	A uniform real distribution for extracting random numbers.

bool	m_rndIsInitialized {false}
	Flag for keeping track if the internal random generator is correctly initialized or not.

Static Private Attributes
static bool	s_globalInit {false}
	flag to indicate whether curl has been initialized already

Detailed Description

Simple class to encapsulate libcurl as used by the ConditionsDatabase.

Definition at line 22 of file Downloader.h.

Constructor & Destructor Documentation

◆ ~Downloader()

~Downloader ( )

Destructor.

Definition at line 140 of file Downloader.cc.

140{ finishSession(); }

Belle2::Conditions::Downloader::finishSession

void finishSession()

Finish an existing curl session if any is active at the moment.

Definition: Downloader.cc:216

Member Function Documentation

◆ calculateChecksum()

std::string calculateChecksum ( std::istream & input )

staticprivate

calculate the digest/checksum on a given string.

Parameters

input input stream containing the data

Returns: the hex digest of the checksum

Definition at line 226 of file Downloader.cc.

  {
    // rewind stream
    input.clear();
    input.seekg(0, std::ios::beg);
    // and calculate md5 checksum by feeding it blockwise to the TMD5 update
    TMD5 md5;
    char buffer[4096];
    while (input.good()) {
      input.read(buffer, 4096);
      if (input.gcount() == 0) break;
      md5.Update((unsigned char*)buffer, input.gcount());
    }
    // finalize and return output
    md5.Final();
    return md5.AsString();
  }

◆ download()

bool download	(	const std::string &	url,
		std::ostream &	stream,
		bool	silentOnMissing = `false`
	)

get an url and save the content to stream This function raises exceptions when there are any problems

Warning: any contents in the stream will be overwritten

Parameters

url	the url to download
stream	the stream to save the output to
silentOnMissing	if true do not emit a warning on 404 Not Found but just return false silently. Useful when checking if a file exists on the server

Returns: true on success, false on any error

Definition at line 260 of file Downloader.cc.

  {
    // make sure we have an active curl session ...
    auto session = ensureSession();
    // and initialize the internal random number generator
    initializeRandomGeneratorSeed();
    B2DEBUG(37, "Download started ..." << LogVar("url", url));
    // we might need to try a few times in case of HTTP error >= 300
    for (unsigned int retry{1};; ++retry) {
      //rewind the stream to the beginning
      buffer.clear();
      buffer.seekp(0, std::ios::beg);
      if (!buffer.good()) {
        throw std::runtime_error("cannot write to stream");
      }
      // Set the exception flags to notify us of any problem during writing
      auto oldExceptionMask = buffer.exceptions();
      buffer.exceptions(std::ios::failbit | std::ios::badbit);
      // build the request ...
      CURLcode res{CURLE_FAILED_INIT};
      // and set all the curl options
      curl_easy_setopt(m_session->curl, CURLOPT_URL, url.c_str());
      curl_easy_setopt(m_session->curl, CURLOPT_WRITEDATA, &buffer);
      // perform the request ...
      res = curl_easy_perform(m_session->curl);
      // flush output
      buffer.exceptions(oldExceptionMask);
      buffer.flush();
      // and check for errors which occurred during download ...
      if (res != CURLE_OK) {
        size_t len = strlen(m_session->errbuf);
        const std::string error = len ? m_session->errbuf : curl_easy_strerror(res);
        if (m_maxRetries > 0 && res == CURLE_HTTP_RETURNED_ERROR) {
          if (retry <= m_maxRetries) {
            // we treat everything below 300 as permanent error with the request,
            // while if 300 or above we retry
            // 404 corresponds to Not Found and we want to treat it differently
            long responseCode{0};
            curl_easy_getinfo(m_session->curl, CURLINFO_RESPONSE_CODE, &responseCode);
            if (responseCode >= 300 and responseCode != 404) {
              // use exponential backoff but don't restrict to exact slots like
              // Ethernet, just use a random wait time between 1s and maxDelay =
              // 2^(retry)-1 * backoffFactor
              double maxDelay = (std::pow(2, retry) - 1) * m_backoffFactor;
              // This is an exception in the whole basf2: instead of relying on gRandom for getting a random number,
              // we rely on a different random number generator, and the reason is:
              // since the request may fail because of several reasons independent from basf2 (bad connection,
              // faulty squid cache, etc.), we might retry a new request altering the internal state of the gRandom
              // instance, spoiling our capability to fully reproduce our results.
              // In this way, relying on a different generator, we are safe.
              m_rndDistribution->param(std::uniform_real_distribution<double>::param_type(1.0, maxDelay));
              double seconds = (*m_rndDistribution)(*m_rnd);
              B2WARNING("Could not download url, retrying ..."
                        << LogVar("url", url) << LogVar("error", error)
                        << LogVar("try", retry) << LogVar("waiting time", seconds));
              std::this_thread::sleep_for(std::chrono::milliseconds((int)(seconds * 1e3)));
              continue;
            }
            // special treatment for 404: if silentOnMissing is true we just return false silently
            // this is useful when checking if a file exists on the server
            if (responseCode == 404 and silentOnMissing) return false;
          }
        }
        throw std::runtime_error(error);
      }
      break;
    }
    // all fine
    B2DEBUG(37, "Download finished successfully." << LogVar("url", url));
    return true;
  }

◆ ensureSession()

ScopeGuard ensureSession ( )

inline

Make sure there's an active session and return a ScopeGuard object that closes the session on destruction in case a new session was created;.

Definition at line 43 of file Downloader.h.

    {
      bool started = startSession();
      return ScopeGuard([this, started] {if (started) finishSession();});
    }

◆ escapeString()

std::string escapeString ( const std::string & text )

Escape a string to make it safe to be used in web requests.

Definition at line 142 of file Downloader.cc.

  {
    //make sure we have an active curl session ...
    auto session = ensureSession(); // cppcheck-suppress unreadVariable
    char* escaped = curl_easy_escape(m_session->curl, text.c_str(), text.size());
    if (!escaped) {
      throw std::runtime_error("Could not escape string");
    }
    std::string escapedStr{escaped};
    curl_free(escaped);
    return escapedStr;
  }

◆ finishSession()

void finishSession ( )

Finish an existing curl session if any is active at the moment.

Definition at line 216 of file Downloader.cc.

  {
    // if there's a session clean it ...
    if (m_session) {
      curl_easy_cleanup(m_session->curl);
      curl_slist_free_all(m_session->headers);
      m_session.reset();
    }
  }

◆ getBackoffFactor()

unsigned int getBackoffFactor ( ) const

inline

Get the backoff factor for retries in seconds.

Definition at line 56 of file Downloader.h.

56{ return m_backoffFactor; }

◆ getConnectionTimeout()

unsigned int getConnectionTimeout ( ) const

inline

Get the timeout to wait for connections in seconds, 0 means the built in curl default.

Definition at line 50 of file Downloader.h.

50{ return m_connectionTimeout; }

Belle2::Conditions::Downloader::m_connectionTimeout

unsigned int m_connectionTimeout

Timeout to wait for connections in seconds.

Definition: Downloader.h:100

◆ getDefaultInstance()

Downloader & getDefaultInstance ( )

static

Return the default instance.

There can be multiple instances without problem but we provide a default one to allow for better pipelining support

Definition at line 134 of file Downloader.cc.

  {
    static Downloader instance;
    return instance;
  }

◆ getMaxRetries()

unsigned int getMaxRetries ( ) const

inline

Get the number of retries to perform when downloading fails with HTTP response code >=500, 0 means no retries.

Definition at line 54 of file Downloader.h.

54{ return m_maxRetries; }

◆ getStalledTimeout()

unsigned int getStalledTimeout ( ) const

inline

Get the timeout to wait for stalled connections (<10KB/s), 0 means no timeout.

Definition at line 52 of file Downloader.h.

52{ return m_stalledTimeout; }

Belle2::Conditions::Downloader::m_stalledTimeout

unsigned int m_stalledTimeout

Timeout to wait for stalled connections (<10KB/s)

Definition: Downloader.h:102

◆ initializeRandomGeneratorSeed()

void initializeRandomGeneratorSeed ( )

private

Initialize the seed of the internal random number generator.

Do nothing if the seed is already set (e.g. this method has been already called before). The hash of the basf2 seed is used as seed for m_rnd.

Definition at line 332 of file Downloader.cc.

  {
    if (not m_rndIsInitialized) {
      // We need to provide a seed for m_rnd: let's take the basf2Seed and hash it
      auto downloaderSeed = std::hash<std::string> {}(RandomNumbers::getSeed());
      m_rnd->seed(downloaderSeed);
      m_rndIsInitialized = true;
    }
  }

◆ joinWithSlash()

std::string joinWithSlash	(	const std::string &	base,
		const std::string &	second
	)

Join two strings and make sure that there is exactly one '/' between them.

Definition at line 156 of file Downloader.cc.

  {
    return boost::trim_right_copy_if(base, boost::is_any_of("/")) + "/" +
           boost::trim_left_copy_if(rest, boost::is_any_of("/"));
  }

◆ setBackoffFactor()

void setBackoffFactor ( unsigned int factor )

inline

Set the backoff factor for retries in seconds.

Minimum is 1 and 0 will be silently converted to 1

Definition at line 64 of file Downloader.h.

64{ m_backoffFactor = std::max(1u, factor); }

◆ setConnectionTimeout()

void setConnectionTimeout ( unsigned int timeout )

Set the timeout to wait for connections in seconds, 0 means built in curl default.

Definition at line 244 of file Downloader.cc.

  {
    m_connectionTimeout = timeout;
    if (m_session) {
      curl_easy_setopt(m_session->curl, CURLOPT_CONNECTTIMEOUT, m_connectionTimeout);
    }
  }

◆ setMaxRetries()

void setMaxRetries ( unsigned int retries )

inline

Set the number of retries to perform when downloading fails with HTTP response code >=500, 0 disables retry.

Definition at line 62 of file Downloader.h.

62{ m_maxRetries = retries; }

◆ setStalledTimeout()

void setStalledTimeout ( unsigned int timeout )

Set the timeout to wait for stalled connections (<10KB/s), 0 disables timeout.

Definition at line 252 of file Downloader.cc.

  {
    m_stalledTimeout = timeout;
    if (m_session) {
      curl_easy_setopt(m_session->curl, CURLOPT_LOW_SPEED_TIME, m_stalledTimeout);
    }
  }

◆ startSession()

bool startSession ( )

Start a new curl session if none is active at the moment.

Returns: true if a new session was started, false if one was active already

Definition at line 162 of file Downloader.cc.

  {
    // start a curl session but if there is already one return false
    if (m_session) return false;
    // make sure curl is initialized correctly
    if (!s_globalInit) {
      curl_global_init(CURL_GLOBAL_ALL);
      s_globalInit = true;
    }
    // create the curl session
    m_session = std::make_unique<CurlSession>();
    m_session->curl = curl_easy_init();
    if (!m_session->curl) {
      B2FATAL("Cannot initialize libcurl");
    }
    m_session->headers = curl_slist_append(nullptr, "Accept: application/json");
    curl_easy_setopt(m_session->curl, CURLOPT_HTTPHEADER, m_session->headers);
    curl_easy_setopt(m_session->curl, CURLOPT_TCP_KEEPALIVE, 1L);
    curl_easy_setopt(m_session->curl, CURLOPT_CONNECTTIMEOUT, m_connectionTimeout);
    curl_easy_setopt(m_session->curl, CURLOPT_LOW_SPEED_LIMIT, 10 * 1024); //10 kB/s
    curl_easy_setopt(m_session->curl, CURLOPT_LOW_SPEED_TIME, m_stalledTimeout);
    curl_easy_setopt(m_session->curl, CURLOPT_WRITEFUNCTION, write_function);
    curl_easy_setopt(m_session->curl, CURLOPT_VERBOSE, 1);
    curl_easy_setopt(m_session->curl, CURLOPT_NOPROGRESS, 0);
    curl_easy_setopt(m_session->curl, CURLOPT_DEBUGFUNCTION, debug_callback);
    curl_easy_setopt(m_session->curl, CURLOPT_XFERINFOFUNCTION, progress_callback);
    curl_easy_setopt(m_session->curl, CURLOPT_XFERINFODATA, m_session.get());
    curl_easy_setopt(m_session->curl, CURLOPT_FAILONERROR, true);
    curl_easy_setopt(m_session->curl, CURLOPT_ERRORBUFFER, m_session->errbuf);
    // enable transparent compression support
    curl_easy_setopt(m_session->curl, CURLOPT_ACCEPT_ENCODING, "");
    // Set proxy if defined
    if (EnvironmentVariables::isSet("BELLE2_CONDB_PROXY")) {
      const std::string proxy = EnvironmentVariables::get("BELLE2_CONDB_PROXY");
      curl_easy_setopt(m_session->curl, CURLOPT_PROXY, proxy.c_str());
    }
    curl_easy_setopt(m_session->curl, CURLOPT_AUTOREFERER, 1L);
    curl_easy_setopt(m_session->curl, CURLOPT_FOLLOWLOCATION, 1L);
    curl_easy_setopt(m_session->curl, CURLOPT_MAXREDIRS, 10L);
    curl_easy_setopt(m_session->curl, CURLOPT_TCP_FASTOPEN, 0L);
    curl_easy_setopt(m_session->curl, CURLOPT_SSL_VERIFYPEER, 0L);
    curl_easy_setopt(m_session->curl, CURLOPT_SSL_VERIFYHOST, 0L);
    curl_easy_setopt(m_session->curl, CURLOPT_SSL_VERIFYSTATUS, 0L);
    curl_easy_setopt(m_session->curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_WHATEVER);
    // Don't cache DNS entries, ask the system every time we need to connect ...
    curl_easy_setopt(m_session->curl, CURLOPT_DNS_CACHE_TIMEOUT, 0L);
    // and shuffle the addresses so we try a different node, otherwise we might
    // always get the same address due to system caching and RFC 3484
    curl_easy_setopt(m_session->curl, CURLOPT_DNS_SHUFFLE_ADDRESSES, 1L);
    auto version = getUserAgent();
    curl_easy_setopt(m_session->curl, CURLOPT_USERAGENT, version.c_str());
    return true;
  }

◆ verifyChecksum()

bool verifyChecksum	(	std::istream &	input,
		const std::string &	checksum
	)

inline

check the digest of a stream

Parameters

input	stream to check, make sure the stream is in a valid state pointing to the correct position
checksum	expected hash digest of the data

Returns: true if digest matches, false otherwise

Definition at line 81 of file Downloader.h.

81{ return calculateChecksum(input) == checksum; }

Belle2::Conditions::Downloader::calculateChecksum

static std::string calculateChecksum(std::istream &input)

calculate the digest/checksum on a given string.

Definition: Downloader.cc:226

Member Data Documentation

◆ m_backoffFactor

unsigned int m_backoffFactor {3}

private

Backoff factor for retries in seconds.

Definition at line 106 of file Downloader.h.

◆ m_connectionTimeout

unsigned int m_connectionTimeout {60}

private

Timeout to wait for connections in seconds.

Definition at line 100 of file Downloader.h.

◆ m_maxRetries

unsigned int m_maxRetries {5}

private

Number of retries to perform when downloading fails with HTTP response code >=300.

Definition at line 104 of file Downloader.h.

◆ m_rnd

std::unique_ptr<std::mt19937> m_rnd {std::make_unique<std::mt19937>()}

private

This is a special exception in basf2 where an instance of gRandom is NOT used: since this class interacts with the Conditions Database, it might alter the state of the random number generator in case of connection troubles, loosing our capability to fully reproduce the results.

Definition at line 119 of file Downloader.h.

◆ m_rndDistribution

std::unique_ptr<std::uniform_real_distribution<double> > m_rndDistribution {std::make_unique<std::uniform_real_distribution<double>>()}

private

A uniform real distribution for extracting random numbers.

See the docstring for m_rnd as well.

Definition at line 121 of file Downloader.h.

◆ m_rndIsInitialized

bool m_rndIsInitialized {false}

private

Flag for keeping track if the internal random generator is correctly initialized or not.

Definition at line 123 of file Downloader.h.

◆ m_session

std::unique_ptr<CurlSession> m_session

private

curl session handle

Definition at line 96 of file Downloader.h.

◆ m_stalledTimeout

unsigned int m_stalledTimeout {60}

private

Timeout to wait for stalled connections (<10KB/s)

Definition at line 102 of file Downloader.h.

◆ s_globalInit

bool s_globalInit {false}

staticprivate

flag to indicate whether curl has been initialized already

Definition at line 98 of file Downloader.h.

The documentation for this class was generated from the following files:

framework/database/include/Downloader.h
framework/database/src/Downloader.cc

Public Member Functions

Static Public Member Functions

Private Member Functions

Static Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ ~Downloader()

Member Function Documentation

◆ calculateChecksum()

◆ download()

◆ ensureSession()

◆ escapeString()

◆ finishSession()

◆ getBackoffFactor()

◆ getConnectionTimeout()

◆ getDefaultInstance()

◆ getMaxRetries()

◆ getStalledTimeout()

◆ initializeRandomGeneratorSeed()

◆ joinWithSlash()

◆ setBackoffFactor()

◆ setConnectionTimeout()

◆ setMaxRetries()

◆ setStalledTimeout()

◆ startSession()

◆ verifyChecksum()

Member Data Documentation

◆ m_backoffFactor

◆ m_connectionTimeout

◆ m_maxRetries

◆ m_rnd

◆ m_rndDistribution

◆ m_rndIsInitialized

◆ m_session

◆ m_stalledTimeout

◆ s_globalInit