Downloader Class Reference

Simple class to encapsulate libcurl as used by the ConditionsDatabase. More...

#include <Downloader.h>

Collaboration diagram for Downloader:

Public Member Functions
	Downloader ()=default
	Create a new payload downloader.
	~Downloader ()
	Destructor.
bool	startSession ()
	Start a new curl session if none is active at the moment. More...
void	finishSession ()
	Finish an existing curl session if any is active at the moment.
ScopeGuard	ensureSession ()
	Make sure there's an active session and return a ScopeGuard object that closes the session on destruction in case a new session was created;.
unsigned int	getConnectionTimeout () const
	Get the timeout to wait for connections in seconds, 0 means the built in curl default.
unsigned int	getStalledTimeout () const
	Get the timeout to wait for stalled connections (<10KB/s), 0 means no timeout.
unsigned int	getMaxRetries () const
	Get the number of retries to perform when downloading fails with HTTP response code >=500, 0 means no retries.
unsigned int	getBackoffFactor () const
	Get the backoff factor for retries in seconds.
void	setConnectionTimeout (unsigned int timeout)
	Set the timeout to wait for connections in seconds, 0 means built in curl default.
void	setStalledTimeout (unsigned int timeout)
	Set the timeout to wait for stalled connections (<10KB/s), 0 disables timeout.
void	setMaxRetries (unsigned int retries)
	Set the number of retries to perform when downloading fails with HTTP response code >=500, 0 disables retry.
void	setBackoffFactor (unsigned int factor)
	Set the backoff factor for retries in seconds. More...
bool	download (const std::string &url, std::ostream &stream, bool silentOnMissing=false)
	get an url and save the content to stream This function raises exceptions when there are any problems More...
bool	verifyChecksum (std::istream &input, const std::string &checksum)
	check the digest of a stream More...
std::string	escapeString (const std::string &text)
	Escape a string to make it safe to be used in web requests.
std::string	joinWithSlash (const std::string &base, const std::string &second)
	Join two strings and make sure that there is exactly one '/' between them.

Static Public Member Functions
static Downloader &	getDefaultInstance ()
	Return the default instance. More...

Private Member Functions
void	initializeRandomGeneratorSeed ()
	Initialize the seed of the internal random number generator. More...

Static Private Member Functions
static std::string	calculateChecksum (std::istream &input)
	calculate the digest/checksum on a given string. More...

Private Attributes
std::unique_ptr< CurlSession >	m_session
	curl session handle
unsigned int	m_connectionTimeout {60}
	Timeout to wait for connections in seconds.
unsigned int	m_stalledTimeout {60}
	Timeout to wait for stalled connections (<10KB/s)
unsigned int	m_maxRetries {5}
	Number of retries to perform when downloading fails with HTTP response code >=300.
unsigned int	m_backoffFactor {3}
	Backoff factor for retries in seconds.
std::unique_ptr< std::mt19937 >	m_rnd {std::make_unique<std::mt19937>()}
	This is a special exception in basf2 where an instance of gRandom is NOT used: since this class interacts with the Conditions Database, it might alter the state of the random number generator in case of connection troubles, loosing our capability to fully reproduce the results.
std::unique_ptr< std::uniform_real_distribution< double > >	m_rndDistribution {std::make_unique<std::uniform_real_distribution<double>>()}
	A uniform real distribution for extracting random numbers. More...
bool	m_rndIsInitialized {false}
	Flag for keeping track if the internal random generator is correctly initialized or not.

Static Private Attributes
static bool	s_globalInit {false}
	flag to indicate whether curl has been initialized already

Detailed Description

Simple class to encapsulate libcurl as used by the ConditionsDatabase.

Definition at line 22 of file Downloader.h.

Member Function Documentation

calculateChecksum()

std::string calculateChecksum ( std::istream &  input )

staticprivate

calculate the digest/checksum on a given string.

Parameters

input

input stream containing the data

Returns

the hex digest of the checksum

Definition at line 226 of file Downloader.cc.

  {
    // rewind stream
    input.clear();
    input.seekg(0, std::ios::beg);
    // and calculate md5 checksum by feeding it blockwise to the TMD5 update
    TMD5 md5;
    char buffer[4096];
    while (input.good()) {
      input.read(buffer, 4096);
      if (input.gcount() == 0) break;
      md5.Update((unsigned char*)buffer, input.gcount());
    }
    // finalize and return output
    md5.Final();
    return md5.AsString();
  }

download()

bool download	(	const std::string &	url,
		std::ostream &	stream,
		bool	silentOnMissing = false
	)

get an url and save the content to stream This function raises exceptions when there are any problems

Warning

any contents in the stream will be overwritten

Parameters

url	the url to download
stream	the stream to save the output to
silentOnMissing	if true do not emit a warning on 404 Not Found but just return false silently. Useful when checking if a file exists on the server

Returns

true on success, false on any error

Definition at line 260 of file Downloader.cc.

getDefaultInstance()

Downloader & getDefaultInstance ( )

static

Return the default instance.

There can be multiple instances without problem but we provide a default one to allow for better pipelining support

Definition at line 134 of file Downloader.cc.

initializeRandomGeneratorSeed()

void initializeRandomGeneratorSeed ( )

private

Initialize the seed of the internal random number generator.

Do nothing if the seed is already set (e.g. this method has been already called before). The hash of the basf2 seed is used as seed for m_rnd.

Definition at line 332 of file Downloader.cc.

setBackoffFactor()

void setBackoffFactor ( unsigned int  factor )

inline

Set the backoff factor for retries in seconds.

Minimum is 1 and 0 will be silently converted to 1

Definition at line 64 of file Downloader.h.

{ m_backoffFactor = std::max(1u, factor); }

startSession()

bool startSession

(

)

Start a new curl session if none is active at the moment.

Returns

true if a new session was started, false if one was active already

Definition at line 162 of file Downloader.cc.

verifyChecksum()

bool verifyChecksum ( std::istream &  input, const std::string &  checksum  )

inline

check the digest of a stream

Parameters

input	stream to check, make sure the stream is in a valid state pointing to the correct position
checksum	expected hash digest of the data

Returns

true if digest matches, false otherwise

Definition at line 81 of file Downloader.h.

Member Data Documentation

m_rndDistribution

std::unique_ptr<std::uniform_real_distribution<double> > m_rndDistribution {std::make_unique<std::uniform_real_distribution<double>>()}

private

A uniform real distribution for extracting random numbers.

See the docstring for m_rnd as well.

Definition at line 121 of file Downloader.h.

---

The documentation for this class was generated from the following files:

• framework/database/include/Downloader.h
• framework/database/src/Downloader.cc