Belle II Software  release-05-02-19
Downloader Class Referencefinal

Simple class to encapsulate libcurl as used by the ConditionsDatabase. More...

#include <Downloader.h>

Collaboration diagram for Downloader:

Public Member Functions

 Downloader ()=default
 Create a new payload downloader.
 
 ~Downloader ()
 Destructor.
 
bool startSession ()
 Start a new curl session if none is active at the moment. More...
 
void finishSession ()
 Finish an existing curl session if any is active at the moment.
 
ScopeGuard ensureSession ()
 Make sure there's an active session and return a ScopeGuard object that closes the session on destruction in case a new session was created;.
 
unsigned int getConnectionTimeout () const
 Get the timeout to wait for connections in seconds, 0 means the built in curl default.
 
unsigned int getStalledTimeout () const
 Get the timeout to wait for stalled connections (<10KB/s), 0 means no timeout.
 
unsigned int getMaxRetries () const
 Get the number of retries to perform when downloading fails with HTTP response code >=500, 0 means no retries.
 
unsigned int getBackoffFactor () const
 Get the backoff factor for retries in seconds.
 
void setConnectionTimeout (unsigned int timeout)
 Set the timeout to wait for connections in seconds, 0 means built in curl default.
 
void setStalledTimeout (unsigned int timeout)
 Set the timeout to wait for stalled connections (<10KB/s), 0 disables timeout.
 
void setMaxRetries (unsigned int retries)
 Set the number of retries to perform when downloading fails with HTTP response code >=500, 0 disables retry.
 
void setBackoffFactor (unsigned int factor)
 Set the backoff factor for retries in seconds. More...
 
bool download (const std::string &url, std::ostream &stream, bool silentOnMissing=false)
 get an url and save the content to stream This function raises exceptions when there are any problems More...
 
bool verifyChecksum (std::istream &input, const std::string &checksum)
 check the digest of a stream More...
 
std::string escapeString (const std::string &text)
 Escape a string to make it safe to be used in web requests.
 
std::string joinWithSlash (const std::string &base, const std::string &second)
 Join two strings and make sure that there is exactly one '/' between them.
 

Static Public Member Functions

static DownloadergetDefaultInstance ()
 Return the default instance. More...
 

Static Private Member Functions

static std::string calculateChecksum (std::istream &input)
 calculate the digest/checksum on a given string. More...
 

Private Attributes

std::unique_ptr< CurlSessionm_session
 curl session handle
 
unsigned int m_connectionTimeout {60}
 Timeout to wait for connections in seconds.
 
unsigned int m_stalledTimeout {60}
 Timeout to wait for stalled connections (<10KB/s)
 
unsigned int m_maxRetries {5}
 Number of retries to perform when downloading fails with HTTP response code >=500.
 
unsigned int m_backoffFactor {5}
 Backoff factor for retries in seconds.
 

Static Private Attributes

static bool s_globalInit {false}
 flag to indicate whether curl has been initialized already
 

Detailed Description

Simple class to encapsulate libcurl as used by the ConditionsDatabase.

Definition at line 31 of file Downloader.h.

Member Function Documentation

◆ calculateChecksum()

std::string calculateChecksum ( std::istream &  input)
staticprivate

calculate the digest/checksum on a given string.

Parameters
inputinput stream containing the data
Returns
the hex digest of the checksum

Definition at line 234 of file Downloader.cc.

234  {
235  input.read(buffer, 4096);
236  if (input.gcount() == 0) break;
237  md5.Update((unsigned char*)buffer, input.gcount());
238  }
239  // finalize and return output
240  md5.Final();
241  return md5.AsString();
242  }
243 
244  void Downloader::setConnectionTimeout(unsigned int timeout)
245  {
246  m_connectionTimeout = timeout;
247  if (m_session) {
248  curl_easy_setopt(m_session->curl, CURLOPT_CONNECTTIMEOUT, m_connectionTimeout);
249  }
250  }

◆ download()

bool download ( const std::string &  url,
std::ostream &  stream,
bool  silentOnMissing = false 
)

get an url and save the content to stream This function raises exceptions when there are any problems

Warning
any contents in the stream will be overwritten
Parameters
urlthe url to download
streamthe stream to save the output to
silentOnMissingif true do not emit a warning on 404 Not Found but just return false silently. Useful when checking if a file exists on the server
Returns
true on success, false on any error

Definition at line 268 of file Downloader.cc.

◆ getDefaultInstance()

Downloader & getDefaultInstance ( )
static

Return the default instance.

There can be multiple instances without problem but we provide a default one to allow for better pipelining support

Definition at line 143 of file Downloader.cc.

◆ setBackoffFactor()

void setBackoffFactor ( unsigned int  factor)
inline

Set the backoff factor for retries in seconds.

Minimum is 1 and 0 will be silently converted to 1

Definition at line 74 of file Downloader.h.

◆ startSession()

bool startSession ( )

Start a new curl session if none is active at the moment.

Returns
true if a new session was started, false if one was active already

Definition at line 171 of file Downloader.cc.

◆ verifyChecksum()

bool verifyChecksum ( std::istream &  input,
const std::string &  checksum 
)
inline

check the digest of a stream

Parameters
inputstream to check, make sure the stream is in a valid state pointing to the correct position
checksumexpected hash digest of the data
Returns
true if digest matches, false otherwise

Definition at line 92 of file Downloader.h.


The documentation for this class was generated from the following files:
Belle2::Conditions::Downloader::m_connectionTimeout
unsigned int m_connectionTimeout
Timeout to wait for connections in seconds.
Definition: Downloader.h:110
Belle2::Conditions::Downloader::setConnectionTimeout
void setConnectionTimeout(unsigned int timeout)
Set the timeout to wait for connections in seconds, 0 means built in curl default.
Definition: Downloader.cc:252
Belle2::Conditions::Downloader::m_session
std::unique_ptr< CurlSession > m_session
curl session handle
Definition: Downloader.h:106