![]() |
Belle II Software development
|
Simple class to encapsulate libcurl as used by the ConditionsDatabase. More...
#include <Downloader.h>
Public Member Functions | |
Downloader ()=default | |
Create a new payload downloader. | |
~Downloader () | |
Destructor. | |
bool | startSession () |
Start a new curl session if none is active at the moment. | |
void | finishSession () |
Finish an existing curl session if any is active at the moment. | |
ScopeGuard | ensureSession () |
Make sure there's an active session and return a ScopeGuard object that closes the session on destruction in case a new session was created;. | |
unsigned int | getConnectionTimeout () const |
Get the timeout to wait for connections in seconds, 0 means the built in curl default. | |
unsigned int | getStalledTimeout () const |
Get the timeout to wait for stalled connections (<10KB/s), 0 means no timeout. | |
unsigned int | getMaxRetries () const |
Get the number of retries to perform when downloading fails with HTTP response code >=500, 0 means no retries. | |
unsigned int | getBackoffFactor () const |
Get the backoff factor for retries in seconds. | |
void | setConnectionTimeout (unsigned int timeout) |
Set the timeout to wait for connections in seconds, 0 means built in curl default. | |
void | setStalledTimeout (unsigned int timeout) |
Set the timeout to wait for stalled connections (<10KB/s), 0 disables timeout. | |
void | setMaxRetries (unsigned int retries) |
Set the number of retries to perform when downloading fails with HTTP response code >=500, 0 disables retry. | |
void | setBackoffFactor (unsigned int factor) |
Set the backoff factor for retries in seconds. | |
bool | download (const std::string &url, std::ostream &stream, bool silentOnMissing=false) |
get an url and save the content to stream This function raises exceptions when there are any problems | |
bool | verifyChecksum (std::istream &input, const std::string &checksum) |
check the digest of a stream | |
std::string | escapeString (const std::string &text) |
Escape a string to make it safe to be used in web requests. | |
std::string | joinWithSlash (const std::string &base, const std::string &second) |
Join two strings and make sure that there is exactly one '/' between them. | |
Static Public Member Functions | |
static Downloader & | getDefaultInstance () |
Return the default instance. | |
Private Member Functions | |
void | initializeRandomGeneratorSeed () |
Initialize the seed of the internal random number generator. | |
Static Private Member Functions | |
static std::string | calculateChecksum (std::istream &input) |
calculate the digest/checksum on a given string. | |
Private Attributes | |
std::unique_ptr< CurlSession > | m_session |
curl session handle | |
unsigned int | m_connectionTimeout {60} |
Timeout to wait for connections in seconds. | |
unsigned int | m_stalledTimeout {60} |
Timeout to wait for stalled connections (<10KB/s) | |
unsigned int | m_maxRetries {5} |
Number of retries to perform when downloading fails with HTTP response code >=300. | |
unsigned int | m_backoffFactor {3} |
Backoff factor for retries in seconds. | |
std::unique_ptr< std::mt19937 > | m_rnd {std::make_unique<std::mt19937>()} |
This is a special exception in basf2 where an instance of gRandom is NOT used: since this class interacts with the Conditions Database, it might alter the state of the random number generator in case of connection troubles, loosing our capability to fully reproduce the results. | |
std::unique_ptr< std::uniform_real_distribution< double > > | m_rndDistribution {std::make_unique<std::uniform_real_distribution<double>>()} |
A uniform real distribution for extracting random numbers. | |
bool | m_rndIsInitialized {false} |
Flag for keeping track if the internal random generator is correctly initialized or not. | |
Static Private Attributes | |
static bool | s_globalInit {false} |
flag to indicate whether curl has been initialized already | |
Simple class to encapsulate libcurl as used by the ConditionsDatabase.
Definition at line 22 of file Downloader.h.
~Downloader | ( | ) |
Destructor.
Definition at line 140 of file Downloader.cc.
|
staticprivate |
calculate the digest/checksum on a given string.
input | input stream containing the data |
Definition at line 226 of file Downloader.cc.
bool download | ( | const std::string & | url, |
std::ostream & | stream, | ||
bool | silentOnMissing = false |
||
) |
get an url and save the content to stream This function raises exceptions when there are any problems
url | the url to download |
stream | the stream to save the output to |
silentOnMissing | if true do not emit a warning on 404 Not Found but just return false silently. Useful when checking if a file exists on the server |
Definition at line 260 of file Downloader.cc.
|
inline |
Make sure there's an active session and return a ScopeGuard object that closes the session on destruction in case a new session was created;.
Definition at line 43 of file Downloader.h.
std::string escapeString | ( | const std::string & | text | ) |
Escape a string to make it safe to be used in web requests.
Definition at line 142 of file Downloader.cc.
void finishSession | ( | ) |
|
inline |
|
inline |
Get the timeout to wait for connections in seconds, 0 means the built in curl default.
Definition at line 50 of file Downloader.h.
|
static |
Return the default instance.
There can be multiple instances without problem but we provide a default one to allow for better pipelining support
Definition at line 134 of file Downloader.cc.
|
inline |
Get the number of retries to perform when downloading fails with HTTP response code >=500, 0 means no retries.
Definition at line 54 of file Downloader.h.
|
inline |
Get the timeout to wait for stalled connections (<10KB/s), 0 means no timeout.
Definition at line 52 of file Downloader.h.
|
private |
Initialize the seed of the internal random number generator.
Do nothing if the seed is already set (e.g. this method has been already called before). The hash of the basf2 seed is used as seed for m_rnd
.
Definition at line 332 of file Downloader.cc.
std::string joinWithSlash | ( | const std::string & | base, |
const std::string & | second | ||
) |
Join two strings and make sure that there is exactly one '/' between them.
Definition at line 156 of file Downloader.cc.
|
inline |
Set the backoff factor for retries in seconds.
Minimum is 1 and 0 will be silently converted to 1
Definition at line 64 of file Downloader.h.
void setConnectionTimeout | ( | unsigned int | timeout | ) |
Set the timeout to wait for connections in seconds, 0 means built in curl default.
Definition at line 244 of file Downloader.cc.
|
inline |
Set the number of retries to perform when downloading fails with HTTP response code >=500, 0 disables retry.
Definition at line 62 of file Downloader.h.
void setStalledTimeout | ( | unsigned int | timeout | ) |
Set the timeout to wait for stalled connections (<10KB/s), 0 disables timeout.
Definition at line 252 of file Downloader.cc.
bool startSession | ( | ) |
Start a new curl session if none is active at the moment.
Definition at line 162 of file Downloader.cc.
|
inline |
check the digest of a stream
input | stream to check, make sure the stream is in a valid state pointing to the correct position |
checksum | expected hash digest of the data |
Definition at line 81 of file Downloader.h.
|
private |
Backoff factor for retries in seconds.
Definition at line 106 of file Downloader.h.
|
private |
Timeout to wait for connections in seconds.
Definition at line 100 of file Downloader.h.
|
private |
Number of retries to perform when downloading fails with HTTP response code >=300.
Definition at line 104 of file Downloader.h.
|
private |
This is a special exception in basf2 where an instance of gRandom is NOT used: since this class interacts with the Conditions Database, it might alter the state of the random number generator in case of connection troubles, loosing our capability to fully reproduce the results.
Definition at line 119 of file Downloader.h.
|
private |
A uniform real distribution for extracting random numbers.
See the docstring for m_rnd
as well.
Definition at line 121 of file Downloader.h.
|
private |
Flag for keeping track if the internal random generator is correctly initialized or not.
Definition at line 123 of file Downloader.h.
|
private |
curl session handle
Definition at line 96 of file Downloader.h.
|
private |
Timeout to wait for stalled connections (<10KB/s)
Definition at line 102 of file Downloader.h.
|
staticprivate |
flag to indicate whether curl has been initialized already
Definition at line 98 of file Downloader.h.