Belle II Software light-2406-ragdoll
|
A module that measures the execution time and memory usage per event. More...
#include <ProfileModule.h>
Classes | |
struct | MemTime |
An internal struct to store pairs of memory usage and time. More... | |
Public Types | |
enum | EModulePropFlags { c_Input = 1 , c_Output = 2 , c_ParallelProcessingCertified = 4 , c_HistogramManager = 8 , c_InternalSerializer = 16 , c_TerminateInAllProcesses = 32 , c_DontCollectStatistics = 64 } |
Each module can be tagged with property flags, which indicate certain features of the module. More... | |
typedef ModuleCondition::EAfterConditionPath | EAfterConditionPath |
Forward the EAfterConditionPath definition from the ModuleCondition. | |
Public Member Functions | |
ProfileModule () | |
Constructor. | |
virtual void | initialize () override |
Initializes the Module. | |
virtual void | event () override |
Event profiling. | |
virtual void | terminate () override |
Terminate the Module. | |
virtual std::vector< std::string > | getFileNames (bool outputFiles) |
Return a list of output filenames for this modules. | |
virtual void | beginRun () |
Called when entering a new run. | |
virtual void | endRun () |
This method is called if the current run ends. | |
const std::string & | getName () const |
Returns the name of the module. | |
const std::string & | getType () const |
Returns the type of the module (i.e. | |
const std::string & | getPackage () const |
Returns the package this module is in. | |
const std::string & | getDescription () const |
Returns the description of the module. | |
void | setName (const std::string &name) |
Set the name of the module. | |
void | setPropertyFlags (unsigned int propertyFlags) |
Sets the flags for the module properties. | |
LogConfig & | getLogConfig () |
Returns the log system configuration. | |
void | setLogConfig (const LogConfig &logConfig) |
Set the log system configuration. | |
void | setLogLevel (int logLevel) |
Configure the log level. | |
void | setDebugLevel (int debugLevel) |
Configure the debug messaging level. | |
void | setAbortLevel (int abortLevel) |
Configure the abort log level. | |
void | setLogInfo (int logLevel, unsigned int logInfo) |
Configure the printed log information for the given level. | |
void | if_value (const std::string &expression, const std::shared_ptr< Path > &path, EAfterConditionPath afterConditionPath=EAfterConditionPath::c_End) |
Add a condition to the module. | |
void | if_false (const std::shared_ptr< Path > &path, EAfterConditionPath afterConditionPath=EAfterConditionPath::c_End) |
A simplified version to add a condition to the module. | |
void | if_true (const std::shared_ptr< Path > &path, EAfterConditionPath afterConditionPath=EAfterConditionPath::c_End) |
A simplified version to set the condition of the module. | |
bool | hasCondition () const |
Returns true if at least one condition was set for the module. | |
const ModuleCondition * | getCondition () const |
Return a pointer to the first condition (or nullptr, if none was set) | |
const std::vector< ModuleCondition > & | getAllConditions () const |
Return all set conditions for this module. | |
bool | evalCondition () const |
If at least one condition was set, it is evaluated and true returned if at least one condition returns true. | |
std::shared_ptr< Path > | getConditionPath () const |
Returns the path of the last true condition (if there is at least one, else reaturn a null pointer). | |
Module::EAfterConditionPath | getAfterConditionPath () const |
What to do after the conditional path is finished. | |
std::vector< std::shared_ptr< Path > > | getAllConditionPaths () const |
Return all condition paths currently set (no matter if the condition is true or not). | |
bool | hasProperties (unsigned int propertyFlags) const |
Returns true if all specified property flags are available in this module. | |
bool | hasUnsetForcedParams () const |
Returns true and prints error message if the module has unset parameters which the user has to set in the steering file. | |
const ModuleParamList & | getParamList () const |
Return module param list. | |
template<typename T > | |
ModuleParam< T > & | getParam (const std::string &name) const |
Returns a reference to a parameter. | |
bool | hasReturnValue () const |
Return true if this module has a valid return value set. | |
int | getReturnValue () const |
Return the return value set by this module. | |
std::shared_ptr< PathElement > | clone () const override |
Create an independent copy of this module. | |
std::shared_ptr< boost::python::list > | getParamInfoListPython () const |
Returns a python list of all parameters. | |
Static Public Member Functions | |
static void | exposePythonAPI () |
Exposes methods of the Module class to Python. | |
Protected Member Functions | |
virtual void | def_initialize () |
Wrappers to make the methods without "def_" prefix callable from Python. | |
virtual void | def_beginRun () |
Wrapper method for the virtual function beginRun() that has the implementation to be used in a call from Python. | |
virtual void | def_event () |
Wrapper method for the virtual function event() that has the implementation to be used in a call from Python. | |
virtual void | def_endRun () |
This method can receive that the current run ends as a call from the Python side. | |
virtual void | def_terminate () |
Wrapper method for the virtual function terminate() that has the implementation to be used in a call from Python. | |
void | setDescription (const std::string &description) |
Sets the description of the module. | |
void | setType (const std::string &type) |
Set the module type. | |
template<typename T > | |
void | addParam (const std::string &name, T ¶mVariable, const std::string &description, const T &defaultValue) |
Adds a new parameter to the module. | |
template<typename T > | |
void | addParam (const std::string &name, T ¶mVariable, const std::string &description) |
Adds a new enforced parameter to the module. | |
void | setReturnValue (int value) |
Sets the return value for this module as integer. | |
void | setReturnValue (bool value) |
Sets the return value for this module as bool. | |
void | setParamList (const ModuleParamList ¶ms) |
Replace existing parameter list. | |
Private Types | |
typedef std::function< unsigned long(MemTime const &) > | MemoryExtractLambda |
Signature of the lambda functions, which are used to extract the memory usage from teh MemTime structure. | |
Private Member Functions | |
void | storeMemoryGraph (const std::string &name, const std::string &title, const std::string &xAxisName, const std::string &imgOutput, const MemoryExtractLambda &lmdMemoryExtract) |
Stores the memory usage of the application over time in plots. | |
std::list< ModulePtr > | getModules () const override |
no submodules, return empty list | |
std::string | getPathString () const override |
return the module name. | |
void | setParamPython (const std::string &name, const boost::python::object &pyObj) |
Implements a method for setting boost::python objects. | |
void | setParamPythonDict (const boost::python::dict &dictionary) |
Implements a method for reading the parameter values from a boost::python dictionary. | |
Private Attributes | |
std::string | m_outputFileName |
Name for image output file. | |
std::string | m_rssOutputFileName |
Name for rss image output file. | |
const MemoryExtractLambda | m_extractVirtualMem = [](MemTime const& m) { return m.virtualMem;} |
Lambda expression to return the virtual memory from a MemTime data structure. | |
const MemoryExtractLambda | m_extractRssMem = [](MemTime const& m) { return m.rssMem;} |
Lambda expression to return the Rss memory from a MemTime data structure. | |
double | m_timeOffset |
time at module creation | |
MemTime | m_initializeInfo |
memory usage and time at initialization | |
MemTime | m_terminateInfo |
memory usage and time at termination | |
int | m_nEvents |
event counter | |
int | m_step |
number of events between profile points | |
std::vector< MemTime > | m_eventInfo |
memory usage and time at individual events | |
MemTime | m_startInfo |
memory usage and time at start of event loop + burn in | |
MemTime | m_endInfo |
memory usage and time at end of event loop | |
std::string | m_name |
The name of the module, saved as a string (user-modifiable) | |
std::string | m_type |
The type of the module, saved as a string. | |
std::string | m_package |
Package this module is found in (may be empty). | |
std::string | m_description |
The description of the module. | |
unsigned int | m_propertyFlags |
The properties of the module as bitwise or (with |) of EModulePropFlags. | |
LogConfig | m_logConfig |
The log system configuration of the module. | |
ModuleParamList | m_moduleParamList |
List storing and managing all parameter of the module. | |
bool | m_hasReturnValue |
True, if the return value is set. | |
int | m_returnValue |
The return value. | |
std::vector< ModuleCondition > | m_conditions |
Module condition, only non-null if set. | |
Static Private Attributes | |
static const int | k_burnIn = 1 |
number of events before the average time measurement is started | |
static const int | k_maxPoints = 100 |
maximal number of profile points | |
A module that measures the execution time and memory usage per event.
This module adds a ProfileInfo object to the datastore and records execution time and memory usage in it.
Both the virtual memory and the resident set size (rss) of the process is reported. Both quantities are important and have a complementary meaning:
virtual size (vsize): The memory region the kernal has reserved for the process in its memory management, included swapped pages (which are not in RAM, but on disk) and half-filled pages. It also includes files which get mapped into the memory region of the process, but are only loaded on-demand in RAM. All libraries loaded are memory-mapped and show up in the vsize.
resident set size (rss): The Rss is the memory which is actually occupied by the process in the physical RAM of the machine. It usually is significantly smaller than the vsize. This is the quantity which should actually be used when checking and optimizing the memory consumption of the application.
Definition at line 41 of file ProfileModule.h.
|
inherited |
Forward the EAfterConditionPath definition from the ModuleCondition.
|
private |
Signature of the lambda functions, which are used to extract the memory usage from teh MemTime structure.
Depending on the type of memory usage (rss, vsize), this lamdba function can contain different statements.
Definition at line 105 of file ProfileModule.h.
|
inherited |
Each module can be tagged with property flags, which indicate certain features of the module.
Enumerator | |
---|---|
c_Input | This module is an input module (reads data). |
c_Output | This module is an output module (writes data). |
c_ParallelProcessingCertified | This module can be run in parallel processing mode safely (All I/O must be done through the data store, in particular, the module must not write any files.) |
c_HistogramManager | This module is used to manage histograms accumulated by other modules. |
c_InternalSerializer | This module is an internal serializer/deserializer for parallel processing. |
c_TerminateInAllProcesses | When using parallel processing, call this module's terminate() function in all processes(). This will also ensure that there is exactly one process (single-core if no parallel modules found) or at least one input, one main and one output process. |
c_DontCollectStatistics | No statistics is collected for this module. |
Definition at line 77 of file Module.h.
ProfileModule | ( | ) |
Constructor.
Sets the description, the properties and the parameters of the module.
Definition at line 33 of file ProfileModule.cc.
|
inlinevirtualinherited |
Called when entering a new run.
Called at the beginning of each run, the method gives you the chance to change run dependent constants like alignment parameters, etc.
This method can be implemented by subclasses.
Reimplemented in EnergyBiasCorrectionModule, ChargedPidMVAModule, ChargedPidMVAMulticlassModule, CurlTaggerModule, LowEnergyPi0IdentificationExpertModule, LowEnergyPi0VetoExpertModule, ParticleVertexFitterModule, PhotonEfficiencySystematicsModule, TagVertexModule, TreeFitterModule, B2BIIMCParticlesMonitorModule, B2BIIConvertBeamParamsModule, B2BIIConvertMdstModule, B2BIIFixMdstModule, B2BIIMdstInputModule, BelleMCOutputModule, HistoModule, SubEventModule, SwitchDataStoreModule, EventInfoPrinterModule, EventLimiterModule, IoVDependentConditionModule, ProgressModule, RandomBarrierModule, GearboxModule, HistoManagerModule, StatisticsSummaryModule, SeqRootInputModule, SeqRootOutputModule, RxModule, TxModule, CreateFieldMapModule, ExportGeometryModule, MVAExpertModule, MVAMultipleExpertsModule, MVAPrototypeModule, AWESOMEBasicModule, and PyModule.
Definition at line 147 of file Module.h.
|
overridevirtualinherited |
Create an independent copy of this module.
Note that parameters are shared, so changing them on a cloned module will also affect the original module.
Implements PathElement.
Definition at line 179 of file Module.cc.
|
inlineprotectedvirtualinherited |
Wrapper method for the virtual function beginRun() that has the implementation to be used in a call from Python.
Reimplemented in PyModule.
Definition at line 426 of file Module.h.
|
inlineprotectedvirtualinherited |
This method can receive that the current run ends as a call from the Python side.
For regular C++-Modules that forwards the call to the regular endRun() method.
Reimplemented in PyModule.
Definition at line 439 of file Module.h.
|
inlineprotectedvirtualinherited |
|
inlineprotectedvirtualinherited |
Wrappers to make the methods without "def_" prefix callable from Python.
Overridden in PyModule. Wrapper method for the virtual function initialize() that has the implementation to be used in a call from Python.
Reimplemented in PyModule.
Definition at line 420 of file Module.h.
|
inlineprotectedvirtualinherited |
Wrapper method for the virtual function terminate() that has the implementation to be used in a call from Python.
Reimplemented in PyModule.
Definition at line 445 of file Module.h.
|
inlinevirtualinherited |
This method is called if the current run ends.
Use this method to store information, which should be aggregated over one run.
This method can be implemented by subclasses.
Reimplemented in CurlTaggerModule, LowEnergyPi0IdentificationExpertModule, LowEnergyPi0VetoExpertModule, B2BIIMCParticlesMonitorModule, B2BIIConvertMdstModule, B2BIIMdstInputModule, BelleMCOutputModule, HistoModule, SubEventModule, SwitchDataStoreModule, EventInfoPrinterModule, RandomBarrierModule, HistoManagerModule, StatisticsSummaryModule, SeqRootInputModule, SeqRootOutputModule, RxModule, TxModule, ZMQTxInputModule, ZMQTxWorkerModule, AWESOMEBasicModule, and PyModule.
Definition at line 166 of file Module.h.
|
inherited |
If at least one condition was set, it is evaluated and true returned if at least one condition returns true.
If no condition or result value was defined, the method returns false. Otherwise, the condition is evaluated and true returned, if at least one condition returns true. To speed up the evaluation, the condition strings were already parsed in the method if_value().
Definition at line 96 of file Module.cc.
|
overridevirtual |
Event profiling.
Records the event profiling information.
Reimplemented from Module.
Definition at line 70 of file ProfileModule.cc.
|
staticinherited |
Exposes methods of the Module class to Python.
Definition at line 325 of file Module.cc.
|
inherited |
What to do after the conditional path is finished.
(defaults to c_End if no condition is set)
Definition at line 133 of file Module.cc.
|
inherited |
|
inlineinherited |
Return all set conditions for this module.
|
inlineinherited |
|
inherited |
Returns the path of the last true condition (if there is at least one, else reaturn a null pointer).
Definition at line 113 of file Module.cc.
|
inlineinherited |
Returns the description of the module.
Definition at line 202 of file Module.h.
|
inlinevirtualinherited |
Return a list of output filenames for this modules.
This will be called when basf2 is run with "--dry-run" if the module has set either the c_Input or c_Output properties.
If the parameter outputFiles
is false (for modules with c_Input) the list of input filenames should be returned (if any). If outputFiles
is true (for modules with c_Output) the list of output files should be returned (if any).
If a module has sat both properties this member is called twice, once for each property.
The module should return the actual list of requested input or produced output filenames (including handling of input/output overrides) so that the grid system can handle input/output files correctly.
This function should return the same value when called multiple times. This is especially important when taking the input/output overrides from Environment as they get consumed when obtained so the finalized list of output files should be stored for subsequent calls.
Reimplemented in RootInputModule, and RootOutputModule.
Definition at line 134 of file Module.h.
|
inlineinherited |
|
inlineoverrideprivatevirtualinherited |
no submodules, return empty list
Implements PathElement.
Definition at line 506 of file Module.h.
|
inlineinherited |
Returns the name of the module.
This can be changed via e.g. set_name() in the steering file to give more useful names if there is more than one module of the same type.
For identifying the type of a module, using getType() (or type() in Python) is recommended.
Definition at line 187 of file Module.h.
|
inlineinherited |
|
inherited |
Returns a python list of all parameters.
Each item in the list consists of the name of the parameter, a string describing its type, a python list of all default values and the description of the parameter.
Definition at line 279 of file Module.cc.
|
inlineinherited |
|
overrideprivatevirtualinherited |
return the module name.
Implements PathElement.
Definition at line 192 of file Module.cc.
|
inlineinherited |
Return the return value set by this module.
This value is only meaningful if hasReturnValue() is true
Definition at line 381 of file Module.h.
|
inherited |
Returns the type of the module (i.e.
class name minus 'Module')
|
inlineinherited |
|
inherited |
Returns true if all specified property flags are available in this module.
propertyFlags | Ored EModulePropFlags which should be compared with the module flags. |
|
inlineinherited |
|
inherited |
Returns true and prints error message if the module has unset parameters which the user has to set in the steering file.
Definition at line 166 of file Module.cc.
|
inherited |
A simplified version to add a condition to the module.
Please note that successive calls of this function will add more than one condition to the module. If more than one condition results in true, only the last of them will be used.
Please be careful: Avoid creating cyclic paths, e.g. by linking a condition to a path which is processed before the path where this module is located in.
It is equivalent to the if_value() method, using the expression "<1". This method is meant to be used together with the setReturnValue(bool value) method.
path | Shared pointer to the Path which will be executed if the return value is false. |
afterConditionPath | What to do after executing 'path'. |
|
inherited |
A simplified version to set the condition of the module.
Please note that successive calls of this function will add more than one condition to the module. If more than one condition results in true, only the last of them will be used.
Please be careful: Avoid creating cyclic paths, e.g. by linking a condition to a path which is processed before the path where this module is located in.
It is equivalent to the if_value() method, using the expression ">=1". This method is meant to be used together with the setReturnValue(bool value) method.
path | Shared pointer to the Path which will be executed if the return value is true. |
afterConditionPath | What to do after executing 'path'. |
|
inherited |
Add a condition to the module.
Please note that successive calls of this function will add more than one condition to the module. If more than one condition results in true, only the last of them will be used.
See https://confluence.desy.de/display/BI/Software+ModCondTut or ModuleCondition for a description of the syntax.
Please be careful: Avoid creating cyclic paths, e.g. by linking a condition to a path which is processed before the path where this module is located in.
expression | The expression of the condition. |
path | Shared pointer to the Path which will be executed if the condition is evaluated to true. |
afterConditionPath | What to do after executing 'path'. |
|
overridevirtual |
Initializes the Module.
Creates the ProfileInfo objects in the data store and starts the profiling.
Reimplemented from Module.
Definition at line 43 of file ProfileModule.cc.
|
inherited |
Configure the abort log level.
Definition at line 67 of file Module.cc.
|
inherited |
Configure the debug messaging level.
Definition at line 61 of file Module.cc.
|
protectedinherited |
Sets the description of the module.
description | A description of the module. |
|
inlineinherited |
Set the log system configuration.
Definition at line 230 of file Module.h.
|
inherited |
Configure the printed log information for the given level.
logLevel | The log level (one of LogConfig::ELogLevel) |
logInfo | What kind of info should be printed? ORed combination of LogConfig::ELogInfo flags. |
Definition at line 73 of file Module.cc.
|
inherited |
Configure the log level.
|
inlineinherited |
|
inlineprotectedinherited |
Replace existing parameter list.
Definition at line 501 of file Module.h.
|
privateinherited |
Implements a method for setting boost::python objects.
The method supports the following types: list, dict, int, double, string, bool The conversion of the python object to the C++ type and the final storage of the parameter value is done in the ModuleParam class.
name | The unique name of the parameter. |
pyObj | The object which should be converted and stored as the parameter value. |
Definition at line 234 of file Module.cc.
|
privateinherited |
Implements a method for reading the parameter values from a boost::python dictionary.
The key of the dictionary has to be the name of the parameter and the value has to be of one of the supported parameter types.
dictionary | The python dictionary from which the parameter values are read. |
Definition at line 249 of file Module.cc.
|
inherited |
Sets the flags for the module properties.
propertyFlags | bitwise OR of EModulePropFlags |
|
protectedinherited |
Sets the return value for this module as bool.
The bool value is saved as an integer with the convention 1 meaning true and 0 meaning false. The value can be used in the steering file to divide the analysis chain into several paths.
value | The value of the return value. |
|
protectedinherited |
Sets the return value for this module as integer.
The value can be used in the steering file to divide the analysis chain into several paths.
value | The value of the return value. |
|
protectedinherited |
Set the module type.
Only for use by internal modules (which don't use the normal REG_MODULE mechanism).
|
private |
Stores the memory usage of the application over time in plots.
name | The name used to store the memory plot in the data store |
title | The title displayed on the memory plot |
xAxisName | Text displayed on the y-axis. Depending on the plotted quantitiy, either vsize or rss, this can vary. |
imgOutput | If this string is not empty a plot of the memory consumption over time is also written to disk using imgOutput as file name (must include file extension) |
lmdMemoryExtract | Lambda expression to extrat the plotted memory quantity from the MemTime data structure. Use either m_extractVirtualMem or m_extractRssMem here |
Definition at line 109 of file ProfileModule.cc.
|
overridevirtual |
Terminate the Module.
Records the final profiling information.
Reimplemented from Module.
Definition at line 149 of file ProfileModule.cc.
|
staticprivate |
number of events before the average time measurement is started
Definition at line 83 of file ProfileModule.h.
|
staticprivate |
maximal number of profile points
Definition at line 84 of file ProfileModule.h.
|
privateinherited |
|
privateinherited |
|
private |
memory usage and time at end of event loop
Definition at line 139 of file ProfileModule.h.
|
private |
memory usage and time at individual events
Definition at line 137 of file ProfileModule.h.
|
private |
Lambda expression to return the Rss memory from a MemTime data structure.
Definition at line 115 of file ProfileModule.h.
|
private |
Lambda expression to return the virtual memory from a MemTime data structure.
Definition at line 110 of file ProfileModule.h.
|
privateinherited |
|
private |
memory usage and time at initialization
Definition at line 133 of file ProfileModule.h.
|
privateinherited |
|
privateinherited |
|
privateinherited |
|
private |
event counter
Definition at line 135 of file ProfileModule.h.
|
private |
Name for image output file.
The name of the file for the memory usage plot.
Definition at line 75 of file ProfileModule.h.
|
privateinherited |
|
privateinherited |
|
private |
Name for rss image output file.
The name of the file for the rss memory usage plot.
Definition at line 81 of file ProfileModule.h.
|
private |
memory usage and time at start of event loop + burn in
Definition at line 138 of file ProfileModule.h.
|
private |
number of events between profile points
Definition at line 136 of file ProfileModule.h.
|
private |
memory usage and time at termination
Definition at line 134 of file ProfileModule.h.
|
private |
time at module creation
Definition at line 132 of file ProfileModule.h.
|
privateinherited |