Belle II Software light-2406-ragdoll
MetadataService Class Reference

This class provides a service for writing metadata about the basf2 execution and about output files to a json file. More...

#include <MetadataService.h>

Collaboration diagram for MetadataService:

Public Member Functions

void setJsonFileName (const std::string &fileName)
 Set the name of the json metadata file.
 
void addRootOutputFile (const std::string &fileName, const FileMetaData *metaData=nullptr)
 Add the metadata of a root output file.
 
void addRootNtupleFile (const std::string &fileName)
 Add the metadata of a root ntuple file.
 
void addHDF5File (const std::string &fileName)
 Add the metadata of a HDF5 file.
 
void addBasf2Status (const std::string &message="")
 Add metadata of basf2 status.
 
void addModuleCallsCount ()
 Add the metadata of number of calls of all modules.
 
void finishBasf2 (bool success=true)
 Add metadata for basf2 completion.
 

Static Public Member Functions

static MetadataServiceInstance ()
 Static method to get a reference to the MetadataService instance.
 

Private Member Functions

 MetadataService ()
 Constructor.
 
 MetadataService (const MetadataService &)=delete
 Disable/Hide the copy constructor.
 
MetadataServiceoperator= (const MetadataService &)=delete
 Disable/Hide the copy assignment operator.
 
void writeJson ()
 Serialize the current json content to the json file.
 

Private Attributes

std::string m_fileName
 The name of the json file.
 
nlohmann::json m_json
 The json object.
 
double m_basf2StartTime
 the start time of basf2
 

Detailed Description

This class provides a service for writing metadata about the basf2 execution and about output files to a json file.

Definition at line 27 of file MetadataService.h.

Constructor & Destructor Documentation

◆ MetadataService()

MetadataService ( )
private

Constructor.

Definition at line 21 of file MetadataService.cc.

22{
23 m_json["basf2_apiversion"] = 1;
24}
double m_basf2StartTime
the start time of basf2
nlohmann::json m_json
The json object.
double getClock()
Return current value of the real-time clock.
Definition: Utils.cc:66

Member Function Documentation

◆ addBasf2Status()

void addBasf2Status ( const std::string &  message = "")

Add metadata of basf2 status.

Definition at line 89 of file MetadataService.cc.

90{
91 if (m_fileName.empty()) return;
92 auto& status = m_json["basf2_status"];
93 status["elapsed_walltime_sec"] = (Utils::getClock() - m_basf2StartTime) / Unit::s;
94 status["resident_memory_mb"] = Utils::getRssMemoryKB() / 1024;
96 if (processStatistics.isValid()) {
97 const auto& stats = processStatistics->getGlobal();
98 status["runs_processed"] = int(stats.getCalls(ModuleStatistics::EStatisticCounters::c_BeginRun));
99 status["events_processed"] = int(stats.getCalls());
100 }
101 if ((status.count("total_events") == 0) || (status["total_events"] == 0)) {
102 status["total_events"] = Environment::Instance().getNumberOfEvents();
103 }
107 status["finished"] = false;
108 status["message"] = message;
109
110 writeJson();
111}
@ c_Persistent
Object is available during entire execution time.
Definition: DataStore.h:60
unsigned int getNumberOfEvents() const
Return the number of events, from either input or EventInfoSetter, or -n command line override (if le...
Definition: Environment.cc:39
static Environment & Instance()
Static method to get a reference to the Environment instance.
Definition: Environment.cc:28
@ c_Error
Error: for things that went wrong and have to be fixed.
Definition: LogConfig.h:30
@ c_Fatal
Fatal: for situations were the program execution can not be continued.
Definition: LogConfig.h:31
@ c_Warning
Warning: for potential problems that the user should pay attention to.
Definition: LogConfig.h:29
int getMessageCounter(LogConfig::ELogLevel logLevel) const
Returns the number of logging calls per log level.
Definition: LogSystem.cc:161
static LogSystem & Instance()
Static method to get a reference to the LogSystem instance.
Definition: LogSystem.cc:31
void writeJson()
Serialize the current json content to the json file.
std::string m_fileName
The name of the json file.
@ c_BeginRun
Counting time/calls in beginRun()
Type-safe access to single objects in the data store.
Definition: StoreObjPtr.h:96
static const double s
[second]
Definition: Unit.h:95
unsigned long getRssMemoryKB()
Returns the amount of memory the process actually occupies in the physical RAM of the machine.
Definition: Utils.cc:84

◆ addHDF5File()

void addHDF5File ( const std::string &  fileName)

Add the metadata of a HDF5 file.

Definition at line 75 of file MetadataService.cc.

76{
77 if (m_fileName.empty()) return;
78 if (!FileSystem::isFile(fileName)) return;
79
80 nlohmann::json file_json = {{"type", "HDF5"}, {"filename", fileName}};
81
82 file_json["checksums"]["adler32"] = FileSystem::calculateAdler32(fileName);
83
84 m_json["output_files"].push_back(file_json);
85
86 writeJson();
87}
static bool isFile(const std::string &filename)
Check if filename points to an existing file.
Definition: FileSystem.cc:45
static std::string calculateAdler32(const std::string &filename)
Calculate the Adler-32 checksum of a given file.
Definition: FileSystem.cc:86

◆ addModuleCallsCount()

void addModuleCallsCount ( )

Add the metadata of number of calls of all modules.

Definition at line 113 of file MetadataService.cc.

114{
115 if (m_fileName.empty()) return;
116
118
119 if (processStatistics.isValid()) {
120
121 std::vector<ModuleStatistics> modulesSortedByIndex(processStatistics->getAll());
122 sort(modulesSortedByIndex.begin(), modulesSortedByIndex.end(), [](const ModuleStatistics & a, const ModuleStatistics & b) { return a.getIndex() < b.getIndex(); });
123
124 for (const ModuleStatistics& stats : modulesSortedByIndex) {
125 m_json["modules_calls"][stats.getName()] = int(stats.getCalls(ModuleStatistics::EStatisticCounters::c_Event));
126 }
127 }
128
129 writeJson();
130}
Keep track of time and memory consumption during processing.
@ c_Event
Counting time/calls in event()

◆ addRootNtupleFile()

void addRootNtupleFile ( const std::string &  fileName)

Add the metadata of a root ntuple file.

Definition at line 57 of file MetadataService.cc.

58{
59 if (m_fileName.empty()) return;
60 if (!FileSystem::isFile(fileName)) return;
61
62 nlohmann::json file_json = {{"type", "RootNtuple"}, {"filename", fileName}};
63
64 // no metadata and no check
65
66 file_json["checksums"]["md5"] = FileSystem::calculateMD5(fileName);
67 file_json["checksums"]["adler32"] = FileSystem::calculateAdler32(fileName);
68 // no sha256 yet
69
70 m_json["output_files"].push_back(file_json);
71
72 writeJson();
73}
static std::string calculateMD5(const std::string &filename)
Calculate the MD5 checksum of a given file.
Definition: FileSystem.cc:78

◆ addRootOutputFile()

void addRootOutputFile ( const std::string &  fileName,
const FileMetaData metaData = nullptr 
)

Add the metadata of a root output file.

Definition at line 32 of file MetadataService.cc.

33{
34 if (m_fileName.empty()) return;
35 if (!FileSystem::isFile(fileName)) return;
36
37 nlohmann::json file_json = {{"type", "RootOutput"}, {"filename", fileName}};
38
39 if (metaData) {
40 file_json["metadata"] = nlohmann::json::parse(metaData->getJsonStr());
41 }
42
43 try {
44 std::string check = Utils::getCommandOutput("b2file-check", {"--json", fileName});
45 file_json.merge_patch(nlohmann::json::parse(check));
46 } catch (...) {}
47
48 file_json["checksums"]["md5"] = FileSystem::calculateMD5(fileName);
49 file_json["checksums"]["adler32"] = FileSystem::calculateAdler32(fileName);
50 // no sha256 yet
51
52 m_json["output_files"].push_back(file_json);
53
54 writeJson();
55}
std::string getJsonStr() const
Get a json representation.
std::string getCommandOutput(const std::string &command, const std::vector< std::string > &arguments={}, bool searchPath=true)
Execute a shell command and return its output.
Definition: Utils.cc:100

◆ finishBasf2()

void finishBasf2 ( bool  success = true)

Add metadata for basf2 completion.

Definition at line 132 of file MetadataService.cc.

133{
134 m_json["basf2_status"]["finished"] = true;
135 m_json["basf2_status"]["success"] = success;
136
137 writeJson();
138}

◆ Instance()

MetadataService & Instance ( )
static

Static method to get a reference to the MetadataService instance.

Returns
A reference to an instance of this class.

Definition at line 26 of file MetadataService.cc.

27{
28 static MetadataService instance;
29 return instance;
30}
This class provides a service for writing metadata about the basf2 execution and about output files t...

◆ setJsonFileName()

void setJsonFileName ( const std::string &  fileName)
inline

Set the name of the json metadata file.

Parameters
fileNamename of the json file

Definition at line 43 of file MetadataService.h.

43{m_fileName = fileName; writeJson();};

◆ writeJson()

void writeJson ( )
private

Serialize the current json content to the json file.

Definition at line 140 of file MetadataService.cc.

141{
142 if (m_fileName.empty()) return;
143 std::ofstream jsonFile(m_fileName.c_str());
144 jsonFile << m_json.dump(2) << std::endl;
145}

Member Data Documentation

◆ m_basf2StartTime

double m_basf2StartTime
private

the start time of basf2

Definition at line 79 of file MetadataService.h.

◆ m_fileName

std::string m_fileName
private

The name of the json file.

Definition at line 77 of file MetadataService.h.

◆ m_json

nlohmann::json m_json
private

The json object.

Definition at line 78 of file MetadataService.h.


The documentation for this class was generated from the following files: