VariablesToHDF5 Class Reference

Inheritance diagram for VariablesToHDF5:

Collaboration diagram for VariablesToHDF5:

Public Member Functions
	__init__ (self, listname, variables, filename, Optional[str] hdf_table_name=None)
	buffer (self)
	buffer_full (self)
	initialize (self)
	event_buffer (self)
	clear_buffer (self)
	append_buffer (self)
	initialize_feather_writer (self)
	initialize_parquet_writer (self)
	initialize_csv_writer (self)
	initialize_hdf5_writer (self)
	fill_event_buffer (self)
	write_buffer (self)
	event (self)
	terminate (self)

Public Attributes
	buffer = "hdf5":
	buffer_full

Protected Attributes
	_filename = filename
	Output filename.
	_listname = listname
	Particle list name.
	_variables = list(set(variables))
	List of variables.
str	_format = "csv"
	Output format.
tuple	_table_name
	Table name in the hdf5 file.
	_event_buffer_size = event_buffer_size
	Event buffer size.
int	_event_buffer_counter = 0
	Event buffer counter.
	_writer_kwargs = writer_kwargs
	writer kwargs
list	_varnames
	variable names
	_std_varnames = variables.std_vector(*self._varnames)
	std.vector of variable names
	_evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")
	Event metadata.
	_plist = ROOT.Belle2.PyStoreObj(self._listname)
	Pointer to the particle list.
	_dtypes = dtypes
	The data type.
	_buffer = np.empty(self._event_buffer_size * 10, dtype=self._dtypes)
	event variables buffer (will be automatically grown if necessary)
int	_buffer_index = 0
	current start index in the event variables buffer
list	_schema
	A list of tuples and py.DataTypes to define the pyarrow schema.
	_feather_writer
	a writer object to write data into a feather file
	_parquet_writer
	A list of tuples and py.DataTypes to define the pyarrow schema.
	_csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema), **self._writer_kwargs)
	A list of tuples and py.DataTypes to define the pyarrow schema.
	_hdf5_writer
	The pytable file.
	_table
	The pytable.

Detailed Description

Legacy class to not break existing code

Definition at line 330 of file b2pandas_utils.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
			listname,
			variables,
			filename,
		Optional[str]	hdf_table_name = None )

Definition at line 335 of file b2pandas_utils.py.

    def __init__(self, listname, variables, filename, hdf_table_name: Optional[str] = None,):
        super().__init__(listname, variables, filename, hdf_table_name)
        assert self._filename.split(".")[-1] in ["h5", "hdf", "hdf5"], (
            "Filename must end with .h5, .hdf or .hdf5 for HDF5 output. "
            f"Got {self._filename}"
        )
 
 

Member Function Documentation

append_buffer()

append_buffer ( self )

inherited

"Append" a new event to the buffer by moving the buffer index forward by particle list size

Automatically replaces the buffer by a larger one if necessary

Definition at line 179 of file b2pandas_utils.py.

    def append_buffer(self):
        """
        "Append" a new event to the buffer by moving the buffer index forward by particle list size
 
        Automatically replaces the buffer by a larger one if necessary
        """
        plist_size = self._plist.getListSize()
        if (plist_size + self._buffer_index) > len(self._buffer):
            new_buffer = np.empty(
                # factor 1.5 larger or at least as large as necessary
                max(int(len(self._buffer) * 1.5), self._buffer_index + plist_size),
                dtype=self._dtypes,
            )
            new_buffer[:self._buffer_index] = self.buffer
            self._buffer = new_buffer
        self._buffer_index += plist_size
        self._event_buffer_counter += 1
 

buffer()

buffer ( self )

inherited

The buffer slice across multiple entries

Definition at line 159 of file b2pandas_utils.py.

    def buffer(self):
        """
        The buffer slice across multiple entries
        """
        return self._buffer[:self._buffer_index]
 

buffer_full()

buffer_full ( self )

inherited

check if the buffer is full

Definition at line 276 of file b2pandas_utils.py.

    def buffer_full(self):
        """
        check if the buffer is full
        """
        return self._event_buffer_counter == self._event_buffer_size
 

clear_buffer()

clear_buffer ( self )

inherited

Reset the buffer event counter and index

Definition at line 172 of file b2pandas_utils.py.

    def clear_buffer(self):
        """
        Reset the buffer event counter and index
        """
        self._event_buffer_counter = 0
        self._buffer_index = 0
 

event()

event ( self )

inherited

Event processing function

executes the fill_buffer function and writes the data to the output file
in chunks of event_buffer_size

Definition at line 299 of file b2pandas_utils.py.

    def event(self):
        """
        Event processing function
 
        executes the fill_buffer function and writes the data to the output file
        in chunks of event_buffer_size
        """
        self.append_buffer()
        self.fill_event_buffer()
        if self.buffer_full:
            self.write_buffer()
            self.clear_buffer()
 

event_buffer()

event_buffer ( self )

inherited

The buffer slice for the current event

Definition at line 166 of file b2pandas_utils.py.

    def event_buffer(self):
        """
        The buffer slice for the current event
        """
        return self._buffer[self._buffer_index - self._plist.getListSize(): self._buffer_index]
 

fill_event_buffer()

fill_event_buffer ( self )

inherited

Assign values for all variables for all particles in the particle list to the current event buffer

Definition at line 255 of file b2pandas_utils.py.

    def fill_event_buffer(self):
        """
        Assign values for all variables for all particles in the particle list to the current event buffer
        """
        buf = self.event_buffer
 
        # add some extra columns for bookkeeping
        buf["__experiment__"] = self._evtmeta.getExperiment()
        buf["__run__"] = self._evtmeta.getRun()
        buf["__event__"] = self._evtmeta.getEvent()
        buf["__production__"] = self._evtmeta.getProduction()
        buf["__ncandidates__"] = len(buf)
        buf["__candidate__"] = np.arange(len(buf))
 
        # fill variables into buffer
        vector = variables.variables.evaluateVariables(self._std_varnames, self._plist)
        values = np.array(vector.data()).reshape(-1, len(self._varnames))
        for name, col in zip(self._varnames, values.T):
            buf[name] = col
 

initialize()

initialize ( self )

inherited

Setup variable lists, pointers, buffers and file writers

Definition at line 102 of file b2pandas_utils.py.

    def initialize(self):
        """
        Setup variable lists, pointers, buffers and file writers
        """
        # Always avoid the top-level 'import ROOT'.
        import ROOT  # noqa
 
        
        self._varnames = [
            str(varname)
            for varname in variables.variables.resolveCollections(
                variables.std_vector(*self._variables)
            )
        ]
 
        
        self._std_varnames = variables.std_vector(*self._varnames)
 
        
        self._evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")
        self._evtmeta.isRequired()
 
        
        self._plist = ROOT.Belle2.PyStoreObj(self._listname)
        self._plist.isRequired()
 
        dtypes = [
            ("__experiment__", np.int32),
            ("__run__", np.int32),
            ("__event__", np.uint32),
            ("__production__", np.uint32),
            ("__candidate__", np.uint32),
            ("__ncandidates__", np.uint32),
        ]
        for name in self._varnames:
            # only float variables for now
            dtypes.append((name, np.float64))
 
        
        self._dtypes = dtypes
 
        
        self._buffer = np.empty(self._event_buffer_size * 10, dtype=self._dtypes)
 
        
        self._buffer_index = 0
 
        if self._format == "hdf5":
            self.initialize_hdf5_writer()
        elif self._format == "parquet":
            self.initialize_parquet_writer()
        elif self._format == "csv":
            self.initialize_csv_writer()
        elif self._format == "feather":
            self.initialize_feather_writer()
 

initialize_csv_writer()

initialize_csv_writer ( self )

inherited

Initialize the csv writer using pyarrow

Definition at line 225 of file b2pandas_utils.py.

    def initialize_csv_writer(self):
        """
        Initialize the csv writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema), **self._writer_kwargs)
 

initialize_feather_writer()

initialize_feather_writer ( self )

inherited

Initialize the feather writer using pyarrow

Definition at line 197 of file b2pandas_utils.py.

    def initialize_feather_writer(self):
        """
        Initialize the feather writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._feather_writer = ipc.RecordBatchFileWriter(
            sink=self._filename,
            schema=pa.schema(self._schema),
            **self._writer_kwargs,
        )
 

initialize_hdf5_writer()

initialize_hdf5_writer ( self )

inherited

Initialize the hdf5 writer using pytables

Definition at line 236 of file b2pandas_utils.py.

    def initialize_hdf5_writer(self):
        """
        Initialize the hdf5 writer using pytables
        """
        
        self._hdf5_writer = tables.open_file(
            self._filename, mode="w", title="Belle2 Variables to HDF5"
        )
        filters = tables.Filters(complevel=1, complib="blosc:lz4", fletcher32=False)
 
        # some variable names are not just A-Za-z0-9 so pytables complains but
        # seems to work. Ignore warning
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            
            self._table = self._hdf5_writer.create_table(
                "/", self._table_name, obj=np.zeros(0, self._dtypes), filters=filters, **self._writer_kwargs
            )
 

initialize_parquet_writer()

initialize_parquet_writer ( self )

inherited

Initialize the parquet writer using pyarrow

Definition at line 212 of file b2pandas_utils.py.

    def initialize_parquet_writer(self):
        """
        Initialize the parquet writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._parquet_writer = ParquetWriter(
            self._filename, schema=pa.schema(self._schema), **self._writer_kwargs
        )
 

terminate()

terminate ( self )

inherited

save and close the output

Definition at line 312 of file b2pandas_utils.py.

    def terminate(self):
        """save and close the output"""
        import ROOT  # noqa
        if len(self.buffer) > 0:
            self.write_buffer()
 
        if self._format == "hdf5":
            self._table.flush()
            self._hdf5_writer.close()
        elif self._format == "parquet":
            self._parquet_writer.close()
        elif self._format == "csv":
            self._csv_writer.close()
        elif self._format == "feather":
            self._feather_writer.close()
        ROOT.Belle2.MetadataService.Instance().addNtuple(self._filename)
 
 

write_buffer()

write_buffer ( self )

inherited

write the buffer to the output file

Definition at line 282 of file b2pandas_utils.py.

    def write_buffer(self):
        """
        write the buffer to the output file
        """
        if self._format == "hdf5":
            """Create a new row in the hdf5 file with for each particle in the list"""
            self._table.append(self.buffer)
        else:
            table = {name: self.buffer[name] for name, _ in self._dtypes}
            pa_table = pa.table(table, schema=pa.schema(self._schema))
            if self._format == "parquet":
                self._parquet_writer.write_table(pa_table)
            elif self._format == "csv":
                self._csv_writer.write(pa_table)
            elif self._format == "feather":
                self._feather_writer.write_table(pa_table)
 

Member Data Documentation

_buffer

_buffer = np.empty(self._event_buffer_size * 10, dtype=self._dtypes)

protectedinherited

event variables buffer (will be automatically grown if necessary)

Definition at line 144 of file b2pandas_utils.py.

_buffer_index

_buffer_index = 0

protectedinherited

current start index in the event variables buffer

Definition at line 147 of file b2pandas_utils.py.

_csv_writer

_csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema), **self._writer_kwargs)

protectedinherited

A list of tuples and py.DataTypes to define the pyarrow schema.

a writer object to write data into a csv file

Definition at line 234 of file b2pandas_utils.py.

_dtypes

_dtypes = dtypes

protectedinherited

The data type.

Definition at line 141 of file b2pandas_utils.py.

_event_buffer_counter

int _event_buffer_counter = 0

protectedinherited

Event buffer counter.

Definition at line 98 of file b2pandas_utils.py.

_event_buffer_size

_event_buffer_size = event_buffer_size

protectedinherited

Event buffer size.

Definition at line 96 of file b2pandas_utils.py.

_evtmeta

_evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")

protectedinherited

Event metadata.

Definition at line 121 of file b2pandas_utils.py.

_feather_writer

_feather_writer

protectedinherited

Initial value:

=  ipc.RecordBatchFileWriter(
            sink=self._filename,
            schema=pa.schema(self._schema),
            **self._writer_kwargs,
        )

a writer object to write data into a feather file

Definition at line 206 of file b2pandas_utils.py.

_filename

_filename = filename

protectedinherited

Output filename.

Definition at line 70 of file b2pandas_utils.py.

_format

str _format = "csv"

protectedinherited

Output format.

Definition at line 79 of file b2pandas_utils.py.

_hdf5_writer

_hdf5_writer

protectedinherited

Initial value:

=  tables.open_file(
            self._filename, mode="w", title="Belle2 Variables to HDF5"
        )

The pytable file.

Definition at line 241 of file b2pandas_utils.py.

_listname

_listname = listname

protectedinherited

Particle list name.

Definition at line 72 of file b2pandas_utils.py.

_parquet_writer

_parquet_writer

protectedinherited

Initial value:

=  ParquetWriter(
            self._filename, schema=pa.schema(self._schema), **self._writer_kwargs
        )

A list of tuples and py.DataTypes to define the pyarrow schema.

a writer object to write data into a parquet file

Definition at line 221 of file b2pandas_utils.py.

_plist

_plist = ROOT.Belle2.PyStoreObj(self._listname)

protectedinherited

Pointer to the particle list.

Definition at line 125 of file b2pandas_utils.py.

_schema

list _schema

protectedinherited

Initial value:

=  [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]

A list of tuples and py.DataTypes to define the pyarrow schema.

Definition at line 202 of file b2pandas_utils.py.

_std_varnames

_std_varnames = variables.std_vector(*self._varnames)

protectedinherited

std.vector of variable names

Definition at line 118 of file b2pandas_utils.py.

_table

_table

protectedinherited

Initial value:

=  self._hdf5_writer.create_table(
                "/", self._table_name, obj=np.zeros(0, self._dtypes), filters=filters, **self._writer_kwargs
            )

The pytable.

Definition at line 251 of file b2pandas_utils.py.

_table_name

tuple _table_name

protectedinherited

Initial value:

=  (
            hdf_table_name if hdf_table_name is not None else self._listname
        )

Table name in the hdf5 file.

Definition at line 92 of file b2pandas_utils.py.

_variables

_variables = list(set(variables))

protectedinherited

List of variables.

Definition at line 74 of file b2pandas_utils.py.

_varnames

_varnames

protectedinherited

Initial value:

=  [
            str(varname)
            for varname in variables.variables.resolveCollections(
                variables.std_vector(*self._variables)
            )
        ]

variable names

Definition at line 110 of file b2pandas_utils.py.

_writer_kwargs

_writer_kwargs = writer_kwargs

protectedinherited

writer kwargs

Definition at line 100 of file b2pandas_utils.py.

buffer

buffer = "hdf5":

inherited

Definition at line 288 of file b2pandas_utils.py.

buffer_full

buffer_full

inherited

Definition at line 308 of file b2pandas_utils.py.

---

The documentation for this class was generated from the following file:

• analysis/scripts/b2pandas_utils.py