VariablesToTable Class Reference

Inheritance diagram for VariablesToTable:

Public Member Functions
def	__init__ (self, str listname, List[str] variables, str filename, str format)
def	initialize (self)
def	initialize_parquet_writer (self)
def	initialize_csv_writer (self)
def	initialize_hdf5_writer (self)
def	fill_buffer (self)
def	event (self)
def	terminate (self)

Protected Attributes
	_filename
	Output filename.
	_listname
	Particle list name.
	_variables
	List of variables.
	_format
	Output format.
	_varnames
	variable names
	_var_objects
	variable objects for each variable
	_evtmeta
	Event metadata.
	_plist
	Pointer to the particle list.
	_dtypes
	The data type.
	_schema
	A list of tuples and py.DataTypes to define the pyarrow schema.
	_parquet_writer
	a writer object to write data into a parquet file
	_csv_writer
	a writer object to write data into a csv file
	_hdf5_writer
	The pytable file.
	_table
	The pytable.

Detailed Description

Base class to dump ntuples into a non root format of your choosing

Definition at line 37 of file b2pandas_utils.py.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
		str	listname,
		List[str]	variables,
		str	filename,
		str	format
	)

Constructor to initialize the internal state

Arguments:
    listname(str): name of the particle list
    variables(list(str)): list of variables to save for each particle
    filename(str): name of the output file to be created
    format(str): format of the output file, one of 'hdf5', 'parquet', 'csv'

Reimplemented in VariablesToHDF5.

Definition at line 42 of file b2pandas_utils.py.

    def __init__(self, listname: str, variables: List[str], filename: str, format: str):
        """Constructor to initialize the internal state
 
        Arguments:
            listname(str): name of the particle list
            variables(list(str)): list of variables to save for each particle
            filename(str): name of the output file to be created
            format(str): format of the output file, one of 'hdf5', 'parquet', 'csv'
        """
        super().__init__()
        
        self._filename = filename
        
        self._listname = listname
        
        self._variables = variables
        
        self._format = format
 

Member Function Documentation

event()

def event

(

self

)

Event processing function
executes the fill_buffer function and writes the data to the output file

Definition at line 157 of file b2pandas_utils.py.

    def event(self):
        """
        Event processing function
        executes the fill_buffer function and writes the data to the output file
        """
        buf = self.fill_buffer()
 
        if self._format == "hdf5":
            """Create a new row in the hdf5 file with for each particle in the list"""
            self._table.append(buf)
        elif self._format == "parquet":
            table = {name: buf[name] for name, _ in self._dtypes}
            pa_table = pa.table(table, schema=pa.schema(self._schema))
            self._parquet_writer.write_table(pa_table)
        elif self._format == "csv":
            table = {name: buf[name] for name, _ in self._dtypes}
            pa_table = pa.table(table, schema=pa.schema(self._schema))
            self._csv_writer.write(pa_table)
 

fill_buffer()

def fill_buffer

(

self

)

collect all variables for the particle in a numpy array

Definition at line 134 of file b2pandas_utils.py.

    def fill_buffer(self):
        """
        collect all variables for the particle in a numpy array
        """
 
        # create a numpy array with the data
        buf = np.empty(self._plist.getListSize(), dtype=self._dtypes)
        # add some extra columns for bookkeeping
        buf["__experiment__"] = self._evtmeta.getExperiment()
        buf["__run__"] = self._evtmeta.getRun()
        buf["__event__"] = self._evtmeta.getEvent()
        buf["__production__"] = self._evtmeta.getProduction()
        buf["__ncandidates__"] = len(buf)
        buf["__candidate__"] = np.arange(len(buf))
 
        for row, p in zip(buf, self._plist):
            for name, v in zip(self._varnames, self._var_objects):
                # pyroot proxy not working with callables, we should fix this.
                # For now we need to go back by name and call it.
                # should be `row[v.name] = v.func(p)`
                row[name] = variables.variables.evaluate(v.name, p)
        return buf
 

initialize()

def initialize

(

self

)

Create the hdf5 file and list of variable objects to be used during
event processing.

Definition at line 61 of file b2pandas_utils.py.

    def initialize(self):
        """Create the hdf5 file and list of variable objects to be used during
        event processing."""
        # Always avoid the top-level 'import ROOT'.
        import ROOT  # noqa
        
        self._varnames = [
            str(varname) for varname in variables.variables.resolveCollections(
                variables.std_vector(
                    *self._variables))]
        
        self._var_objects = [variables.variables.getVariable(n) for n in self._varnames]
 
        
        self._evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")
        self._evtmeta.isRequired()
        
        self._plist = ROOT.Belle2.PyStoreObj(self._listname)
        self._plist.isRequired()
 
        dtypes = [
            ("__experiment__", np.int32), ("__run__", np.int32), ("__event__", np.uint32),
            ("__production__", np.uint32), ("__candidate__", np.uint32), ("__ncandidates__", np.uint32)
        ]
        for name in self._varnames:
            # only float variables for now
            dtypes.append((name, np.float64))
 
        
        self._dtypes = dtypes
 
        if self._format == "hdf5":
            self.initialize_hdf5_writer()
        elif self._format == "parquet":
            self.initialize_parquet_writer()
        elif self._format == "csv":
            self.initialize_csv_writer()
        else:
            raise ValueError(f"Unknown format {self._format}, supported formats are 'hdf5', 'parquet', 'csv'.")
 

initialize_csv_writer()

def initialize_csv_writer

(

self

)

Initialize the csv writer using pyarrow

Definition at line 110 of file b2pandas_utils.py.

    def initialize_csv_writer(self):
        """
        Initialize the csv writer using pyarrow
        """
        
        self._schema = [(name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes]
        
        self._csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema))
 

initialize_hdf5_writer()

def initialize_hdf5_writer

(

self

)

Initialize the hdf5 writer using pytables

Definition at line 119 of file b2pandas_utils.py.

    def initialize_hdf5_writer(self):
        """
        Initialize the hdf5 writer using pytables
        """
        
        self._hdf5_writer = tables.open_file(self._filename, mode="w", title="Belle2 Variables to HDF5")
        filters = tables.Filters(complevel=1, complib='blosc:lz4', fletcher32=False)
 
        # some variable names are not just A-Za-z0-9 so pytables complains but
        # seems to work. Ignore warning
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            
            self._table = self._hdf5_writer.create_table("/", self._listname, obj=np.zeros(0, self._dtypes), filters=filters)
 

initialize_parquet_writer()

def initialize_parquet_writer

(

self

)

Initialize the parquet writer using pyarrow

Definition at line 101 of file b2pandas_utils.py.

    def initialize_parquet_writer(self):
        """
        Initialize the parquet writer using pyarrow
        """
        
        self._schema = [(name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes]
        
        self._parquet_writer = ParquetWriter(self._filename, schema=pa.schema(self._schema))
 

terminate()

def terminate

(

self

)

save and close the output

Definition at line 176 of file b2pandas_utils.py.

    def terminate(self):
        """save and close the output"""
        import ROOT  # noqa
        if self._format == "hdf5":
            self._table.flush()
            self._hdf5_writer.close()
        elif self._format == "parquet":
            self._parquet_writer.close()
        elif self._format == "csv":
            self._csv_writer.close()
        ROOT.Belle2.MetadataService.Instance().addNtuple(self._filename)
 
 

Member Data Documentation

_csv_writer

_csv_writer

protected

a writer object to write data into a csv file

Definition at line 117 of file b2pandas_utils.py.

_dtypes

_dtypes

protected

The data type.

Definition at line 90 of file b2pandas_utils.py.

_evtmeta

_evtmeta

protected

Event metadata.

Definition at line 75 of file b2pandas_utils.py.

_filename

_filename

protected

Output filename.

Definition at line 53 of file b2pandas_utils.py.

_format

_format

protected

Output format.

Definition at line 59 of file b2pandas_utils.py.

_hdf5_writer

_hdf5_writer

protected

The pytable file.

Definition at line 124 of file b2pandas_utils.py.

_listname

_listname

protected

Particle list name.

Definition at line 55 of file b2pandas_utils.py.

_parquet_writer

_parquet_writer

protected

a writer object to write data into a parquet file

Definition at line 108 of file b2pandas_utils.py.

_plist

_plist

protected

Pointer to the particle list.

Definition at line 78 of file b2pandas_utils.py.

_schema

_schema

protected

A list of tuples and py.DataTypes to define the pyarrow schema.

Definition at line 106 of file b2pandas_utils.py.

_table

_table

protected

The pytable.

Definition at line 132 of file b2pandas_utils.py.

_var_objects

_var_objects

protected

variable objects for each variable

Definition at line 72 of file b2pandas_utils.py.

_variables

_variables

protected

List of variables.

Definition at line 57 of file b2pandas_utils.py.

_varnames

_varnames

protected

variable names

Definition at line 67 of file b2pandas_utils.py.

---

The documentation for this class was generated from the following file:

• analysis/scripts/b2pandas_utils.py