VariablesToTable Class Reference

Inheritance diagram for VariablesToTable:

Collaboration diagram for VariablesToTable:

Public Member Functions
def	__init__ (self, str listname, List[str] variables, str filename, Optional[str] hdf_table_name=None, int event_buffer_size=100, **writer_kwargs)
def	initialize (self)
def	initialize_feather_writer (self)
def	initialize_parquet_writer (self)
def	initialize_csv_writer (self)
def	initialize_hdf5_writer (self)
def	fill_event_buffer (self)
def	fill_buffer (self)
def	write_buffer (self, buf)
def	event (self)
def	terminate (self)

Protected Attributes
	_filename
	Output filename.
	_listname
	Particle list name.
	_variables
	List of variables.
	_format
	Output format.
	_table_name
	Table name in the hdf5 file.
	_event_buffer_size
	Event buffer size.
	_buffer
	Event buffer.
	_event_buffer_counter
	Event buffer counter.
	_writer_kwargs
	writer kwargs
	_varnames
	variable names
	_var_objects
	variable objects for each variable
	_evtmeta
	Event metadata.
	_plist
	Pointer to the particle list.
	_dtypes
	The data type.
	_schema
	A list of tuples and py.DataTypes to define the pyarrow schema.
	_feather_writer
	a writer object to write data into a feather file
	_parquet_writer
	a writer object to write data into a parquet file
	_csv_writer
	a writer object to write data into a csv file
	_hdf5_writer
	The pytable file.
	_table
	The pytable.

Detailed Description

Base class to dump ntuples into a non root format of your choosing

Definition at line 38 of file b2pandas_utils.py.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
		str	listname,
		List[str]	variables,
		str	filename,
		Optional[str]	hdf_table_name = None,
		int	event_buffer_size = 100,
		**	writer_kwargs
	)

Constructor to initialize the internal state

Arguments:
    listname(str): name of the particle list
    variables(list(str)): list of variables to save for each particle
    filename(str): name of the output file to be created.
        Needs to end with `.csv` for csv output, `.parquet` or `.pq` for parquet output,
        `.h5`, `.hdf` or `.hdf5` for hdf5 output and `.feather` or `.arrow` for feather output
    hdf_table_name(str): name of the table in the hdf5 file.
        If not provided, it will be the same as the listname
    event_buffer_size(int): number of events to buffer before writing to disk,
        higher values will use more memory but write faster and result in smaller files
    **writer_kwargs: additional keyword arguments to pass to the writer.
        For details, see the documentation of the writer in the apache arrow documentation.
        Only use, if you know what you are doing!

Reimplemented in VariablesToHDF5.

Definition at line 43 of file b2pandas_utils.py.

    ):
        """Constructor to initialize the internal state
 
        Arguments:
            listname(str): name of the particle list
            variables(list(str)): list of variables to save for each particle
            filename(str): name of the output file to be created.
                Needs to end with `.csv` for csv output, `.parquet` or `.pq` for parquet output,
                `.h5`, `.hdf` or `.hdf5` for hdf5 output and `.feather` or `.arrow` for feather output
            hdf_table_name(str): name of the table in the hdf5 file.
                If not provided, it will be the same as the listname
            event_buffer_size(int): number of events to buffer before writing to disk,
                higher values will use more memory but write faster and result in smaller files
            **writer_kwargs: additional keyword arguments to pass to the writer.
                For details, see the documentation of the writer in the apache arrow documentation.
                Only use, if you know what you are doing!
        """
        super().__init__()
        
        self._filename = filename
        
        self._listname = listname
        
        self._variables = list(set(variables))
        
        file_type = self._filename.split(".")[-1]
        if file_type in ["csv"]:
            self._format = "csv"
        elif file_type in ["parquet", "pq"]:
            self._format = "parquet"
        elif file_type in ["h5", "hdf", "hdf5"]:
            self._format = "hdf5"
        elif file_type in ["feather", "arrow"]:
            self._format = "feather"
        else:
            raise ValueError(
                f"Unknown file type ending .{file_type}, supported types are 'csv', "
                "'parquet', 'pq', 'h5', 'hdf', 'hdf5', 'feather' or 'arrow'"
            )
        
        self._table_name = (
            hdf_table_name if hdf_table_name is not None else self._listname
        )
        
        self._event_buffer_size = event_buffer_size
        
        self._buffer = None
        
        self._event_buffer_counter = 0
        
        self._writer_kwargs = writer_kwargs
 

Member Function Documentation

event()

def event

(

self

)

Event processing function
executes the fill_buffer function and writes the data to the output file

Definition at line 264 of file b2pandas_utils.py.

    def event(self):
        """
        Event processing function
        executes the fill_buffer function and writes the data to the output file
        """
        buf = self.fill_buffer()
        if buf is None:
            return
        self.write_buffer(buf)
 

fill_buffer()

def fill_buffer

(

self

)

fill a buffer over multiple events and return it, when self.

Definition at line 231 of file b2pandas_utils.py.

    def fill_buffer(self):
        """
        fill a buffer over multiple events and return it, when self.
        """
        if self._event_buffer_counter == 0:
            self._buffer = self.fill_event_buffer()
        else:
            self._buffer = np.concatenate((self._buffer, self.fill_event_buffer()))
 
        self._event_buffer_counter += 1
        if self._event_buffer_counter == self._event_buffer_size:
            self._event_buffer_counter = 0
            return self._buffer
        return None
 

fill_event_buffer()

def fill_event_buffer

(

self

)

collect all variables for the particle in a numpy array

Definition at line 208 of file b2pandas_utils.py.

    def fill_event_buffer(self):
        """
        collect all variables for the particle in a numpy array
        """
 
        # create a numpy array with the data
        buf = np.empty(self._plist.getListSize(), dtype=self._dtypes)
        # add some extra columns for bookkeeping
        buf["__experiment__"] = self._evtmeta.getExperiment()
        buf["__run__"] = self._evtmeta.getRun()
        buf["__event__"] = self._evtmeta.getEvent()
        buf["__production__"] = self._evtmeta.getProduction()
        buf["__ncandidates__"] = len(buf)
        buf["__candidate__"] = np.arange(len(buf))
 
        for row, p in zip(buf, self._plist):
            for name, v in zip(self._varnames, self._var_objects):
                # pyroot proxy not working with callables, we should fix this.
                # For now we need to go back by name and call it.
                # should be `row[v.name] = v.func(p)`
                row[name] = variables.variables.evaluate(v.name, p)
        return buf
 

initialize()

def initialize

(

self

)

Create the hdf5 file and list of variable objects to be used during
event processing.

Definition at line 103 of file b2pandas_utils.py.

    def initialize(self):
        """Create the hdf5 file and list of variable objects to be used during
        event processing."""
        # Always avoid the top-level 'import ROOT'.
        import ROOT  # noqa
 
        
        self._varnames = [
            str(varname)
            for varname in variables.variables.resolveCollections(
                variables.std_vector(*self._variables)
            )
        ]
        
        self._var_objects = [variables.variables.getVariable(n) for n in self._varnames]
 
        
        self._evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")
        self._evtmeta.isRequired()
        
        self._plist = ROOT.Belle2.PyStoreObj(self._listname)
        self._plist.isRequired()
 
        dtypes = [
            ("__experiment__", np.int32),
            ("__run__", np.int32),
            ("__event__", np.uint32),
            ("__production__", np.uint32),
            ("__candidate__", np.uint32),
            ("__ncandidates__", np.uint32),
        ]
        for name in self._varnames:
            # only float variables for now
            dtypes.append((name, np.float64))
 
        
        self._dtypes = dtypes
 
        if self._format == "hdf5":
            self.initialize_hdf5_writer()
        elif self._format == "parquet":
            self.initialize_parquet_writer()
        elif self._format == "csv":
            self.initialize_csv_writer()
        elif self._format == "feather":
            self.initialize_feather_writer()
 

initialize_csv_writer()

def initialize_csv_writer

(

self

)

Initialize the csv writer using pyarrow

Definition at line 178 of file b2pandas_utils.py.

    def initialize_csv_writer(self):
        """
        Initialize the csv writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema), **self._writer_kwargs)
 

initialize_feather_writer()

def initialize_feather_writer

(

self

)

Initialize the feather writer using pyarrow

Definition at line 150 of file b2pandas_utils.py.

    def initialize_feather_writer(self):
        """
        Initialize the feather writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._feather_writer = ipc.RecordBatchFileWriter(
            sink=self._filename,
            schema=pa.schema(self._schema),
            **self._writer_kwargs
        )
 

initialize_hdf5_writer()

def initialize_hdf5_writer

(

self

)

Initialize the hdf5 writer using pytables

Definition at line 189 of file b2pandas_utils.py.

    def initialize_hdf5_writer(self):
        """
        Initialize the hdf5 writer using pytables
        """
        
        self._hdf5_writer = tables.open_file(
            self._filename, mode="w", title="Belle2 Variables to HDF5"
        )
        filters = tables.Filters(complevel=1, complib="blosc:lz4", fletcher32=False)
 
        # some variable names are not just A-Za-z0-9 so pytables complains but
        # seems to work. Ignore warning
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            
            self._table = self._hdf5_writer.create_table(
                "/", self._table_name, obj=np.zeros(0, self._dtypes), filters=filters, **self._writer_kwargs
            )
 

initialize_parquet_writer()

def initialize_parquet_writer

(

self

)

Initialize the parquet writer using pyarrow

Definition at line 165 of file b2pandas_utils.py.

    def initialize_parquet_writer(self):
        """
        Initialize the parquet writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._parquet_writer = ParquetWriter(
            self._filename, schema=pa.schema(self._schema), **self._writer_kwargs
        )
 

terminate()

def terminate

(

self

)

save and close the output

Definition at line 274 of file b2pandas_utils.py.

    def terminate(self):
        """save and close the output"""
        import ROOT  # noqa
        if self._event_buffer_counter > 0:
            self.write_buffer(self._buffer)
 
        if self._format == "hdf5":
            self._table.flush()
            self._hdf5_writer.close()
        elif self._format == "parquet":
            self._parquet_writer.close()
        elif self._format == "csv":
            self._csv_writer.close()
        elif self._format == "feather":
            self._feather_writer.close()
        ROOT.Belle2.MetadataService.Instance().addNtuple(self._filename)
 
 

write_buffer()

def write_buffer	(		self,
			buf
	)

write the buffer to the output file

Definition at line 246 of file b2pandas_utils.py.

    def write_buffer(self, buf):
        """
        write the buffer to the output file
        """
 
        if self._format == "hdf5":
            """Create a new row in the hdf5 file with for each particle in the list"""
            self._table.append(buf)
        else:
            table = {name: buf[name] for name, _ in self._dtypes}
            pa_table = pa.table(table, schema=pa.schema(self._schema))
            if self._format == "parquet":
                self._parquet_writer.write_table(pa_table)
            elif self._format == "csv":
                self._csv_writer.write(pa_table)
            elif self._format == "feather":
                self._feather_writer.write_table(pa_table)
 

Member Data Documentation

_buffer

_buffer

protected

Event buffer.

Definition at line 97 of file b2pandas_utils.py.

_csv_writer

_csv_writer

protected

a writer object to write data into a csv file

Definition at line 187 of file b2pandas_utils.py.

_dtypes

_dtypes

protected

The data type.

Definition at line 139 of file b2pandas_utils.py.

_event_buffer_counter

_event_buffer_counter

protected

Event buffer counter.

Definition at line 99 of file b2pandas_utils.py.

_event_buffer_size

_event_buffer_size

protected

Event buffer size.

Definition at line 95 of file b2pandas_utils.py.

_evtmeta

_evtmeta

protected

Event metadata.

Definition at line 120 of file b2pandas_utils.py.

_feather_writer

_feather_writer

protected

a writer object to write data into a feather file

Definition at line 159 of file b2pandas_utils.py.

_filename

_filename

protected

Output filename.

Definition at line 70 of file b2pandas_utils.py.

_format

_format

protected

Output format.

Definition at line 78 of file b2pandas_utils.py.

_hdf5_writer

_hdf5_writer

protected

The pytable file.

Definition at line 194 of file b2pandas_utils.py.

_listname

_listname

protected

Particle list name.

Definition at line 72 of file b2pandas_utils.py.

_parquet_writer

_parquet_writer

protected

a writer object to write data into a parquet file

Definition at line 174 of file b2pandas_utils.py.

_plist

_plist

protected

Pointer to the particle list.

Definition at line 123 of file b2pandas_utils.py.

_schema

_schema

protected

A list of tuples and py.DataTypes to define the pyarrow schema.

Definition at line 155 of file b2pandas_utils.py.

_table

_table

protected

The pytable.

Definition at line 204 of file b2pandas_utils.py.

_table_name

_table_name

protected

Table name in the hdf5 file.

Definition at line 91 of file b2pandas_utils.py.

_var_objects

_var_objects

protected

variable objects for each variable

Definition at line 117 of file b2pandas_utils.py.

_variables

_variables

protected

List of variables.

Definition at line 74 of file b2pandas_utils.py.

_varnames

_varnames

protected

variable names

Definition at line 110 of file b2pandas_utils.py.

_writer_kwargs

_writer_kwargs

protected

writer kwargs

Definition at line 101 of file b2pandas_utils.py.

---

The documentation for this class was generated from the following file:

• analysis/scripts/b2pandas_utils.py