Inheritance diagram for VariablesToTable:

Public Member Functions
	__init__ (self, str listname, List[str] variables, str filename, Optional[str] hdf_table_name=None, int event_buffer_size=100, **writer_kwargs)

	initialize (self)

	buffer (self)

	event_buffer (self)

	clear_buffer (self)

	append_buffer (self)

	initialize_feather_writer (self)

	initialize_parquet_writer (self)

	initialize_csv_writer (self)

	initialize_hdf5_writer (self)

	fill_event_buffer (self)

	buffer_full (self)

	write_buffer (self)

	event (self)

	terminate (self)

Public Attributes
	buffer = "hdf5":

	buffer_full

Protected Attributes
	_filename = filename
	Output filename.

	_listname = listname
	Particle list name.

	_variables = list(set(variables))
	List of variables.

str	_format = "csv"
	Output format.

tuple	_table_name
	Table name in the hdf5 file.

	_event_buffer_size = event_buffer_size
	Event buffer size.

int	_event_buffer_counter = 0
	Event buffer counter.

	_writer_kwargs = writer_kwargs
	writer kwargs

list	_varnames
	variable names

	_std_varnames = variables.std_vector(*self._varnames)
	std.vector of variable names

	_evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")
	Event metadata.

	_plist = ROOT.Belle2.PyStoreObj(self._listname)
	Pointer to the particle list.

	_dtypes = dtypes
	The data type.

	_buffer = np.empty(self._event_buffer_size * 10, dtype=self._dtypes)
	event variables buffer (will be automatically grown if necessary)

int	_buffer_index = 0
	current start index in the event variables buffer

list	_schema
	A list of tuples and py.DataTypes to define the pyarrow schema.

	_feather_writer
	a writer object to write data into a feather file

	_parquet_writer
	A list of tuples and py.DataTypes to define the pyarrow schema.

	_csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema), **self._writer_kwargs)
	A list of tuples and py.DataTypes to define the pyarrow schema.

	_hdf5_writer
	The pytable file.

	_table
	The pytable.

Detailed Description

Base class to dump ntuples into a non root format of your choosing

Definition at line 38 of file b2pandas_utils.py.

Constructor & Destructor Documentation

◆ init()

__init__	(		self,
		str	listname,
		List[str]	variables,
		str	filename,
		Optional[str]	hdf_table_name = None,
		int	event_buffer_size = 100,
		**	writer_kwargs )

Constructor to initialize the internal state

Arguments:
    listname(str): name of the particle list
    variables(list(str)): list of variables to save for each particle
    filename(str): name of the output file to be created.
        Needs to end with `.csv` for csv output, `.parquet` or `.pq` for parquet output,
        `.h5`, `.hdf` or `.hdf5` for hdf5 output and `.feather` or `.arrow` for feather output
    hdf_table_name(str): name of the table in the hdf5 file.
        If not provided, it will be the same as the listname
    event_buffer_size(int): number of events to buffer before writing to disk,
        higher values will use more memory but write faster and result in smaller files
    **writer_kwargs: additional keyword arguments to pass to the writer.
        For details, see the documentation of the writer in the apache arrow documentation.
        Only use, if you know what you are doing!

Definition at line 43 of file b2pandas_utils.py.

    ):
        """Constructor to initialize the internal state
 
        Arguments:
            listname(str): name of the particle list
            variables(list(str)): list of variables to save for each particle
            filename(str): name of the output file to be created.
                Needs to end with `.csv` for csv output, `.parquet` or `.pq` for parquet output,
                `.h5`, `.hdf` or `.hdf5` for hdf5 output and `.feather` or `.arrow` for feather output
            hdf_table_name(str): name of the table in the hdf5 file.
                If not provided, it will be the same as the listname
            event_buffer_size(int): number of events to buffer before writing to disk,
                higher values will use more memory but write faster and result in smaller files
            **writer_kwargs: additional keyword arguments to pass to the writer.
                For details, see the documentation of the writer in the apache arrow documentation.
                Only use, if you know what you are doing!
        """
        super().__init__()
        
        self._filename = filename
        
        self._listname = listname
        
        self._variables = list(set(variables))
        
        file_type = self._filename.split(".")[-1]
        
        if file_type in ["csv"]:
            self._format = "csv"
        elif file_type in ["parquet", "pq"]:
            self._format = "parquet"
        elif file_type in ["h5", "hdf", "hdf5"]:
            self._format = "hdf5"
        elif file_type in ["feather", "arrow"]:
            self._format = "feather"
        else:
            raise ValueError(
                f"Unknown file type ending .{file_type}, supported types are 'csv', "
                "'parquet', 'pq', 'h5', 'hdf', 'hdf5', 'feather' or 'arrow'"
            )
        
        self._table_name = (
            hdf_table_name if hdf_table_name is not None else self._listname
        )
        
        self._event_buffer_size = event_buffer_size
        
        self._event_buffer_counter = 0
        
        self._writer_kwargs = writer_kwargs
 

Member Function Documentation

◆ append_buffer()

append_buffer ( self )

"Append" a new event to the buffer by moving the buffer index forward by particle list size

Automatically replaces the buffer by a larger one if necessary

Definition at line 179 of file b2pandas_utils.py.

    def append_buffer(self):
        """
        "Append" a new event to the buffer by moving the buffer index forward by particle list size
 
        Automatically replaces the buffer by a larger one if necessary
        """
        plist_size = self._plist.getListSize()
        if (plist_size + self._buffer_index) > len(self._buffer):
            new_buffer = np.empty(
                # factor 1.5 larger or at least as large as necessary
                max(int(len(self._buffer) * 1.5), self._buffer_index + plist_size),
                dtype=self._dtypes,
            )
            new_buffer[:self._buffer_index] = self.buffer
            self._buffer = new_buffer
        self._buffer_index += plist_size
        self._event_buffer_counter += 1
 

◆ buffer()

buffer ( self )

The buffer slice across multiple entries

Definition at line 159 of file b2pandas_utils.py.

    def buffer(self):
        """
        The buffer slice across multiple entries
        """
        return self._buffer[:self._buffer_index]
 

◆ buffer_full()

buffer_full ( self )

check if the buffer is full

Definition at line 276 of file b2pandas_utils.py.

    def buffer_full(self):
        """
        check if the buffer is full
        """
        return self._event_buffer_counter == self._event_buffer_size
 

◆ clear_buffer()

clear_buffer ( self )

Reset the buffer event counter and index

Definition at line 172 of file b2pandas_utils.py.

    def clear_buffer(self):
        """
        Reset the buffer event counter and index
        """
        self._event_buffer_counter = 0
        self._buffer_index = 0
 

◆ event()

event ( self )

Event processing function

executes the fill_buffer function and writes the data to the output file
in chunks of event_buffer_size

Definition at line 299 of file b2pandas_utils.py.

    def event(self):
        """
        Event processing function
 
        executes the fill_buffer function and writes the data to the output file
        in chunks of event_buffer_size
        """
        self.append_buffer()
        self.fill_event_buffer()
        if self.buffer_full:
            self.write_buffer()
            self.clear_buffer()
 

◆ event_buffer()

event_buffer ( self )

The buffer slice for the current event

Definition at line 166 of file b2pandas_utils.py.

    def event_buffer(self):
        """
        The buffer slice for the current event
        """
        return self._buffer[self._buffer_index - self._plist.getListSize(): self._buffer_index]
 

◆ fill_event_buffer()

fill_event_buffer ( self )

Assign values for all variables for all particles in the particle list to the current event buffer

Definition at line 255 of file b2pandas_utils.py.

    def fill_event_buffer(self):
        """
        Assign values for all variables for all particles in the particle list to the current event buffer
        """
        buf = self.event_buffer
 
        # add some extra columns for bookkeeping
        buf["__experiment__"] = self._evtmeta.getExperiment()
        buf["__run__"] = self._evtmeta.getRun()
        buf["__event__"] = self._evtmeta.getEvent()
        buf["__production__"] = self._evtmeta.getProduction()
        buf["__ncandidates__"] = len(buf)
        buf["__candidate__"] = np.arange(len(buf))
 
        # fill variables into buffer
        vector = variables.variables.evaluateVariables(self._std_varnames, self._plist)
        values = np.array(vector.data()).reshape(-1, len(self._varnames))
        for name, col in zip(self._varnames, values.T):
            buf[name] = col
 

◆ initialize()

initialize ( self )

Setup variable lists, pointers, buffers and file writers

Definition at line 102 of file b2pandas_utils.py.

    def initialize(self):
        """
        Setup variable lists, pointers, buffers and file writers
        """
        # Always avoid the top-level 'import ROOT'.
        import ROOT  # noqa
 
        
        self._varnames = [
            str(varname)
            for varname in variables.variables.resolveCollections(
                variables.std_vector(*self._variables)
            )
        ]
 
        
        self._std_varnames = variables.std_vector(*self._varnames)
 
        
        self._evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")
        self._evtmeta.isRequired()
 
        
        self._plist = ROOT.Belle2.PyStoreObj(self._listname)
        self._plist.isRequired()
 
        dtypes = [
            ("__experiment__", np.int32),
            ("__run__", np.int32),
            ("__event__", np.uint32),
            ("__production__", np.uint32),
            ("__candidate__", np.uint32),
            ("__ncandidates__", np.uint32),
        ]
        for name in self._varnames:
            # only float variables for now
            dtypes.append((name, np.float64))
 
        
        self._dtypes = dtypes
 
        
        self._buffer = np.empty(self._event_buffer_size * 10, dtype=self._dtypes)
 
        
        self._buffer_index = 0
 
        if self._format == "hdf5":
            self.initialize_hdf5_writer()
        elif self._format == "parquet":
            self.initialize_parquet_writer()
        elif self._format == "csv":
            self.initialize_csv_writer()
        elif self._format == "feather":
            self.initialize_feather_writer()
 

◆ initialize_csv_writer()

initialize_csv_writer ( self )

Initialize the csv writer using pyarrow

Definition at line 225 of file b2pandas_utils.py.

    def initialize_csv_writer(self):
        """
        Initialize the csv writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema), **self._writer_kwargs)
 

◆ initialize_feather_writer()

initialize_feather_writer ( self )

Initialize the feather writer using pyarrow

Definition at line 197 of file b2pandas_utils.py.

    def initialize_feather_writer(self):
        """
        Initialize the feather writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._feather_writer = ipc.RecordBatchFileWriter(
            sink=self._filename,
            schema=pa.schema(self._schema),
            **self._writer_kwargs,
        )
 

◆ initialize_hdf5_writer()

initialize_hdf5_writer ( self )

Initialize the hdf5 writer using pytables

Definition at line 236 of file b2pandas_utils.py.

    def initialize_hdf5_writer(self):
        """
        Initialize the hdf5 writer using pytables
        """
        
        self._hdf5_writer = tables.open_file(
            self._filename, mode="w", title="Belle2 Variables to HDF5"
        )
        filters = tables.Filters(complevel=1, complib="blosc:lz4", fletcher32=False)
 
        # some variable names are not just A-Za-z0-9 so pytables complains but
        # seems to work. Ignore warning
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            
            self._table = self._hdf5_writer.create_table(
                "/", self._table_name, obj=np.zeros(0, self._dtypes), filters=filters, **self._writer_kwargs
            )
 

◆ initialize_parquet_writer()

initialize_parquet_writer ( self )

Initialize the parquet writer using pyarrow

Definition at line 212 of file b2pandas_utils.py.

    def initialize_parquet_writer(self):
        """
        Initialize the parquet writer using pyarrow
        """
        
        self._schema = [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]
        
        self._parquet_writer = ParquetWriter(
            self._filename, schema=pa.schema(self._schema), **self._writer_kwargs
        )
 

◆ terminate()

terminate ( self )

save and close the output

Definition at line 312 of file b2pandas_utils.py.

    def terminate(self):
        """save and close the output"""
        import ROOT  # noqa
        if len(self.buffer) > 0:
            self.write_buffer()
 
        if self._format == "hdf5":
            self._table.flush()
            self._hdf5_writer.close()
        elif self._format == "parquet":
            self._parquet_writer.close()
        elif self._format == "csv":
            self._csv_writer.close()
        elif self._format == "feather":
            self._feather_writer.close()
        ROOT.Belle2.MetadataService.Instance().addNtuple(self._filename)
 
 

◆ write_buffer()

write_buffer ( self )

write the buffer to the output file

Definition at line 282 of file b2pandas_utils.py.

    def write_buffer(self):
        """
        write the buffer to the output file
        """
        if self._format == "hdf5":
            """Create a new row in the hdf5 file with for each particle in the list"""
            self._table.append(self.buffer)
        else:
            table = {name: self.buffer[name] for name, _ in self._dtypes}
            pa_table = pa.table(table, schema=pa.schema(self._schema))
            if self._format == "parquet":
                self._parquet_writer.write_table(pa_table)
            elif self._format == "csv":
                self._csv_writer.write(pa_table)
            elif self._format == "feather":
                self._feather_writer.write_table(pa_table)
 

Member Data Documentation

◆ _buffer

_buffer = np.empty(self._event_buffer_size * 10, dtype=self._dtypes)

protected

event variables buffer (will be automatically grown if necessary)

Definition at line 144 of file b2pandas_utils.py.

◆ _buffer_index

_buffer_index = 0

protected

current start index in the event variables buffer

Definition at line 147 of file b2pandas_utils.py.

◆ _csv_writer

_csv_writer = CSVWriter(self._filename, schema=pa.schema(self._schema), **self._writer_kwargs)

protected

A list of tuples and py.DataTypes to define the pyarrow schema.

a writer object to write data into a csv file

Definition at line 234 of file b2pandas_utils.py.

◆ _dtypes

_dtypes = dtypes

protected

The data type.

Definition at line 141 of file b2pandas_utils.py.

◆ _event_buffer_counter

int _event_buffer_counter = 0

protected

Event buffer counter.

Definition at line 98 of file b2pandas_utils.py.

◆ _event_buffer_size

_event_buffer_size = event_buffer_size

protected

Event buffer size.

Definition at line 96 of file b2pandas_utils.py.

◆ _evtmeta

_evtmeta = ROOT.Belle2.PyStoreObj("EventMetaData")

protected

Event metadata.

Definition at line 121 of file b2pandas_utils.py.

◆ _feather_writer

_feather_writer

protected

Initial value:

=  ipc.RecordBatchFileWriter(
            sink=self._filename,
            schema=pa.schema(self._schema),
            **self._writer_kwargs,
        )

a writer object to write data into a feather file

Definition at line 206 of file b2pandas_utils.py.

◆ _filename

_filename = filename

protected

Output filename.

Definition at line 70 of file b2pandas_utils.py.

◆ _format

str _format = "csv"

protected

Output format.

Definition at line 79 of file b2pandas_utils.py.

◆ _hdf5_writer

_hdf5_writer

protected

Initial value:

=  tables.open_file(
            self._filename, mode="w", title="Belle2 Variables to HDF5"
        )

The pytable file.

Definition at line 241 of file b2pandas_utils.py.

◆ _listname

_listname = listname

protected

Particle list name.

Definition at line 72 of file b2pandas_utils.py.

◆ _parquet_writer

_parquet_writer

protected

Initial value:

=  ParquetWriter(
            self._filename, schema=pa.schema(self._schema), **self._writer_kwargs
        )

A list of tuples and py.DataTypes to define the pyarrow schema.

a writer object to write data into a parquet file

Definition at line 221 of file b2pandas_utils.py.

◆ _plist

_plist = ROOT.Belle2.PyStoreObj(self._listname)

protected

Pointer to the particle list.

Definition at line 125 of file b2pandas_utils.py.

◆ _schema

list _schema

protected

Initial value:

=  [
            (name, numpy_to_pyarrow_type_map[dt]) for name, dt in self._dtypes
        ]

A list of tuples and py.DataTypes to define the pyarrow schema.

Definition at line 202 of file b2pandas_utils.py.

◆ _std_varnames

_std_varnames = variables.std_vector(*self._varnames)

protected

std.vector of variable names

Definition at line 118 of file b2pandas_utils.py.

◆ _table

_table

protected

Initial value:

=  self._hdf5_writer.create_table(
                "/", self._table_name, obj=np.zeros(0, self._dtypes), filters=filters, **self._writer_kwargs
            )

The pytable.

Definition at line 251 of file b2pandas_utils.py.

◆ _table_name

tuple _table_name

protected

Initial value:

=  (
            hdf_table_name if hdf_table_name is not None else self._listname
        )

Table name in the hdf5 file.

Definition at line 92 of file b2pandas_utils.py.

◆ _variables

_variables = list(set(variables))

protected

List of variables.

Definition at line 74 of file b2pandas_utils.py.

◆ _varnames

_varnames

protected

Initial value:

=  [
            str(varname)
            for varname in variables.variables.resolveCollections(
                variables.std_vector(*self._variables)
            )
        ]

variable names

Definition at line 110 of file b2pandas_utils.py.

◆ _writer_kwargs

_writer_kwargs = writer_kwargs

protected

writer kwargs

Definition at line 100 of file b2pandas_utils.py.

◆ buffer

buffer = "hdf5":

Definition at line 288 of file b2pandas_utils.py.

◆ buffer_full

buffer_full

Definition at line 308 of file b2pandas_utils.py.

The documentation for this class was generated from the following file:

analysis/scripts/b2pandas_utils.py

Public Member Functions

Public Attributes

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ append_buffer()

◆ buffer()

◆ buffer_full()

◆ clear_buffer()

◆ event()

◆ event_buffer()

◆ fill_event_buffer()

◆ initialize()

◆ initialize_csv_writer()

◆ initialize_feather_writer()

◆ initialize_hdf5_writer()

◆ initialize_parquet_writer()

◆ terminate()

◆ write_buffer()

Member Data Documentation

◆ _buffer

◆ _buffer_index

◆ _csv_writer

◆ _dtypes

◆ _event_buffer_counter

◆ _event_buffer_size

◆ _evtmeta

◆ _feather_writer

◆ _filename

◆ _format

◆ _hdf5_writer

◆ _listname

◆ _parquet_writer

◆ _plist

◆ _schema

◆ _std_varnames

◆ _table

◆ _table_name

◆ _variables

◆ _varnames

◆ _writer_kwargs

◆ buffer

◆ buffer_full

◆ init()