HarvestingModule Class Reference

Inheritance diagram for HarvestingModule:

Public Member Functions
def	__init__ (self, foreach, output_file_name, name=None, title=None, contact=None, expert_level=None, show_results=False)
def	id (self)
def	initialize (self)
def	event (self)
def	terminate (self)
def	barn (self)
def	gather (self)
def	prepare (self)
def	peel (self, crop)
def	pick (self, crop)
def	refine (self, crops)

Static Public Member Functions
def	create_crop_part_collection ()
def	iter_store_obj (store_obj)

Public Attributes
	foreach
	Name of the StoreArray or iterable StoreObjPtr that contains the objects to be harvested.
	output_file_name
	Name of the ROOT output file to be generated.
	title
	Name of this harvest.
	contact
	Contact email address to be displayed on the validation page.
	expert_level
	Integer expert level that controls to detail of plots to be generated.
	refiners
	A list of additional refiner instances to be executed on top of the refiner methods that are members of this class.
	show_results
	Switch to show the result ROOT file in a TBrowser on terminate.
	stash
	stash of the harvested crops (start with those in the barn)
	raw_crops
	the dictionaries from peel as a numpy.array of doubles
	crops
	the dictionaries from peel

Static Public Attributes
int	default_expert_level = 1
	The default value of expert_level if not specified explicitly by the caller.

Detailed Description

Python module to generate summary figures of merits, plots and/or trees
from on StoreArray.

It runs as a proper module in the main path and examines each object in a StoreArray
in each event.


Notes
-----
Methods to be overwritten
``prepare``
    Method called at the start of each event, that may prepare things
    (e.g. setup lookup tables or precomputed list) used in the following methods.
``pick``
    Method called with each object in the StoreArray.
    Returns a False value if the object should be skipped.

``peel``
    Method called with each object in the StoreArray.
    Extractes the parts relevant for analysis and
    returns them as MutableMapping (e.g. a dict) of part_name and values.
    Currently only float values or values convertible to floats are supported.
    If requested that can change in the future.

On termination all the collected values are recasted to numpy arrays and
the whole ``crops`` of the harvest are casted to MutableMapping of numpy.array
with the same part_name and the same MutableMapping class as returned from peel.

Also in the termination phase refiners a invoked with the aggregated crops.
Refiners can be given in two ways.

First way is as a class methods marked as refiners like

    @refiners.Refiner
    def plot(self, crops, tdirectory, **kwds):
        ...

where self is the module instance, crops is the MutableMapping of numpy arrays and tdirectory
is the current tdirectory to which the current output shall be written.
The additional kwds leave room for future additional arguments.

Second way is to define the refiner method (like plot) out of line and add it to
the harvesting module instance refiners list like harvesting_module.refiners.append(plot).

Other specialised decorators to mark a function as a Refiner such as

* refiners.filter
* refiners.select
* refiners.groupby

exist.

Predefined refiner functions exist in the refiners python module as well.
For instance

    save_tree = refiners.save_tree()

is a predefined method to output the MutableMapping of numpy arrays as a TTree.

Definition at line 93 of file harvesting.py.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
			foreach,
			output_file_name,
			name = None,
			title = None,
			contact = None,
			expert_level = None,
			show_results = False
	)

Constructor of the harvesting module.

Parameters
----------
foreach : string
    Name of a StoreArray, which objects should be investigated
output_file_name : string
    Name of the ROOT file to which the results should be written.
    Giving an opened ROOT file is also allowed.
    If None is given write to the current ROOT file.
name : string, optional
    Name of the harvest that is used in the names of ROOT plots and trees.
    Defaults to the class name.
title : string, optional
    Name of the harvest that is used in the title of ROOT plots and trees.
    Defaults to the name.
contact : string, optional
    Contact email address to be used in the validation plots contact. Defaults to None.
expert_level : int, optional
    Expert level that can be used to switch on more plots.
    Generally the higher the more detailed to analysis.
    Meaning depends entirely on the subclass implementing a certain policy.
    Defaults to default_expert_level.
show_results : bool, optional
   Indicator to show the refined results at termination of the path

Reimplemented in MCTrajectoryHarvester, MCParticleHarvester, VxdCdcMergerHarvester, VxdCdcMergerHarvesterPRSide, SegmentFinderParameterExtractorModule, FitValidationModule, VxdCdcMergerHarvesterMCSide, VxdCdcPartFinderHarvester, SegmentFakeRatesModule, SeedsAnalyser, ReconstructionPositionHarvester, WrongRLInfoCounter, QueueHarvester, SummarizeTriggerResults, and SummarizeTriggerVariables.

Definition at line 158 of file harvesting.py.

                 show_results=False):
        """Constructor of the harvesting module.
 
        Parameters
        ----------
        foreach : string
            Name of a StoreArray, which objects should be investigated
        output_file_name : string
            Name of the ROOT file to which the results should be written.
            Giving an opened ROOT file is also allowed.
            If None is given write to the current ROOT file.
        name : string, optional
            Name of the harvest that is used in the names of ROOT plots and trees.
            Defaults to the class name.
        title : string, optional
            Name of the harvest that is used in the title of ROOT plots and trees.
            Defaults to the name.
        contact : string, optional
            Contact email address to be used in the validation plots contact. Defaults to None.
        expert_level : int, optional
            Expert level that can be used to switch on more plots.
            Generally the higher the more detailed to analysis.
            Meaning depends entirely on the subclass implementing a certain policy.
            Defaults to default_expert_level.
        show_results : bool, optional
           Indicator to show the refined results at termination of the path
           """
 
        super().__init__()
 
        
        self.foreach = foreach
 
        
        self.output_file_name = output_file_name
 
        if not isinstance(self.output_file_name, (ROOT.TFile, str)):
            raise TypeError("output_file_name is allowed to be a string or a ROOT.TFile object")
 
        
        self.set_name(name or self.__class__.__name__)
 
        
        self.title = title or self.name()
 
        
        self.contact = contact
 
        
        self.expert_level = self.default_expert_level if expert_level is None else expert_level
 
        
        self.refiners = []
 
        
        self.show_results = show_results
 

Member Function Documentation

barn()

def barn

(

self

)

Coroutine that receives the dictionaries of names and values from peel and store them.

Reimplemented in PickleHarvestingModule.

Definition at line 287 of file harvesting.py.

    def barn(self):
        """Coroutine that receives the dictionaries of names and values from peel and store them."""
        crop = (yield)
        raw_crops = copy.copy(crop)
        crops = copy.copy(crop)
 
        if isinstance(crop, numbers.Number):
            raw_crops = self.create_crop_part_collection()
            try:
                while True:
                    raw_crops.append(crop)
                    # next crop
                    crop = (yield)
            except GeneratorExit:
                crops = np.array(raw_crops)
 
        elif isinstance(crop, collections.abc.MutableMapping):
            for part_name in crop:
                raw_crops[part_name] = self.create_crop_part_collection()
 
            try:
                while True:
                    for part_name, parts in list(raw_crops.items()):
                        if part_name in crop:
                            parts.append(crop[part_name])
                        else:
                            parts.append(np.NaN)
                    # next crop
                    crop = (yield)
            except GeneratorExit:
                for part_name, parts in list(raw_crops.items()):
                    crops[part_name] = np.array(parts)
 
        else:
            msg = f"Unrecognised crop {crop} of type {type(crop)}"
            raise ValueError(msg)
 
        
        self.raw_crops = raw_crops
        
        self.crops = crops
 

create_crop_part_collection()

def create_crop_part_collection ( )

static

Create the storing objects for the crop values

Currently a numpy.array of doubles is used to store all values in memory.

Definition at line 279 of file harvesting.py.

    def create_crop_part_collection():
        """Create the storing objects for the crop values
 
        Currently a numpy.array of doubles is used to store all values in memory.
        """
        return array.array("d")
 

event()

def event

(

self

)

Event method of the module

* Does invoke the prepare method before the iteration starts.
* In each event fetch the StoreArray / iterable StoreObjPtr,
* Iterate through all instances
* Feed each instance to the pick method to decide it the instance is relevant
* Forward it to the peel method that should generated a dictionary of values
* Store each dictionary of values

Definition at line 239 of file harvesting.py.

    def event(self):
        """Event method of the module
 
        * Does invoke the prepare method before the iteration starts.
        * In each event fetch the StoreArray / iterable StoreObjPtr,
        * Iterate through all instances
        * Feed each instance to the pick method to decide it the instance is relevant
        * Forward it to the peel method that should generated a dictionary of values
        * Store each dictionary of values
        """
        self.prepare()
        stash = self.stash.send
        pick = self.pick
        peel = self.peel
        for crop in self.gather():
            if pick(crop):
                crop = peel(crop)
                if isinstance(crop, types.GeneratorType):
                    many_crops = crop
                    for crop in many_crops:
                        stash(crop)
                else:
                    stash(crop)
 

gather()

def gather

(

self

)

Iterator that yield the instances form the StoreArray / iterable StoreObj.

Yields
------
Object instances from the StoreArray, iterable StoreObj or the StoreObj itself
in case it is not iterable.

Definition at line 329 of file harvesting.py.

    def gather(self):
        """Iterator that yield the instances form the StoreArray / iterable StoreObj.
 
        Yields
        ------
        Object instances from the StoreArray, iterable StoreObj or the StoreObj itself
        in case it is not iterable.
        """
 
        registered_store_arrays = Belle2.PyStoreArray.list()
        registered_store_objs = Belle2.PyStoreObj.list()
 
        foreach = self.foreach
        foreach_is_store_obj = foreach in registered_store_objs
        foreach_is_store_array = foreach in registered_store_arrays
 
        if foreach is not None:
            if foreach_is_store_array:
                store_array = Belle2.PyStoreArray(self.foreach)
                yield from store_array
 
            elif foreach_is_store_obj:
                store_obj = Belle2.PyStoreObj(self.foreach)
                try:
                    yield from self.iter_store_obj(store_obj)
                except TypeError:
                    # Cannot iter the store object. Yield it instead.
                    yield store_obj.obj()
 
            else:
                msg = f"Name {self.foreach} does not refer to a valid object on the data store"
                raise KeyError(msg)
        else:
            yield None
 

id()

def id

(

self

)

Working around that name() is a method.

Exposing the name as a property using a different name

Definition at line 224 of file harvesting.py.

    def id(self):
        """Working around that name() is a method.
 
        Exposing the name as a property using a different name
        """
        return self.name()
 

initialize()

def initialize

(

self

)

Initialisation method of the module.

Prepares the receiver stash of objects to be harvestered.

Definition at line 231 of file harvesting.py.

    def initialize(self):
        """Initialisation method of the module.
 
        Prepares the receiver stash of objects to be harvestered.
        """
        
        self.stash = self.barn()
 

iter_store_obj()

def iter_store_obj (   store_obj )

static

Obtain a iterator from a StoreObj

Repeatedly calls iter(store_obj) or store_obj.__iter__()
until the final iterator returns itself

Returns
-------
iterator of the StoreObj

Definition at line 443 of file harvesting.py.

    def iter_store_obj(store_obj):
        """Obtain a iterator from a StoreObj
 
        Repeatedly calls iter(store_obj) or store_obj.__iter__()
        until the final iterator returns itself
 
        Returns
        -------
        iterator of the StoreObj
        """
        iterable = store_obj.obj()
        last_iterable = None
        while iterable is not last_iterable:
            if hasattr(iterable, "__iter__"):
                iterable, last_iterable = iterable.__iter__(), iterable
            else:
                iterable, last_iterable = iter(iterable), iterable
        return iterable
 
 

peel()

def peel	(		self,
			crop
	)

Unpack the the instances and return and dictionary of names to values or
a generator of those dictionaries to be saved.

Returns
-------
dict(str -> float)
    Unpacked names and values
or

Yields
------
dict(str -> float)
    Unpacked names and values

Reimplemented in SeedsAnalyser, FitValidationModule, SegmentFakeRatesModule, SegmentFinderParameterExtractorModule, VxdCdcMergerHarvesterMCSide, MCParticleHarvester, VxdCdcMergerHarvester, MCTrajectoryHarvester, VxdCdcPartFinderHarvester, VxdCdcMergerHarvesterPRSide, SummarizeTriggerResults, ReconstructionPositionHarvester, WrongRLInfoCounter, and SummarizeTriggerVariables.

Definition at line 371 of file harvesting.py.

    def peel(self, crop):
        """Unpack the the instances and return and dictionary of names to values or
        a generator of those dictionaries to be saved.
 
        Returns
        -------
        dict(str -> float)
            Unpacked names and values
        or
 
        Yields
        ------
        dict(str -> float)
            Unpacked names and values
 
        """
        return {"name": np.nan}
 

pick()

def pick	(		self,
			crop
	)

Indicate whether the instance should be forwarded to the peeling

Returns
-------
bool : Indicator if the instance is valuable in the current harvest.

Reimplemented in VxdCdcMergerHarvesterMCSide, MCParticleHarvester, and VxdCdcMergerHarvester.

Definition at line 389 of file harvesting.py.

    def pick(self, crop):
        """Indicate whether the instance should be forwarded to the peeling
 
        Returns
        -------
        bool : Indicator if the instance is valuable in the current harvest.
        """
        return True
 

prepare()

def prepare

(

self

)

Default implementation of prepare.

Can be overridden by subclasses.

Reimplemented in SegmentFakeRatesModule, and FitValidationModule.

Definition at line 364 of file harvesting.py.

    def prepare(self):
        """Default implementation of prepare.
 
        Can be overridden by subclasses.
        """
        return
 

refine()

def refine	(		self,
			crops
	)

Receive the gathered crops and forward them to the refiners.

Reimplemented in PickleHarvestingModule.

Definition at line 398 of file harvesting.py.

    def refine(self, crops):
        """Receive the gathered crops and forward them to the refiners."""
 
        kwds = {}
        if self.output_file_name:
            # Save everything to a ROOT file
            if isinstance(self.output_file_name, ROOT.TFile):
                output_tdirectory = self.output_file_name
            else:
                output_tfile = ROOT.TFile(self.output_file_name, 'recreate')
                output_tdirectory = output_tfile
 
        else:
            output_tdirectory = None
 
        try:
            with root_cd(output_tdirectory):
                for refiner in self.refiners:
                    refiner(self, crops, tdirectory=output_tdirectory, **kwds)
 
                # Get the methods marked as refiners from the class
                cls = type(self)
                for name in dir(cls):
                    if isinstance(getattr(cls, name), Refiner):
                        refiner = getattr(self, name)
                        # Getattr already binds self
                        refiner(crops, tdirectory=output_tdirectory, **kwds)
 
        finally:
            # If we opened the TFile ourself, close it again
            if self.output_file_name:
                if isinstance(self.output_file_name, str):
                    output_tfile.Close()
 
        if self.show_results and self.output_file_name:
            if isinstance(self.output_file_name, str):
                output_tfile = ROOT.TFile(self.output_file_name)
                root_browse(output_tfile)
                input("Press enter to close")
                output_tfile.Close()
            else:
                root_browse(self.output_file_name)
                input("Press enter to close")
 

terminate()

def terminate

(

self

)

Termination method of the module.

Finalize the collected crops.
Start the refinement.

Definition at line 263 of file harvesting.py.

    def terminate(self):
        """Termination method of the module.
 
        Finalize the collected crops.
        Start the refinement.
        """
 
        self.stash.close()
        del self.stash
 
        try:
            self.refine(self.crops)
        except AttributeError:
            pass
 

Member Data Documentation

contact

contact

Contact email address to be displayed on the validation page.

Definition at line 211 of file harvesting.py.

crops

crops

the dictionaries from peel

Definition at line 327 of file harvesting.py.

default_expert_level

int default_expert_level = 1

static

The default value of expert_level if not specified explicitly by the caller.

Definition at line 156 of file harvesting.py.

expert_level

expert_level

Integer expert level that controls to detail of plots to be generated.

Definition at line 214 of file harvesting.py.

foreach

foreach

Name of the StoreArray or iterable StoreObjPtr that contains the objects to be harvested.

Definition at line 196 of file harvesting.py.

output_file_name

output_file_name

Name of the ROOT output file to be generated.

Definition at line 199 of file harvesting.py.

raw_crops

raw_crops

the dictionaries from peel as a numpy.array of doubles

Definition at line 325 of file harvesting.py.

refiners

refiners

A list of additional refiner instances to be executed on top of the refiner methods that are members of this class.

Definition at line 218 of file harvesting.py.

show_results

show_results

Switch to show the result ROOT file in a TBrowser on terminate.

Definition at line 221 of file harvesting.py.

stash

stash

stash of the harvested crops (start with those in the barn)

Definition at line 237 of file harvesting.py.

title

title

Name of this harvest.

Title particle of this harvest

Definition at line 208 of file harvesting.py.

---

The documentation for this class was generated from the following file:

• tracking/scripts/tracking/harvest/harvesting.py