RecoTrackQEDataCollectionTask Class Reference

Inheritance diagram for RecoTrackQEDataCollectionTask:

Public Member Functions
	get_records_file_name (self, n_events=None, random_seed=None, recotrack_option=None)
	Filename of the recorded/collected data for the final QE MVA training.
	get_input_files (self, n_events=None, random_seed=None)
	requires (self)
	output (self)
	create_path (self)

Static Public Attributes
	n_events = b2luigi.IntParameter()
	Number of events to generate.
	experiment_number = b2luigi.IntParameter()
	Experiment number of the conditions database, e.g.
	random_seed = b2luigi.Parameter()
	Random basf2 seed used by the GenerateSimTask.
	cdc_training_target = b2luigi.Parameter()
	Feature/variable to use as truth label for the CDC track quality estimator.
	recotrack_option
	RecoTrack option, use string that is additive: deleteCDCQI0XY (= deletes CDCTracks with CDC-QI below 0.XY), useCDC (= uses trained CDC stored in datafiles/), useVXD (uses trained VXD stored in datafiles/), noVXD (= doesn't use the VXD MVA at all)
	fast_bdt_option
	Hyperparameter option of the FastBDT algorithm.
	process_type
	files (USESIMBB/EE) or running on existing reconstructed files (USERECBB/EE)
str	queue = 'l'
	specify queue.

Detailed Description

Collect variables/features from the reco track reconstruction including the
fit and write them to a ROOT file.

These variables are to be used as labelled training data for the MVA
classifier which is the MVA track quality estimator.  The collected
variables include the classifier outputs from the VXD and CDC quality
estimators, namely the CDC and VXD quality indicators, combined with fit,
merger, timing, energy loss information etc.  This task requires the
subdetector quality estimators to be trained.

Definition at line 917 of file combined_quality_estimator_teacher.py.

Member Function Documentation

create_path()

create_path

(

self

)

Create basf2 reconstruction path that should mirror the default path
from ``add_tracking_reconstruction()``, but with modules for the VXD QE
and CDC QE application and for collection of variables for the reco
track quality estimator.

Definition at line 1053 of file combined_quality_estimator_teacher.py.

    def create_path(self):
        """
        Create basf2 reconstruction path that should mirror the default path
        from ``add_tracking_reconstruction()``, but with modules for the VXD QE
        and CDC QE application and for collection of variables for the reco
        track quality estimator.
        """
        path = basf2.create_path()
        inputFileNames = self.get_input_files()
        path.add_module(
            "RootInput",
            inputFileNames=inputFileNames,
        )
        path.add_module("Gearbox")
 
        # First add tracking reconstruction with default quality estimation modules
        mvaCDC = True
        mvaVXD = True
        if 'noCDC' in self.recotrack_option:
            mvaCDC = False
        if 'noVXD' in self.recotrack_option:
            mvaVXD = False
        if 'DATA' in self.random_seed:
            from rawdata import add_unpackers
            add_unpackers(path)
        tracking.add_tracking_reconstruction(path, add_cdcTrack_QI=mvaCDC, add_vxdTrack_QI=mvaVXD, add_recoTrack_QI=True)
 
        cdc_identifier = ""
        # if data shall be processed check if newly trained mva files are available. Otherwise use default ones (CDB payloads):
        # if useCDC/VXD is specified, use the identifier lying in datafiles/ Otherwise, replace weightfile identifiers from
        # defaults (CDB payloads) to new weightfiles created by this b2luigi script
        if ('DATA' in self.random_seed or 'useCDC' in self.recotrack_option) and 'noCDC' not in self.recotrack_option:
            cdc_identifier = 'datafiles/' + \
               CDCQETeacherTask.get_weightfile_identifier(CDCQETeacherTask, fast_bdt_option=self.fast_bdt_option) + '.xml'
            if os.path.exists(cdc_identifier):
                replace_cdc_qi = True
            elif 'useCDC' in self.recotrack_option:
                raise ValueError(f"CDC QI Identifier not found: {cdc_identifier}")
            else:
                replace_cdc_qi = False
        elif 'noCDC' in self.recotrack_option:
            replace_cdc_qi = False
        else:
            cdc_identifier = self.get_input_file_names(
                CDCQETeacherTask.get_weightfile_identifier(
                    CDCQETeacherTask, fast_bdt_option=self.fast_bdt_option) + '.xml')[0]
            replace_cdc_qi = True
        if ('DATA' in self.random_seed or 'useVXD' in self.recotrack_option) and 'noVXD' not in self.recotrack_option:
            vxd_identifier = 'datafiles/' + \
               VXDQETeacherTask.get_weightfile_identifier(VXDQETeacherTask, fast_bdt_option=self.fast_bdt_option) + '.xml'
            if os.path.exists(vxd_identifier):
                replace_vxd_qi = True
                print(f"vxd_identifier is {vxd_identifier}")
            elif 'useVXD' in self.recotrack_option:
                raise ValueError(f"VXD QI Identifier not found: {vxd_identifier}")
            else:
                replace_vxd_qi = False
        elif 'noVXD' in self.recotrack_option:
            replace_vxd_qi = False
        else:
            vxd_identifier = self.get_input_file_names(
                VXDQETeacherTask.get_weightfile_identifier(
                    VXDQETeacherTask, fast_bdt_option=self.fast_bdt_option) + '.xml')[0]
            replace_vxd_qi = True
 
        cdc_qe_mva_filter_parameters = None
        # if tracks below a certain CDC QI index shall be deleted online, this needs to be specified in the filter parameters.
        # this is also possible in case of the default (CBD) payloads.
        cut = 0
        if 'deleteCDCQI' in self.recotrack_option:
            cut_index = self.recotrack_option.find('deleteCDCQI') + len('deleteCDCQI')
            cut = int(self.recotrack_option[cut_index:cut_index+3])/100.
            if replace_cdc_qi:
                cdc_qe_mva_filter_parameters = {
                    "identifier": cdc_identifier, "cut": cut}
            else:
                cdc_qe_mva_filter_parameters = {
                    "cut": cut}
        elif replace_cdc_qi:
            cdc_qe_mva_filter_parameters = {
                "identifier": cdc_identifier}
        basf2.conditions.prepend_testing_payloads("localdb/database.txt")
 
        if cdc_qe_mva_filter_parameters is not None and cdc_identifier is not None:
            name = 'TrackingMVAFilterParameters'
            cdc_qe_mva_filter_parameters = {'DBPayloadName': name}
        if replace_vxd_qi and vxd_identifier is not None:
            vxd_name = 'VXDQualityEstimatorMVA'
        # Added for debugging
        if cdc_qe_mva_filter_parameters is not None:
            # if no cut is specified, the default value is at zero and nothing is deleted.
            print(f'cdc_qe_mva_filter_parameters is {cdc_qe_mva_filter_parameters}')
            basf2.set_module_parameters(
                path,
                name="TFCDC_TrackQualityEstimator",
                filterParameters=cdc_qe_mva_filter_parameters,
                deleteTracks=True,
                resetTakenFlag=True,
                deactivateIfDeadBoard=False,  # original behavior before deactivateIfDeadBoard was introduced
                )
        if replace_vxd_qi:
            print(f"replace_vxd_qi is true and vxd_identifier is {vxd_identifier}")
            basf2.set_module_parameters(
                path,
                name="VXDQualityEstimatorMVA",
                WeightFileIdentifier=vxd_name)
 
        # Replace final quality estimator module by training data collector module
        track_qe_module_name = "TrackQualityEstimatorMVA"
        mc_track_matcher_module_name = "MCRecoTracksMatcher"
        qe_module_found = False
        mc_matcher_module_found = False
        new_path = basf2.create_path()
        for module in path.modules():
            if module.name() == track_qe_module_name:
                # the TrackCreator needs to be conducted before the Collector such that
                # MDSTTracks are related to RecoTracks and d0 and z0 can be read out
                new_path.add_module(
                    'TrackCreator',
                    pdgCodes=[
                        211,
                        321,
                        2212],
                    recoTrackColName='RecoTracks',
                    trackColName='MDSTTracks')  # , useClosestHitToIP=True, useBFieldAtHit=True)
                qe_module_found = True
            elif module.name() == mc_track_matcher_module_name:
                new_path.add_module(module)
                # move TrackQETrainingDataCollector module after the MCRecoTracksMatcher module
                new_path.add_module(
                    "TrackQETrainingDataCollector",
                    TrainingDataOutputName=self.get_output_file_name(self.get_records_file_name()),
                    collectEventFeatures=True
                )
                mc_matcher_module_found = True
            else:
                new_path.add_module(module)
        if not qe_module_found:
            raise KeyError(f"No module {track_qe_module_name} found in path")
        if not mc_matcher_module_found:
            raise KeyError(f"No module {mc_matcher_module_found} found in path")
        path = new_path
        return path
 
 

get_input_files()

get_input_files	(		self,
			n_events = None,
			random_seed = None )

Get input file names depending on the use case: If they already exist, search in
the corresponding folders, for data check the specified list and if they are created
in the same run, check for the task that produced them.

Definition at line 988 of file combined_quality_estimator_teacher.py.

    def get_input_files(self, n_events=None, random_seed=None):
        """
        Get input file names depending on the use case: If they already exist, search in
        the corresponding folders, for data check the specified list and if they are created
        in the same run, check for the task that produced them.
        """
        if n_events is None:
            n_events = self.n_events
        if random_seed is None:
            random_seed = self.random_seed
        if "USESIM" in random_seed:
            if 'USESIMBB' in random_seed:
                random_seed = 'BBBAR_' + random_seed.split("_", 1)[1]
            elif 'USESIMEE' in random_seed:
                random_seed = 'BHABHA_' + random_seed.split("_", 1)[1]
            return ['datafiles/' + GenerateSimTask.output_file_name(GenerateSimTask,
                                                                    n_events=n_events, random_seed=random_seed)]
        elif "DATA" in random_seed:
            return MasterTask.datafiles
        else:
            return self.get_input_file_names(GenerateSimTask.output_file_name(
                GenerateSimTask, n_events=n_events, random_seed=random_seed))
 

get_records_file_name()

get_records_file_name	(		self,
			n_events = None,
			random_seed = None,
			recotrack_option = None )

Filename of the recorded/collected data for the final QE MVA training.

Create output file name depending on number of events and production
mode that is specified in the random_seed string.

Definition at line 963 of file combined_quality_estimator_teacher.py.

    def get_records_file_name(self, n_events=None, random_seed=None, recotrack_option=None):
        """
        Create output file name depending on number of events and production
        mode that is specified in the random_seed string.
        """
        if n_events is None:
            n_events = self.n_events
        if random_seed is None:
            random_seed = self.random_seed
        if recotrack_option is None:
            if isinstance(self.recotrack_option, str):
                recotrack_option = self.recotrack_option
            else:
                recotrack_option = self.recotrack_option._default
        if 'rec' not in random_seed:
            random_seed += '_rec'
        if 'DATA' in random_seed:
            return 'qe_records_DATA_rec.root'
        else:
            if 'USESIMBB' in random_seed:
                random_seed = 'BBBAR_' + random_seed.split("_", 1)[1]
            elif 'USESIMEE' in random_seed:
                random_seed = 'BHABHA_' + random_seed.split("_", 1)[1]
            return 'qe_records_N' + str(n_events) + '_' + random_seed + '_' + recotrack_option + '.root'
 

output()

output

(

self

)

Generate list of output files that the task should produce.
The task is considered finished if and only if the outputs all exist.

Definition at line 1046 of file combined_quality_estimator_teacher.py.

    def output(self):
        """
        Generate list of output files that the task should produce.
        The task is considered finished if and only if the outputs all exist.
        """
        yield self.add_to_output(self.get_records_file_name())
 

requires()

requires

(

self

)

Generate list of luigi Tasks that this Task depends on.

Definition at line 1011 of file combined_quality_estimator_teacher.py.

    def requires(self):
        """
        Generate list of luigi Tasks that this Task depends on.
        """
        if "USESIM" in self.random_seed or "DATA" in self.random_seed:
            for filename in self.get_input_files():
                yield CheckExistingFile(
                    filename=filename,
                    )
        else:
            yield SplitNMergeSimTask(
                bkgfiles_dir=MasterTask.bkgfiles_by_exp[self.experiment_number],
                random_seed=self.random_seed,
                n_events=self.n_events,
                experiment_number=self.experiment_number,
                )
        if "DATA" not in self.random_seed:
            if 'useCDC' not in self.recotrack_option and 'noCDC' not in self.recotrack_option:
                yield CDCQETeacherTask(
                    n_events_training=MasterTask.n_events_training,
                    experiment_number=self.experiment_number,
                    training_target=self.cdc_training_target,
                    process_type=self.random_seed.split("_", 1)[0],
                    exclude_variables=MasterTask.exclude_variables_cdc,
                    fast_bdt_option=self.fast_bdt_option,
                    )
            if 'useVXD' not in self.recotrack_option and 'noVXD' not in self.recotrack_option:
                yield VXDQETeacherTask(
                    n_events_training=MasterTask.n_events_training,
                    experiment_number=self.experiment_number,
                    process_type=self.random_seed.split("_", 1)[0],
                    exclude_variables=MasterTask.exclude_variables_vxd,
                    fast_bdt_option=self.fast_bdt_option,
                    )
 

Member Data Documentation

cdc_training_target

cdc_training_target = b2luigi.Parameter()

static

Feature/variable to use as truth label for the CDC track quality estimator.

Definition at line 938 of file combined_quality_estimator_teacher.py.

experiment_number

experiment_number = b2luigi.IntParameter()

static

Experiment number of the conditions database, e.g.

defines simulation geometry

Definition at line 933 of file combined_quality_estimator_teacher.py.

fast_bdt_option

fast_bdt_option

static

Initial value:

=  b2luigi.ListParameter(
        
 
 
    )

Hyperparameter option of the FastBDT algorithm.

default are the FastBDT default values.

Definition at line 948 of file combined_quality_estimator_teacher.py.

n_events

n_events = b2luigi.IntParameter()

static

Number of events to generate.

Definition at line 931 of file combined_quality_estimator_teacher.py.

process_type

process_type

static

Initial value:

=  b2luigi.Parameter(
        
 
 
    )

files (USESIMBB/EE) or running on existing reconstructed files (USERECBB/EE)

Definition at line 954 of file combined_quality_estimator_teacher.py.

queue

str queue = 'l'

static

specify queue.

E.g. choose between 'l' (long), 's' (short) or 'sx' (short, extra ram)

Definition at line 960 of file combined_quality_estimator_teacher.py.

random_seed

random_seed = b2luigi.Parameter()

static

Random basf2 seed used by the GenerateSimTask.

It is further used to read of the production process to preserve clearness in the b2luigi output.

Definition at line 936 of file combined_quality_estimator_teacher.py.

recotrack_option

recotrack_option

static

Initial value:

=  b2luigi.Parameter(
        
 
 
    )

RecoTrack option, use string that is additive: deleteCDCQI0XY (= deletes CDCTracks with CDC-QI below 0.XY), useCDC (= uses trained CDC stored in datafiles/), useVXD (uses trained VXD stored in datafiles/), noVXD (= doesn't use the VXD MVA at all)

Definition at line 942 of file combined_quality_estimator_teacher.py.

---

The documentation for this class was generated from the following file:

• tracking/scripts/tracking/train/combined_quality_estimator_teacher.py