Calibration Class Reference

Inheritance diagram for Calibration:

Public Member Functions
	__init__ (self, name, collector=None, algorithms=None, input_files=None, pre_collector_path=None, database_chain=None, output_patterns=None, max_files_per_collector_job=None, max_collector_jobs=None, backend_args=None)
	add_collection (self, name, collection)
	is_valid (self)
	reset_database (self, apply_to_default_collection=True)
	use_central_database (self, global_tag, apply_to_default_collection=True)
	use_local_database (self, filename, directory="", apply_to_default_collection=True)
	collector (self)
	collector (self, collector)
	input_files (self)
	input_files (self, files)
	files_to_iovs (self)
	files_to_iovs (self, file_map)
	pre_collector_path (self)
	pre_collector_path (self, path)
	output_patterns (self)
	output_patterns (self, patterns)
	max_files_per_collector_job (self)
	max_files_per_collector_job (self, max_files)
	max_collector_jobs (self)
	max_collector_jobs (self, max_jobs)
	backend_args (self)
	backend_args (self, args)
	algorithms (self)
	algorithms (self, value)
	pre_algorithms (self)
	pre_algorithms (self, func)
	strategies (self)
	strategies (self, strategy)
	__repr__ (self)
	run (self)
	state (self)
	state (self, state)
	iteration (self)
	iteration (self, iteration)
	depends_on (self, calibration)
	dependencies_met (self)
	failed_dependencies (self)

Public Attributes
dict	collections = {}
	Collections stored for this calibration.
	algorithms = algorithms
	Algorithm classes that will be run by this Calibration.
dict	results = {}
	Output results of algorithms for each iteration.
	max_iterations = None
	Variable to define the maximum number of iterations for this calibration specifically.
	ignored_runs = None
	List of ExpRun that will be ignored by this Calibration.
	strategies = strategies.SingleIOV
	The strategy that the algorithm(s) will be run against.
list	database_chain = []
	The database chain that is applied to the algorithms.
	algorithms_runner = runners.SeqAlgorithmsRunner
	The class that runs all the algorithms in this Calibration using their assigned :py:class:caf.strategies.AlgorithmStrategy.
	backend = None
	The backend <backends.Backend> we'll use for our collector submission in this calibration.
int	collector_full_update_interval = 30
	While checking if the collector is finished we don't bother wastefully checking every subjob's status.
int	heartbeat = 3
	This calibration's sleep time before rechecking to see if it can move state.
	machine = None
	The caf.state_machines.CalibrationMachine that we will run to process this calibration start to finish.
str	state = initial_state
	state
	name = name
	Name of calibration object.
list	future_dependencies = []
	List of calibration objects that depend on this one.
list	dependencies = []
	List of calibration objects, where each one is a dependency of this one.
	iov = None
	IoV which will be calibrated.
str	output_database_dir = ""
	The directory where we'll store the local database payloads from this calibration.
bool	save_payloads = True
	Marks this Calibration as one which has payloads that should be copied and uploaded.
list	jobs_to_submit = []
	A simple list of jobs that this Calibration wants submitted at some point.

Static Public Attributes
list	moves = ["submit_collector", "complete", "run_algorithms", "iterate", "fail_fully"]
	Allowed transitions that we will use to progress.
str	alg_output_dir = "algorithm_output"
	Subdirectory name for algorithm output.
list	checkpoint_states = ["init", "collector_completed", "completed"]
	Checkpoint states which we are allowed to restart from.
str	default_collection_name = "default"
	Default collection name.
str	end_state = "completed"
	The name of the successful completion state.
str	fail_state = "failed"
	The name of the failure state.

Protected Member Functions
	_get_default_collection_attribute (self, attr)
	_set_default_collection_attribute (self, attr, value)
	_poll_collector (self)
	_apply_calibration_defaults (self, defaults)

Protected Attributes
list	_algorithms = []
	Internal calibration algorithms stored for this calibration.
	_db_path = None
	Location of a SQLite database that will save the state of the calibration so that it can be restarted from failure.

Detailed Description

Every Calibration object must have at least one collector at least one algorithm.
You have the option to add in your collector/algorithm by argument here, or add them
later by changing the properties.

If you plan to use multiple `Collection` objects I recommend that you only set the name here and add the Collections
separately via `add_collection()`.

Parameters:
    name (str): Name of this calibration. It should be unique for use in the `CAF`
Keyword Arguments:
    collector (str, `basf2.Module`): Should be set to a CalibrationCollectorModule() or a string with the module name.
    algorithms (list, ``ROOT.Belle2.CalibrationAlgorithm``): The algorithm(s) to use for this `Calibration`.
    input_files (str, list[str]): Input files for use by this Calibration. May contain wildcards usable by `glob.glob`

A Calibration won't be valid in the `CAF` until it has all of these four attributes set. For example:

>>> cal = Calibration('TestCalibration1')
>>> col1 = register_module('CaTest')
>>> cal.add_collection('TestColl', col1)

or equivalently

>>> cal = Calibration('TestCalibration1', 'CaTest')

If you want to run a basf2 :py:class:`path <basf2.Path>` before your collector module when running over data

>>> cal.pre_collector_path = my_basf2_path

You don't have to put a RootInput module in this pre-collection path, but you can if
you need some special parameters. If you want to process sroot files the you have to explicitly add
SeqRootInput to your pre-collection path.
The inputFileNames parameter of (Seq)RootInput will be set by the CAF automatically for you.


You can use optional arguments to pass in some/all during initialisation of the `Calibration` class

>>> cal = Calibration( 'TestCalibration1', 'CaTest', [alg1,alg2], ['/path/to/file.root'])

you can change the input file list later on, before running with `CAF`

>>> cal.input_files = ['path/to/*.root', 'other/path/to/file2.root']

If you have multiple collections from calling `add_collection()` then you should instead set the pre_collector_path,
input_files, database chain etc from there. See `Collection`.

Adding the CalibrationAlgorithm(s) is easy

>>> alg1 = TestAlgo()
>>> cal.algorithms = alg1

Or equivalently

>>> cal.algorithms = [alg1]

Or for multiple algorithms for one collector

>>> alg2 = TestAlgo()
>>> cal.algorithms = [alg1, alg2]

Note that when you set the algorithms, they are automatically wrapped and stored as a Python class
`Algorithm`. To access the C++ algorithm clas underneath directly do:

>>> cal.algorithms[i].algorithm

If you have a setup function that you want to run before each of the algorithms, set that with

>>> cal.pre_algorithms = my_function_object

If you want a different setup for each algorithm use a list with the same number of elements
as your algorithm list.

>>> cal.pre_algorithms = [my_function1, my_function2, ...]

You can also specify the dependencies of the calibration on others

>>> cal.depends_on(cal2)

By doing this, the `CAF` will respect the ordering of the calibrations and will pass the
calibration constants created by earlier completed calibrations to dependent ones.

Definition at line 447 of file framework.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
			name,
			collector = None,
			algorithms = None,
			input_files = None,
			pre_collector_path = None,
			database_chain = None,
			output_patterns = None,
			max_files_per_collector_job = None,
			max_collector_jobs = None,
			backend_args = None )

Definition at line 538 of file framework.py.

                 ):
        """
        """
        
        self.collections = {}
        
        self._algorithms = []
 
        # Default collection added, will have None type and requires setting later via `self.collector`, or will take the
        # CollectorModule/module name directly.
        self.add_collection(self.default_collection_name,
                            Collection(collector,
                                       input_files,
                                       pre_collector_path,
                                       database_chain,
                                       output_patterns,
                                       max_files_per_collector_job,
                                       max_collector_jobs,
                                       backend_args
                                       ))
 
        super().__init__(name, input_files)
        if algorithms:
            
            self.algorithms = algorithms
        
        self.results = {}
        
        self.max_iterations = None
        
        self.ignored_runs = None
        if self.algorithms:
            
            self.strategies = strategies.SingleIOV
        
        self.database_chain = []
        if database_chain:
            self.database_chain = database_chain
        else:
            # This database is already applied to the `Collection` automatically, so don't do it again
            for tag in reversed(b2conditions.default_globaltags):
                self.use_central_database(tag, apply_to_default_collection=False)
        
        self.algorithms_runner = runners.SeqAlgorithmsRunner
        
        self.backend = None
        
        self.collector_full_update_interval = 30
        
        self.heartbeat = 3
        
        self.machine = None
        
        self._db_path = None
 

Member Function Documentation

__repr__()

__repr__

(

self

)

Definition at line 1012 of file framework.py.

    def __repr__(self):
        """
        """
        return self.name
 

_apply_calibration_defaults()

_apply_calibration_defaults ( self, defaults )

protectedinherited

We pass in default calibration options from the `CAF` instance here if called.
Won't overwrite any options already set.

Definition at line 434 of file framework.py.

    def _apply_calibration_defaults(self, defaults):
        """
        We pass in default calibration options from the `CAF` instance here if called.
        Won't overwrite any options already set.
        """
        for key, value in defaults.items():
            try:
                if getattr(self, key) is None:
                    setattr(self, key, value)
            except AttributeError:
                print(f"The calibration {self.name} does not support the attribute {key}.")
 
 

_get_default_collection_attribute()

_get_default_collection_attribute ( self, attr )

protected

Definition at line 774 of file framework.py.

    def _get_default_collection_attribute(self, attr):
        """
        """
        if self.default_collection_name in self.collections:
            return getattr(self.collections[self.default_collection_name], attr)
        else:
            B2WARNING(f"You tried to get the attribute '{attr}' from the Calibration '{self.name}', "
                      "but the default collection doesn't exist."
                      f"You should use the cal.collections['CollectionName'].{attr} to access a custom "
                      "collection's attributes directly.")
            return None
 

_poll_collector()

_poll_collector ( self )

protected

Definition at line 1072 of file framework.py.

    def _poll_collector(self):
        """
        """
        while self.state == "running_collector":
            try:
                self.machine.complete()
            # ConditionError is thrown when the conditions for the transition have returned false, it's not serious.
            except ConditionError:
                try:
                    B2DEBUG(29, f"Checking if collector jobs for calibration {self.name} have failed.")
                    self.machine.fail()
                except ConditionError:
                    pass
            sleep(self.heartbeat)  # Sleep until we want to check again
 

_set_default_collection_attribute()

_set_default_collection_attribute ( self, attr, value )

protected

Definition at line 786 of file framework.py.

    def _set_default_collection_attribute(self, attr, value):
        """
        """
        if self.default_collection_name in self.collections:
            setattr(self.collections[self.default_collection_name], attr, value)
        else:
            B2WARNING(f"You tried to set the attribute '{attr}' from the Calibration '{self.name}', "
                      "but the default collection doesn't exist."
                      f"You should use the cal.collections['CollectionName'].{attr} to access a custom "
                      "collection's attributes directly.")
 

add_collection()

add_collection	(		self,
			name,
			collection )

Parameters:
    name (str): Unique name of this `Collection` in the Calibration.
    collection (`Collection`): `Collection` object to use.

Adds a new `Collection` object to the `Calibration`. Any valid Collection will be used in the Calibration.
A default Collection is automatically added but isn't valid and won't run unless you have assigned a collector
+ input files.
You can ignore the default one and only add your own custom Collections. You can configure the default from the
Calibration(...) arguments or after creating the Calibration object via directly setting the cal.collector, cal.input_files
attributes.

Definition at line 619 of file framework.py.

    def add_collection(self, name, collection):
        """
        Parameters:
            name (str): Unique name of this `Collection` in the Calibration.
            collection (`Collection`): `Collection` object to use.
 
        Adds a new `Collection` object to the `Calibration`. Any valid Collection will be used in the Calibration.
        A default Collection is automatically added but isn't valid and won't run unless you have assigned a collector
        + input files.
        You can ignore the default one and only add your own custom Collections. You can configure the default from the
        Calibration(...) arguments or after creating the Calibration object via directly setting the cal.collector, cal.input_files
        attributes.
        """
        if name not in self.collections:
            self.collections[name] = collection
        else:
            B2WARNING(f"A Collection with the name '{name}' already exists in this Calibration. It has not been added."
                      "Please use another name.")
 

algorithms() [1/2]

algorithms

(

self

)

Definition at line 903 of file framework.py.

    def algorithms(self):
        """
        """
        return self._algorithms
 

algorithms() [2/2]

algorithms	(		self,
			value )

Definition at line 910 of file framework.py.

    def algorithms(self, value):
        """
        """
        from ROOT import Belle2  # noqa: make the Belle2 namespace available
        from ROOT.Belle2 import CalibrationAlgorithm
        if isinstance(value, CalibrationAlgorithm):
            self._algorithms = [Algorithm(value)]
        else:
            B2ERROR(f"Something other than CalibrationAlgorithm instance passed in ({type(value)}). "
                    "Algorithm needs to inherit from Belle2::CalibrationAlgorithm")
 

backend_args() [1/2]

backend_args

(

self

)

Definition at line 891 of file framework.py.

    def backend_args(self):
        """
        """
        return self._get_default_collection_attribute("backend_args")
 

backend_args() [2/2]

backend_args	(		self,
			args )

Definition at line 897 of file framework.py.

    def backend_args(self, args):
        """
        """
        self._set_default_collection_attribute("backend_args", args)
 

collector() [1/2]

collector

(

self

)

Definition at line 798 of file framework.py.

    def collector(self):
        """
        """
        return self._get_default_collection_attribute("collector")
 

collector() [2/2]

collector	(		self,
			collector )

Definition at line 804 of file framework.py.

    def collector(self, collector):
        """
        """
        # check if collector is already a module or if we need to create one
        # from the name
        from basf2 import Module
        if isinstance(collector, str):
            from basf2 import register_module
            collector = register_module(collector)
        if not isinstance(collector, Module):
            B2ERROR("Collector needs to be either a Module or the name of such a module")
 
        self._set_default_collection_attribute("collector", collector)
 

dependencies_met()

dependencies_met ( self )

inherited

Checks if all of the Calibrations that this one depends on have reached a successful end state.

Definition at line 415 of file framework.py.

    def dependencies_met(self):
        """
        Checks if all of the Calibrations that this one depends on have reached a successful end state.
        """
        return all(map(lambda x: x.state == x.end_state, self.dependencies))
 

depends_on()

depends_on ( self, calibration )

inherited

Parameters:
    calibration (`CalibrationBase`): The Calibration object which will produce constants that this one depends on.

Adds dependency of this calibration on another i.e. This calibration
will not run until the dependency has completed, and the constants produced
will be used via the database chain.

You can define multiple dependencies for a single calibration simply
by calling this multiple times. Be careful when adding the calibration into
the `CAF` not to add a circular/cyclic dependency. If you do the sort will return an
empty order and the `CAF` processing  will fail.

This function appends to the `CalibrationBase.dependencies` and `CalibrationBase.future_dependencies` attributes of this
`CalibrationBase` and the input one respectively. This prevents us having to do too much recalculation later on.

Definition at line 387 of file framework.py.

    def depends_on(self, calibration):
        """
        Parameters:
            calibration (`CalibrationBase`): The Calibration object which will produce constants that this one depends on.
 
        Adds dependency of this calibration on another i.e. This calibration
        will not run until the dependency has completed, and the constants produced
        will be used via the database chain.
 
        You can define multiple dependencies for a single calibration simply
        by calling this multiple times. Be careful when adding the calibration into
        the `CAF` not to add a circular/cyclic dependency. If you do the sort will return an
        empty order and the `CAF` processing  will fail.
 
        This function appends to the `CalibrationBase.dependencies` and `CalibrationBase.future_dependencies` attributes of this
        `CalibrationBase` and the input one respectively. This prevents us having to do too much recalculation later on.
        """
        # Check that we don't have two calibration names that are the same
        if self.name != calibration.name:
            # Tests if we have the calibrations added as dependencies already and adds if not
            if calibration not in self.dependencies:
                self.dependencies.append(calibration)
            if self not in calibration.dependencies:
                calibration.future_dependencies.append(self)
        else:
            B2WARNING(f"Tried to add {calibration} as a dependency for {self} but they have the same name."
                      "Dependency was not added.")
 

failed_dependencies()

failed_dependencies ( self )

inherited

Returns the list of calibrations in our dependency list that have failed.

Definition at line 421 of file framework.py.

    def failed_dependencies(self):
        """
        Returns the list of calibrations in our dependency list that have failed.
        """
        failed = []
        # Use local variable to avoid doxygen "no uniquely matching class member" warning
        # (multiple classes in this file define self.dependencies with doc comments)
        dependencies = self.dependencies
        for calibration in dependencies:
            if calibration.state == self.fail_state:
                failed.append(calibration)
        return failed
 

files_to_iovs() [1/2]

files_to_iovs

(

self

)

Reimplemented from CalibrationBase.

Definition at line 831 of file framework.py.

    def files_to_iovs(self):
        """
        """
        return self._get_default_collection_attribute("files_to_iovs")
 

files_to_iovs() [2/2]

files_to_iovs	(		self,
			file_map )

Reimplemented from CalibrationBase.

Definition at line 837 of file framework.py.

    def files_to_iovs(self, file_map):
        """
        """
        self._set_default_collection_attribute("files_to_iovs", file_map)
 

input_files() [1/2]

input_files

(

self

)

Reimplemented from CalibrationBase.

Definition at line 819 of file framework.py.

    def input_files(self):
        """
        """
        return self._get_default_collection_attribute("input_files")
 

input_files() [2/2]

input_files	(		self,
			files )

Reimplemented from CalibrationBase.

Definition at line 825 of file framework.py.

    def input_files(self, files):
        """
        """
        self._set_default_collection_attribute("input_files", files)
 

is_valid()

is_valid

(

self

)

A full calibration consists of a collector AND an associated algorithm AND input_files.

Returns False if:
    1) We are missing any of the above.
    2) There are multiple Collections and the Collectors have mis-matched granularities.
    3) Any of our Collectors have granularities that don't match what our Strategy can use.

Reimplemented from CalibrationBase.

Definition at line 638 of file framework.py.

    def is_valid(self):
        """
        A full calibration consists of a collector AND an associated algorithm AND input_files.
 
        Returns False if:
            1) We are missing any of the above.
            2) There are multiple Collections and the Collectors have mis-matched granularities.
            3) Any of our Collectors have granularities that don't match what our Strategy can use.
        """
        if not self.algorithms:
            B2WARNING(f"Empty algorithm list for {self.name}.")
            return False
 
        if not any([collection.is_valid() for collection in self.collections.values()]):
            B2WARNING(f"No valid Collections for {self.name}.")
            return False
 
        granularities = []
        for collection in self.collections.values():
            if collection.is_valid():
                collector_params = collection.collector.available_params()
                for param in collector_params:
                    if param.name == "granularity":
                        granularities.append(param.values)
        if len(set(granularities)) > 1:
            B2WARNING("Multiple different granularities set for the Collections in this Calibration.")
            return False
 
        for alg in self.algorithms:
            alg_type = type(alg.algorithm).__name__
            incorrect_gran = [granularity not in alg.strategy.allowed_granularities for granularity in granularities]
            if any(incorrect_gran):
                B2WARNING(f"Selected strategy for {alg_type} does not match a collector's granularity.")
                return False
        return True
 

iteration() [1/2]

iteration

(

self

)

Retrieves the current iteration number in the database file.

Returns:
    int: The current iteration number

Definition at line 1108 of file framework.py.

    def iteration(self):
        """
        Retrieves the current iteration number in the database file.
 
        Returns:
            int: The current iteration number
        """
        with CAFDB(self._db_path, read_only=True) as db:
            iteration = db.get_calibration_value(self.name, "iteration")
        return iteration
 

iteration() [2/2]

iteration	(		self,
			iteration )

Definition at line 1120 of file framework.py.

    def iteration(self, iteration):
        """
        """
        B2DEBUG(29, f"Setting {self.name} to {iteration}.")
        with CAFDB(self._db_path) as db:
            db.update_calibration_value(self.name, "iteration", iteration)
        B2DEBUG(29, f"{self.name} set to {self.iteration}.")
 
 

max_collector_jobs() [1/2]

max_collector_jobs

(

self

)

Definition at line 879 of file framework.py.

    def max_collector_jobs(self):
        """
        """
        return self._get_default_collection_attribute("max_collector_jobs")
 

max_collector_jobs() [2/2]

max_collector_jobs	(		self,
			max_jobs )

Definition at line 885 of file framework.py.

    def max_collector_jobs(self, max_jobs):
        """
        """
        self._set_default_collection_attribute("max_collector_jobs", max_jobs)
 

max_files_per_collector_job() [1/2]

max_files_per_collector_job

(

self

)

Definition at line 867 of file framework.py.

    def max_files_per_collector_job(self):
        """
        """
        return self._get_default_collection_attribute("max_files_per_collector_job")
 

max_files_per_collector_job() [2/2]

max_files_per_collector_job	(		self,
			max_files )

Definition at line 873 of file framework.py.

    def max_files_per_collector_job(self, max_files):
        """
        """
        self._set_default_collection_attribute("max_files_per_collector_job", max_files)
 

output_patterns() [1/2]

output_patterns

(

self

)

Definition at line 855 of file framework.py.

    def output_patterns(self):
        """
        """
        return self._get_default_collection_attribute("output_patterns")
 

output_patterns() [2/2]

output_patterns	(		self,
			patterns )

Definition at line 861 of file framework.py.

    def output_patterns(self, patterns):
        """
        """
        self._set_default_collection_attribute("output_patterns", patterns)
 

pre_algorithms() [1/2]

pre_algorithms

(

self

)

Callback run prior to each algorithm iteration.

Definition at line 943 of file framework.py.

    def pre_algorithms(self):
        """
        Callback run prior to each algorithm iteration.
        """
        return [alg.pre_algorithm for alg in self.algorithms]
 

pre_algorithms() [2/2]

pre_algorithms	(		self,
			func )

Definition at line 951 of file framework.py.

    def pre_algorithms(self, func):
        """
        """
        if func:
            for alg in self.algorithms:
                alg.pre_algorithm = func
        else:
            B2ERROR("Something evaluated as False passed in as pre_algorithm function.")
 

pre_collector_path() [1/2]

pre_collector_path

(

self

)

Definition at line 843 of file framework.py.

    def pre_collector_path(self):
        """
        """
        return self._get_default_collection_attribute("pre_collector_path")
 

pre_collector_path() [2/2]

pre_collector_path	(		self,
			path )

Definition at line 849 of file framework.py.

    def pre_collector_path(self, path):
        """
        """
        self._set_default_collection_attribute("pre_collector_path", path)
 

reset_database()

reset_database	(		self,
			apply_to_default_collection = True )

Keyword Arguments:
    apply_to_default_collection (bool): Should we also reset the default collection?

Remove everything in the database_chain of this Calibration, including the default central database tag automatically
included from `basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`. This will NOT affect the
database chain of any `Collection` other than the default one. You can prevent the default Collection from having its chain
reset by setting 'apply_to_default_collection' to False.

Definition at line 674 of file framework.py.

    def reset_database(self, apply_to_default_collection=True):
        """
        Keyword Arguments:
            apply_to_default_collection (bool): Should we also reset the default collection?
 
        Remove everything in the database_chain of this Calibration, including the default central database tag automatically
        included from `basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`. This will NOT affect the
        database chain of any `Collection` other than the default one. You can prevent the default Collection from having its chain
        reset by setting 'apply_to_default_collection' to False.
        """
        self.database_chain = []
        if self.default_collection_name in self.collections and apply_to_default_collection:
            self.collections[self.default_collection_name].reset_database()
 

run()

run

(

self

)

Main logic of the Calibration object.
Will be run in a new Thread by calling the start() method.

Reimplemented from CalibrationBase.

Definition at line 1017 of file framework.py.

    def run(self):
        """
        Main logic of the Calibration object.
        Will be run in a new Thread by calling the start() method.
        """
        with CAFDB(self._db_path, read_only=True) as db:
            initial_state = db.get_calibration_value(self.name, "checkpoint")
            initial_iteration = db.get_calibration_value(self.name, "iteration")
        B2INFO(f"Initial status of {self.name} found to be state={initial_state}, iteration={initial_iteration}")
        self.machine = CalibrationMachine(self,
                                          iov_to_calibrate=self.iov,
                                          initial_state=initial_state,
                                          iteration=initial_iteration)
        
        self.state = initial_state
        self.machine.root_dir = Path(os.getcwd(), self.name)
        self.machine.collector_backend = self.backend
 
        # Before we start running, let's clean up any iteration directories from iterations above our initial one.
        # Should prevent confusion between attempts if we fail again.
        all_iteration_paths = find_int_dirs(self.machine.root_dir)
        for iteration_path in all_iteration_paths:
            if int(iteration_path.name) > initial_iteration:
                shutil.rmtree(iteration_path)
 
        while self.state != self.end_state and self.state != self.fail_state:
            if self.state == "init":
                try:
                    B2INFO(f"Attempting collector submission for calibration {self.name}.")
                    self.machine.submit_collector()
                except Exception as err:
                    B2FATAL(str(err))
 
                self._poll_collector()
 
            # If we failed take us to the final fail state
            if self.state == "collector_failed":
                self.machine.fail_fully()
                return
 
            # It's possible that we might raise an error while attempting to run due
            # to some problems e.g. Missing collector output files
            # We catch the error and exit with failed state so the CAF will stop
            try:
                B2INFO(f"Attempting to run algorithms for calibration {self.name}.")
                self.machine.run_algorithms()
            except MachineError as err:
                B2ERROR(str(err))
                self.machine.fail()
 
            # If we failed take us to the final fail state
            if self.machine.state == "algorithms_failed":
                self.machine.fail_fully()
                return
 

state() [1/2]

state

(

self

)

The current major state of the calibration in the database file. The machine may have a different state.

Definition at line 1088 of file framework.py.

    def state(self):
        """
        The current major state of the calibration in the database file. The machine may have a different state.
        """
        with CAFDB(self._db_path, read_only=True) as db:
            state = db.get_calibration_value(self.name, "state")
        return state
 

state() [2/2]

state	(		self,
			state )

Definition at line 1097 of file framework.py.

    def state(self, state):
        """
        """
        B2DEBUG(29, f"Setting {self.name} to state {state}.")
        with CAFDB(self._db_path) as db:
            db.update_calibration_value(self.name, "state", str(state))
            if state in self.checkpoint_states:
                db.update_calibration_value(self.name, "checkpoint", str(state))
        B2DEBUG(29, f"{self.name} set to {state}.")
 

strategies() [1/2]

strategies

(

self

)

The `caf.strategies.AlgorithmStrategy` or `list` of them used when running the algorithm(s).

Definition at line 978 of file framework.py.

    def strategies(self):
        """
        The `caf.strategies.AlgorithmStrategy` or `list` of them used when running the algorithm(s).
        """
        return [alg.strategy for alg in self.algorithms]
 

strategies() [2/2]

strategies	(		self,
			strategy )

Definition at line 986 of file framework.py.

    def strategies(self, strategy):
        """
        """
        if strategy:
            for alg in self.algorithms:
                alg.strategy = strategy
        else:
            B2ERROR("Something evaluated as False passed in as a strategy.")
 

use_central_database()

use_central_database	(		self,
			global_tag,
			apply_to_default_collection = True )

Parameters:
    global_tag (str): The central database global tag to use for this calibration.

Keyword Arguments:
    apply_to_default_collection (bool): Should we also call use_central_database on the default collection (if it exists)

Using this allows you to append a central database to the database chain for this calibration.
The default database chain is just the central one from
`basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`.
To turn off central database completely or use a custom tag as the base, you should call `Calibration.reset_database`
and start adding databases with `Calibration.use_local_database` and `Calibration.use_central_database`.

Note that the database chain attached to the `Calibration` will only affect the default `Collection` (if it exists),
and the algorithm processes. So calling:

>> cal.use_central_database("global_tag")

will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
assigned to

>> cal.collections['default'].database_chain

But calling

>> cal.use_central_database(file_path, payload_dir, False)

will add the database to the Algorithm processes, but leave the default Collection database chain untouched.
So if you have multiple Collections in this Calibration *their database chains are separate*.
To specify an additional `CentralDatabase` for a different collection, you will have to call:

>> cal.collections['OtherCollection'].use_central_database("global_tag")

Definition at line 688 of file framework.py.

    def use_central_database(self, global_tag, apply_to_default_collection=True):
        """
        Parameters:
            global_tag (str): The central database global tag to use for this calibration.
 
        Keyword Arguments:
            apply_to_default_collection (bool): Should we also call use_central_database on the default collection (if it exists)
 
        Using this allows you to append a central database to the database chain for this calibration.
        The default database chain is just the central one from
        `basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`.
        To turn off central database completely or use a custom tag as the base, you should call `Calibration.reset_database`
        and start adding databases with `Calibration.use_local_database` and `Calibration.use_central_database`.
 
        Note that the database chain attached to the `Calibration` will only affect the default `Collection` (if it exists),
        and the algorithm processes. So calling:
 
        >> cal.use_central_database("global_tag")
 
        will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
        assigned to
 
        >> cal.collections['default'].database_chain
 
        But calling
 
        >> cal.use_central_database(file_path, payload_dir, False)
 
        will add the database to the Algorithm processes, but leave the default Collection database chain untouched.
        So if you have multiple Collections in this Calibration *their database chains are separate*.
        To specify an additional `CentralDatabase` for a different collection, you will have to call:
 
        >> cal.collections['OtherCollection'].use_central_database("global_tag")
        """
        central_db = CentralDatabase(global_tag)
        self.database_chain.append(central_db)
        if self.default_collection_name in self.collections and apply_to_default_collection:
            self.collections[self.default_collection_name].use_central_database(global_tag)
 

use_local_database()

use_local_database	(		self,
			filename,
			directory = "",
			apply_to_default_collection = True )

Parameters:
    filename (str): The path to the database.txt of the local database

Keyword Argumemts:
    directory (str): The path to the payloads directory for this local database.
    apply_to_default_collection (bool): Should we also call use_local_database on the default collection (if it exists)

Append a local database to the chain for this calibration.
You can call this function multiple times and each database will be added to the chain IN ORDER.
The databases are applied to this calibration ONLY.
The Local and Central databases applied via these functions are applied to the algorithm processes and optionally
the default `Collection` job as a database chain.
There are other databases applied to the processes later, checked by basf2 in this order:

1) Local Database from previous iteration of this Calibration.
2) Local Database chain from output of previous dependent Calibrations.
3) This chain of Local and Central databases where the last added is checked first.

Note that this function on the `Calibration` object will only affect the default `Collection` if it exists and if
'apply_to_default_collection' remains True. So calling:

>> cal.use_local_database(file_path, payload_dir)

will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
assigned to

>> cal.collections['default'].database_chain

But calling

>> cal.use_local_database(file_path, payload_dir, False)

will add the database to the Algorithm processes, but leave the default Collection database chain untouched.

If you have multiple Collections in this Calibration *their database chains are separate*.
To specify an additional `LocalDatabase` for a different collection, you will have to call:

>> cal.collections['OtherCollection'].use_local_database(file_path, payload_dir)

Definition at line 727 of file framework.py.

    def use_local_database(self, filename, directory="", apply_to_default_collection=True):
        """
        Parameters:
            filename (str): The path to the database.txt of the local database
 
        Keyword Argumemts:
            directory (str): The path to the payloads directory for this local database.
            apply_to_default_collection (bool): Should we also call use_local_database on the default collection (if it exists)
 
        Append a local database to the chain for this calibration.
        You can call this function multiple times and each database will be added to the chain IN ORDER.
        The databases are applied to this calibration ONLY.
        The Local and Central databases applied via these functions are applied to the algorithm processes and optionally
        the default `Collection` job as a database chain.
        There are other databases applied to the processes later, checked by basf2 in this order:
 
        1) Local Database from previous iteration of this Calibration.
        2) Local Database chain from output of previous dependent Calibrations.
        3) This chain of Local and Central databases where the last added is checked first.
 
        Note that this function on the `Calibration` object will only affect the default `Collection` if it exists and if
        'apply_to_default_collection' remains True. So calling:
 
        >> cal.use_local_database(file_path, payload_dir)
 
        will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
        assigned to
 
        >> cal.collections['default'].database_chain
 
        But calling
 
        >> cal.use_local_database(file_path, payload_dir, False)
 
        will add the database to the Algorithm processes, but leave the default Collection database chain untouched.
 
        If you have multiple Collections in this Calibration *their database chains are separate*.
        To specify an additional `LocalDatabase` for a different collection, you will have to call:
 
        >> cal.collections['OtherCollection'].use_local_database(file_path, payload_dir)
 
        """
        local_db = LocalDatabase(filename, directory)
        self.database_chain.append(local_db)
        if self.default_collection_name in self.collections and apply_to_default_collection:
            self.collections[self.default_collection_name].use_local_database(filename, directory)
 

Member Data Documentation

_algorithms

list _algorithms = []

protected

Internal calibration algorithms stored for this calibration.

Definition at line 555 of file framework.py.

_db_path

_db_path = None

protected

Location of a SQLite database that will save the state of the calibration so that it can be restarted from failure.

Definition at line 617 of file framework.py.

alg_output_dir

str alg_output_dir = "algorithm_output"

static

Subdirectory name for algorithm output.

Definition at line 532 of file framework.py.

algorithms

algorithms = algorithms

Algorithm classes that will be run by this Calibration.

You should set this attribute to either a single CalibrationAlgorithm C++ class, or a list of them if you want to run multiple CalibrationAlgorithms using one CalibrationCollectorModule.

Definition at line 575 of file framework.py.

algorithms_runner

algorithms_runner = runners.SeqAlgorithmsRunner

The class that runs all the algorithms in this Calibration using their assigned :py:class:caf.strategies.AlgorithmStrategy.

Plugin your own runner class to change how your calibration will run the list of algorithms.

Definition at line 603 of file framework.py.

backend

backend = None

The backend <backends.Backend> we'll use for our collector submission in this calibration.

If None it will be set by the CAF used to run this Calibration (recommended!).

Definition at line 606 of file framework.py.

checkpoint_states

list checkpoint_states = ["init", "collector_completed", "completed"]

static

Checkpoint states which we are allowed to restart from.

Definition at line 534 of file framework.py.

collections

dict collections = {}

Collections stored for this calibration.

Definition at line 553 of file framework.py.

collector_full_update_interval

int collector_full_update_interval = 30

While checking if the collector is finished we don't bother wastefully checking every subjob's status.

We exit once we find the first subjob that isn't ready. But after this interval has elapsed we do a full :py:meth:caf.backends.Job.update_status call and print the fraction of SubJobs completed.

Definition at line 611 of file framework.py.

database_chain

list database_chain = []

The database chain that is applied to the algorithms.

This is often updated at the same time as the database chain for the default Collection.

Definition at line 593 of file framework.py.

default_collection_name

default_collection_name = "default"

static

Default collection name.

Definition at line 536 of file framework.py.

dependencies

list dependencies = []

inherited

List of calibration objects, where each one is a dependency of this one.

Definition at line 349 of file framework.py.

end_state

str end_state = "completed"

staticinherited

The name of the successful completion state.

The :py:class:CAF will use this as the state to decide when the Calibration is completed.

Definition at line 335 of file framework.py.

fail_state

str fail_state = "failed"

staticinherited

The name of the failure state.

The :py:class:CAF will use this as the state to decide when the Calibration failed.

Definition at line 338 of file framework.py.

future_dependencies

list future_dependencies = []

inherited

List of calibration objects that depend on this one.

Definition at line 347 of file framework.py.

heartbeat

heartbeat = 3

This calibration's sleep time before rechecking to see if it can move state.

Definition at line 613 of file framework.py.

ignored_runs

ignored_runs = None

List of ExpRun that will be ignored by this Calibration.

This runs will not have Collector jobs run on them (if possible). And the algorithm execution will exclude them from a ExpRun list. However, the algorithm execution may merge IoVs of final payloads to cover the 'gaps' caused by these runs. You should pay attention to what the AlgorithmStrategy you choose will do in these cases.

Definition at line 585 of file framework.py.

iov

iov = None

inherited

IoV which will be calibrated.

This is set by the CAF itself when calling CAF.run()

Definition at line 364 of file framework.py.

jobs_to_submit

list jobs_to_submit = []

inherited

A simple list of jobs that this Calibration wants submitted at some point.

Definition at line 371 of file framework.py.

machine

machine = None

The caf.state_machines.CalibrationMachine that we will run to process this calibration start to finish.

Definition at line 615 of file framework.py.

max_iterations

max_iterations = None

Variable to define the maximum number of iterations for this calibration specifically.

It overrides the CAF calibration_defaults value if set.

Definition at line 580 of file framework.py.

moves

list moves = ["submit_collector", "complete", "run_algorithms", "iterate", "fail_fully"]

static

Allowed transitions that we will use to progress.

Definition at line 530 of file framework.py.

name

name = name

inherited

Name of calibration object.

This must be unique when adding into the py:class:CAF.

Definition at line 345 of file framework.py.

output_database_dir

str output_database_dir = ""

inherited

The directory where we'll store the local database payloads from this calibration.

Definition at line 366 of file framework.py.

results

dict results = {}

Output results of algorithms for each iteration.

Definition at line 577 of file framework.py.

save_payloads

bool save_payloads = True

inherited

Marks this Calibration as one which has payloads that should be copied and uploaded.

Defaults to True, and should only be False if this is an intermediate Calibration who's payloads are never needed.

Definition at line 369 of file framework.py.

state

str state = initial_state

state

Definition at line 1031 of file framework.py.

strategies

strategies = strategies.SingleIOV

The strategy that the algorithm(s) will be run against.

Assign a list of strategies the same length as the number of algorithms, or assign a single strategy to apply it to all algorithms in this Calibration. You can see the choices in :py:mod:caf.strategies.

Definition at line 590 of file framework.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/framework.py