Inheritance diagram for Calibration:

Public Member Functions
	__init__ (self, name, collector=None, algorithms=None, input_files=None, pre_collector_path=None, database_chain=None, output_patterns=None, max_files_per_collector_job=None, max_collector_jobs=None, backend_args=None)

	add_collection (self, name, collection)

	is_valid (self)

	reset_database (self, apply_to_default_collection=True)

	use_central_database (self, global_tag, apply_to_default_collection=True)

	use_local_database (self, filename, directory="", apply_to_default_collection=True)

	collector (self)

	collector (self, collector)

	input_files (self)

	input_files (self, files)

	files_to_iovs (self)

	files_to_iovs (self, file_map)

	pre_collector_path (self)

	pre_collector_path (self, path)

	output_patterns (self)

	output_patterns (self, patterns)

	max_files_per_collector_job (self)

	max_files_per_collector_job (self, max_files)

	max_collector_jobs (self)

	max_collector_jobs (self, max_jobs)

	backend_args (self)

	backend_args (self, args)

	algorithms (self)

	algorithms (self, value)

	pre_algorithms (self)

	pre_algorithms (self, func)

	strategies (self)

	strategies (self, strategy)

	__repr__ (self)

	run (self)

	state (self)

	state (self, state)

	iteration (self)

	iteration (self, iteration)

	depends_on (self, calibration)

	dependencies_met (self)

	failed_dependencies (self)

Public Attributes
dict	collections = {}
	Collections stored for this calibration.

	algorithms = algorithms
	`Algorithm` classes that will be run by this `Calibration`.

dict	results = {}
	Output results of algorithms for each iteration.

	max_iterations = None
	Variable to define the maximum number of iterations for this calibration specifically.

	ignored_runs = None
	List of ExpRun that will be ignored by this Calibration.

	strategies = strategies.SingleIOV
	The strategy that the algorithm(s) will be run against.

list	database_chain = database_chain
	The database chain that is applied to the algorithms.

	algorithms_runner = runners.SeqAlgorithmsRunner
	The class that runs all the algorithms in this Calibration using their assigned :py:class:`caf.strategies.AlgorithmStrategy`.

	backend = None
	The `backend <backends.Backend>` we'll use for our collector submission in this calibration.

int	collector_full_update_interval = 30
	While checking if the collector is finished we don't bother wastefully checking every subjob's status.

int	heartbeat = 3
	This calibration's sleep time before rechecking to see if it can move state.

	machine = None
	The `caf.state_machines.CalibrationMachine` that we will run to process this calibration start to finish.

str	state = initial_state
	state

	name = name
	Name of calibration object.

list	future_dependencies = []
	List of calibration objects that depend on this one.

list	dependencies = []
	List of calibration objects, where each one is a dependency of this one.

	iov = None
	IoV which will be calibrated.

str	output_database_dir = ""
	The directory where we'll store the local database payloads from this calibration.

bool	save_payloads = True
	Marks this Calibration as one which has payloads that should be copied and uploaded.

list	jobs_to_submit = []
	A simple list of jobs that this Calibration wants submitted at some point.

Static Public Attributes
list	moves = ["submit_collector", "complete", "run_algorithms", "iterate", "fail_fully"]
	Allowed transitions that we will use to progress.

str	alg_output_dir = "algorithm_output"
	Subdirectory name for algorithm output.

list	checkpoint_states = ["init", "collector_completed", "completed"]
	Checkpoint states which we are allowed to restart from.

str	default_collection_name = "default"
	Default collection name.

str	end_state = "completed"
	The name of the successful completion state.

str	fail_state = "failed"
	The name of the failure state.

Protected Member Functions
	_get_default_collection_attribute (self, attr)

	_set_default_collection_attribute (self, attr, value)

	_ (self, value)

	_ (self, values)

	_ (self, values)

	_poll_collector (self)

	_apply_calibration_defaults (self, defaults)

Protected Attributes
list	_algorithms = []
	Internal calibration algorithms stored for this calibration.

	_db_path = None
	Location of a SQLite database that will save the state of the calibration so that it can be restarted from failure.

Detailed Description

Every Calibration object must have at least one collector at least one algorithm.
You have the option to add in your collector/algorithm by argument here, or add them
later by changing the properties.

If you plan to use multiple `Collection` objects I recommend that you only set the name here and add the Collections
separately via `add_collection()`.

Parameters:
    name (str): Name of this calibration. It should be unique for use in the `CAF`
Keyword Arguments:
    collector (str, `basf2.Module`): Should be set to a CalibrationCollectorModule() or a string with the module name.
    algorithms (list, ``ROOT.Belle2.CalibrationAlgorithm``): The algorithm(s) to use for this `Calibration`.
    input_files (str, list[str]): Input files for use by this Calibration. May contain wildcards usable by `glob.glob`

A Calibration won't be valid in the `CAF` until it has all of these four attributes set. For example:

>>> cal = Calibration('TestCalibration1')
>>> col1 = register_module('CaTest')
>>> cal.add_collection('TestColl', col1)

or equivalently

>>> cal = Calibration('TestCalibration1', 'CaTest')

If you want to run a basf2 :py:class:`path <basf2.Path>` before your collector module when running over data

>>> cal.pre_collector_path = my_basf2_path

You don't have to put a RootInput module in this pre-collection path, but you can if
you need some special parameters. If you want to process sroot files the you have to explicitly add
SeqRootInput to your pre-collection path.
The inputFileNames parameter of (Seq)RootInput will be set by the CAF automatically for you.


You can use optional arguments to pass in some/all during initialisation of the `Calibration` class

>>> cal = Calibration( 'TestCalibration1', 'CaTest', [alg1,alg2], ['/path/to/file.root'])

you can change the input file list later on, before running with `CAF`

>>> cal.input_files = ['path/to/*.root', 'other/path/to/file2.root']

If you have multiple collections from calling `add_collection()` then you should instead set the pre_collector_path,
input_files, database chain etc from there. See `Collection`.

Adding the CalibrationAlgorithm(s) is easy

>>> alg1 = TestAlgo()
>>> cal.algorithms = alg1

Or equivalently

>>> cal.algorithms = [alg1]

Or for multiple algorithms for one collector

>>> alg2 = TestAlgo()
>>> cal.algorithms = [alg1, alg2]

Note that when you set the algorithms, they are automatically wrapped and stored as a Python class
`Algorithm`. To access the C++ algorithm clas underneath directly do:

>>> cal.algorithms[i].algorithm

If you have a setup function that you want to run before each of the algorithms, set that with

>>> cal.pre_algorithms = my_function_object

If you want a different setup for each algorithm use a list with the same number of elements
as your algorithm list.

>>> cal.pre_algorithms = [my_function1, my_function2, ...]

You can also specify the dependencies of the calibration on others

>>> cal.depends_on(cal2)

By doing this, the `CAF` will respect the ordering of the calibrations and will pass the
calibration constants created by earlier completed calibrations to dependent ones.

Definition at line 444 of file framework.py.

Constructor & Destructor Documentation

◆ init()

__init__	(	self,
		name,
		collector = None,
		algorithms = None,
		input_files = None,
		pre_collector_path = None,
		database_chain = None,
		output_patterns = None,
		max_files_per_collector_job = None,
		max_collector_jobs = None,
		backend_args = None )

Definition at line 535 of file framework.py.

                 ):
        """
        """
        
        self.collections = {}
        
        self._algorithms = []
 
        # Default collection added, will have None type and requires setting later via `self.collector`, or will take the
        # CollectorModule/module name directly.
        self.add_collection(self.default_collection_name,
                            Collection(collector,
                                       input_files,
                                       pre_collector_path,
                                       database_chain,
                                       output_patterns,
                                       max_files_per_collector_job,
                                       max_collector_jobs,
                                       backend_args
                                       ))
 
        super().__init__(name, input_files)
        if algorithms:
            
            self.algorithms = algorithms
        
        self.results = {}
        
        self.max_iterations = None
        
        self.ignored_runs = None
        if self.algorithms:
            
            self.strategies = strategies.SingleIOV
        if database_chain:
            
            self.database_chain = database_chain
        else:
            self.database_chain = []
            # This database is already applied to the `Collection` automatically, so don't do it again
            for tag in reversed(b2conditions.default_globaltags):
                self.use_central_database(tag, apply_to_default_collection=False)
        
        self.algorithms_runner = runners.SeqAlgorithmsRunner
        
        self.backend = None
        
        self.collector_full_update_interval = 30
        
        self.heartbeat = 3
        
        self.machine = None
        
        self._db_path = None
 

Member Function Documentation

◆ _() [1/3]

_	(		self,
			value )

protected

Alternate algorithms setter for lists and tuples of CalibrationAlgorithms.

Definition at line 920 of file framework.py.

    def _(self, value):
        """
        Alternate algorithms setter for lists and tuples of CalibrationAlgorithms.
        """
        from ROOT import Belle2  # noqa: make the Belle2 namespace available
        from ROOT.Belle2 import CalibrationAlgorithm
        if value:
            self._algorithms = []
            for alg in value:
                if isinstance(alg, CalibrationAlgorithm):
                    self._algorithms.append(Algorithm(alg))
                else:
                    B2ERROR(f"Something other than CalibrationAlgorithm instance passed in {type(value)}."
                            "Algorithm needs to inherit from Belle2::CalibrationAlgorithm")
 

◆ _() [2/3]

_	(		self,
			values )

protected

Alternate pre_algorithms setter for lists and tuples of functions, should be one per algorithm.

Definition at line 955 of file framework.py.

    def _(self, values):
        """
        Alternate pre_algorithms setter for lists and tuples of functions, should be one per algorithm.
        """
        if values:
            if len(values) == len(self.algorithms):
                for func, alg in zip(values, self.algorithms):
                    alg.pre_algorithm = func
            else:
                B2ERROR("Number of functions and number of algorithms doesn't match.")
        else:
            B2ERROR("Empty container passed in for pre_algorithm functions")
 

◆ _() [3/3]

_	(		self,
			values )

protected

Alternate strategies setter for lists and tuples of functions, should be one per algorithm.

Definition at line 988 of file framework.py.

    def _(self, values):
        """
        Alternate strategies setter for lists and tuples of functions, should be one per algorithm.
        """
        if values:
            if len(values) == len(self.algorithms):
                for strategy, alg in zip(strategies, self.algorithms):
                    alg.strategy = strategy
            else:
                B2ERROR("Number of strategies and number of algorithms doesn't match.")
        else:
            B2ERROR("Empty container passed in for strategies list")
 

◆ repr()

__repr__ ( self )

Definition at line 1001 of file framework.py.

    def __repr__(self):
        """
        """
        return self.name
 

◆ _apply_calibration_defaults()

_apply_calibration_defaults	(		self,
			defaults )

protectedinherited

We pass in default calibration options from the `CAF` instance here if called.
Won't overwrite any options already set.

Definition at line 431 of file framework.py.

    def _apply_calibration_defaults(self, defaults):
        """
        We pass in default calibration options from the `CAF` instance here if called.
        Won't overwrite any options already set.
        """
        for key, value in defaults.items():
            try:
                if getattr(self, key) is None:
                    setattr(self, key, value)
            except AttributeError:
                print(f"The calibration {self.name} does not support the attribute {key}.")
 
 

◆ _get_default_collection_attribute()

_get_default_collection_attribute	(		self,
			attr )

protected

Definition at line 771 of file framework.py.

    def _get_default_collection_attribute(self, attr):
        """
        """
        if self.default_collection_name in self.collections:
            return getattr(self.collections[self.default_collection_name], attr)
        else:
            B2WARNING(f"You tried to get the attribute '{attr}' from the Calibration '{self.name}', "
                      "but the default collection doesn't exist."
                      f"You should use the cal.collections['CollectionName'].{attr} to access a custom "
                      "collection's attributes directly.")
            return None
 

◆ _poll_collector()

_poll_collector ( self )

protected

Definition at line 1061 of file framework.py.

    def _poll_collector(self):
        """
        """
        while self.state == "running_collector":
            try:
                self.machine.complete()
            # ConditionError is thrown when the conditions for the transition have returned false, it's not serious.
            except ConditionError:
                try:
                    B2DEBUG(29, f"Checking if collector jobs for calibration {self.name} have failed.")
                    self.machine.fail()
                except ConditionError:
                    pass
            sleep(self.heartbeat)  # Sleep until we want to check again
 

◆ _set_default_collection_attribute()

_set_default_collection_attribute	(	self,
		attr,
		value )

protected

Definition at line 783 of file framework.py.

    def _set_default_collection_attribute(self, attr, value):
        """
        """
        if self.default_collection_name in self.collections:
            setattr(self.collections[self.default_collection_name], attr, value)
        else:
            B2WARNING(f"You tried to set the attribute '{attr}' from the Calibration '{self.name}', "
                      "but the default collection doesn't exist."
                      f"You should use the cal.collections['CollectionName'].{attr} to access a custom "
                      "collection's attributes directly.")
 

◆ add_collection()

add_collection	(	self,
		name,
		collection )

Parameters:
    name (str): Unique name of this `Collection` in the Calibration.
    collection (`Collection`): `Collection` object to use.

Adds a new `Collection` object to the `Calibration`. Any valid Collection will be used in the Calibration.
A default Collection is automatically added but isn't valid and won't run unless you have assigned a collector
+ input files.
You can ignore the default one and only add your own custom Collections. You can configure the default from the
Calibration(...) arguments or after creating the Calibration object via directly setting the cal.collector, cal.input_files
attributes.

Definition at line 616 of file framework.py.

    def add_collection(self, name, collection):
        """
        Parameters:
            name (str): Unique name of this `Collection` in the Calibration.
            collection (`Collection`): `Collection` object to use.
 
        Adds a new `Collection` object to the `Calibration`. Any valid Collection will be used in the Calibration.
        A default Collection is automatically added but isn't valid and won't run unless you have assigned a collector
        + input files.
        You can ignore the default one and only add your own custom Collections. You can configure the default from the
        Calibration(...) arguments or after creating the Calibration object via directly setting the cal.collector, cal.input_files
        attributes.
        """
        if name not in self.collections:
            self.collections[name] = collection
        else:
            B2WARNING(f"A Collection with the name '{name}' already exists in this Calibration. It has not been added."
                      "Please use another name.")
 

◆ algorithms() [1/2]

algorithms ( self )

Definition at line 900 of file framework.py.

    def algorithms(self):
        """
        """
        return self._algorithms
 

◆ algorithms() [2/2]

algorithms	(		self,
			value )

Definition at line 907 of file framework.py.

    def algorithms(self, value):
        """
        """
        from ROOT import Belle2  # noqa: make the Belle2 namespace available
        from ROOT.Belle2 import CalibrationAlgorithm
        if isinstance(value, CalibrationAlgorithm):
            self._algorithms = [Algorithm(value)]
        else:
            B2ERROR(f"Something other than CalibrationAlgorithm instance passed in ({type(value)}). "
                    "Algorithm needs to inherit from Belle2::CalibrationAlgorithm")
 

◆ backend_args() [1/2]

backend_args ( self )

Definition at line 888 of file framework.py.

    def backend_args(self):
        """
        """
        return self._get_default_collection_attribute("backend_args")
 

◆ backend_args() [2/2]

backend_args	(		self,
			args )

Definition at line 894 of file framework.py.

    def backend_args(self, args):
        """
        """
        self._set_default_collection_attribute("backend_args", args)
 

◆ collector() [1/2]

collector ( self )

Definition at line 795 of file framework.py.

    def collector(self):
        """
        """
        return self._get_default_collection_attribute("collector")
 

◆ collector() [2/2]

collector	(		self,
			collector )

Definition at line 801 of file framework.py.

    def collector(self, collector):
        """
        """
        # check if collector is already a module or if we need to create one
        # from the name
        from basf2 import Module
        if isinstance(collector, str):
            from basf2 import register_module
            collector = register_module(collector)
        if not isinstance(collector, Module):
            B2ERROR("Collector needs to be either a Module or the name of such a module")
 
        self._set_default_collection_attribute("collector", collector)
 

◆ dependencies_met()

dependencies_met ( self )

inherited

Checks if all of the Calibrations that this one depends on have reached a successful end state.

Definition at line 415 of file framework.py.

    def dependencies_met(self):
        """
        Checks if all of the Calibrations that this one depends on have reached a successful end state.
        """
        return all(map(lambda x: x.state == x.end_state, self.dependencies))
 

◆ depends_on()

depends_on	(		self,
			calibration )

inherited

Parameters:
    calibration (`CalibrationBase`): The Calibration object which will produce constants that this one depends on.

Adds dependency of this calibration on another i.e. This calibration
will not run until the dependency has completed, and the constants produced
will be used via the database chain.

You can define multiple dependencies for a single calibration simply
by calling this multiple times. Be careful when adding the calibration into
the `CAF` not to add a circular/cyclic dependency. If you do the sort will return an
empty order and the `CAF` processing  will fail.

This function appends to the `CalibrationBase.dependencies` and `CalibrationBase.future_dependencies` attributes of this
`CalibrationBase` and the input one respectively. This prevents us having to do too much recalculation later on.

Definition at line 387 of file framework.py.

    def depends_on(self, calibration):
        """
        Parameters:
            calibration (`CalibrationBase`): The Calibration object which will produce constants that this one depends on.
 
        Adds dependency of this calibration on another i.e. This calibration
        will not run until the dependency has completed, and the constants produced
        will be used via the database chain.
 
        You can define multiple dependencies for a single calibration simply
        by calling this multiple times. Be careful when adding the calibration into
        the `CAF` not to add a circular/cyclic dependency. If you do the sort will return an
        empty order and the `CAF` processing  will fail.
 
        This function appends to the `CalibrationBase.dependencies` and `CalibrationBase.future_dependencies` attributes of this
        `CalibrationBase` and the input one respectively. This prevents us having to do too much recalculation later on.
        """
        # Check that we don't have two calibration names that are the same
        if self.name != calibration.name:
            # Tests if we have the calibrations added as dependencies already and adds if not
            if calibration not in self.dependencies:
                self.dependencies.append(calibration)
            if self not in calibration.dependencies:
                calibration.future_dependencies.append(self)
        else:
            B2WARNING(f"Tried to add {calibration} as a dependency for {self} but they have the same name."
                      "Dependency was not added.")
 

◆ failed_dependencies()

failed_dependencies ( self )

inherited

Returns the list of calibrations in our dependency list that have failed.

Definition at line 421 of file framework.py.

    def failed_dependencies(self):
        """
        Returns the list of calibrations in our dependency list that have failed.
        """
        failed = []
        for calibration in self.dependencies:
            if calibration.state == self.fail_state:
                failed.append(calibration)
        return failed
 

◆ files_to_iovs() [1/2]

files_to_iovs ( self )

Reimplemented from CalibrationBase.

Definition at line 828 of file framework.py.

    def files_to_iovs(self):
        """
        """
        return self._get_default_collection_attribute("files_to_iovs")
 

◆ files_to_iovs() [2/2]

files_to_iovs	(		self,
			file_map )

Reimplemented from CalibrationBase.

Definition at line 834 of file framework.py.

    def files_to_iovs(self, file_map):
        """
        """
        self._set_default_collection_attribute("files_to_iovs", file_map)
 

◆ input_files() [1/2]

input_files ( self )

Reimplemented from CalibrationBase.

Definition at line 816 of file framework.py.

    def input_files(self):
        """
        """
        return self._get_default_collection_attribute("input_files")
 

◆ input_files() [2/2]

input_files	(		self,
			files )

Reimplemented from CalibrationBase.

Definition at line 822 of file framework.py.

    def input_files(self, files):
        """
        """
        self._set_default_collection_attribute("input_files", files)
 

◆ is_valid()

is_valid ( self )

A full calibration consists of a collector AND an associated algorithm AND input_files.

Returns False if:
    1) We are missing any of the above.
    2) There are multiple Collections and the Collectors have mis-matched granularities.
    3) Any of our Collectors have granularities that don't match what our Strategy can use.

Reimplemented from CalibrationBase.

Definition at line 635 of file framework.py.

    def is_valid(self):
        """
        A full calibration consists of a collector AND an associated algorithm AND input_files.
 
        Returns False if:
            1) We are missing any of the above.
            2) There are multiple Collections and the Collectors have mis-matched granularities.
            3) Any of our Collectors have granularities that don't match what our Strategy can use.
        """
        if not self.algorithms:
            B2WARNING(f"Empty algorithm list for {self.name}.")
            return False
 
        if not any([collection.is_valid() for collection in self.collections.values()]):
            B2WARNING(f"No valid Collections for {self.name}.")
            return False
 
        granularities = []
        for collection in self.collections.values():
            if collection.is_valid():
                collector_params = collection.collector.available_params()
                for param in collector_params:
                    if param.name == "granularity":
                        granularities.append(param.values)
        if len(set(granularities)) > 1:
            B2WARNING("Multiple different granularities set for the Collections in this Calibration.")
            return False
 
        for alg in self.algorithms:
            alg_type = type(alg.algorithm).__name__
            incorrect_gran = [granularity not in alg.strategy.allowed_granularities for granularity in granularities]
            if any(incorrect_gran):
                B2WARNING(f"Selected strategy for {alg_type} does not match a collector's granularity.")
                return False
        return True
 

◆ iteration() [1/2]

iteration ( self )

Retrieves the current iteration number in the database file.

Returns:
    int: The current iteration number

Definition at line 1097 of file framework.py.

    def iteration(self):
        """
        Retrieves the current iteration number in the database file.
 
        Returns:
            int: The current iteration number
        """
        with CAFDB(self._db_path, read_only=True) as db:
            iteration = db.get_calibration_value(self.name, "iteration")
        return iteration
 

◆ iteration() [2/2]

iteration	(		self,
			iteration )

Definition at line 1109 of file framework.py.

    def iteration(self, iteration):
        """
        """
        B2DEBUG(29, f"Setting {self.name} to {iteration}.")
        with CAFDB(self._db_path) as db:
            db.update_calibration_value(self.name, "iteration", iteration)
        B2DEBUG(29, f"{self.name} set to {self.iteration}.")
 
 

◆ max_collector_jobs() [1/2]

max_collector_jobs ( self )

Definition at line 876 of file framework.py.

    def max_collector_jobs(self):
        """
        """
        return self._get_default_collection_attribute("max_collector_jobs")
 

◆ max_collector_jobs() [2/2]

max_collector_jobs	(		self,
			max_jobs )

Definition at line 882 of file framework.py.

    def max_collector_jobs(self, max_jobs):
        """
        """
        self._set_default_collection_attribute("max_collector_jobs", max_jobs)
 

◆ max_files_per_collector_job() [1/2]

max_files_per_collector_job ( self )

Definition at line 864 of file framework.py.

    def max_files_per_collector_job(self):
        """
        """
        return self._get_default_collection_attribute("max_files_per_collector_job")
 

◆ max_files_per_collector_job() [2/2]

max_files_per_collector_job	(		self,
			max_files )

Definition at line 870 of file framework.py.

    def max_files_per_collector_job(self, max_files):
        """
        """
        self._set_default_collection_attribute("max_files_per_collector_job", max_files)
 

◆ output_patterns() [1/2]

output_patterns ( self )

Definition at line 852 of file framework.py.

    def output_patterns(self):
        """
        """
        return self._get_default_collection_attribute("output_patterns")
 

◆ output_patterns() [2/2]

output_patterns	(		self,
			patterns )

Definition at line 858 of file framework.py.

    def output_patterns(self, patterns):
        """
        """
        self._set_default_collection_attribute("output_patterns", patterns)
 

◆ pre_algorithms() [1/2]

pre_algorithms ( self )

Callback run prior to each algorithm iteration.

Definition at line 936 of file framework.py.

    def pre_algorithms(self):
        """
        Callback run prior to each algorithm iteration.
        """
        return [alg.pre_algorithm for alg in self.algorithms]
 

◆ pre_algorithms() [2/2]

pre_algorithms	(		self,
			func )

Definition at line 944 of file framework.py.

    def pre_algorithms(self, func):
        """
        """
        if func:
            for alg in self.algorithms:
                alg.pre_algorithm = func
        else:
            B2ERROR("Something evaluated as False passed in as pre_algorithm function.")
 

◆ pre_collector_path() [1/2]

pre_collector_path ( self )

Definition at line 840 of file framework.py.

    def pre_collector_path(self):
        """
        """
        return self._get_default_collection_attribute("pre_collector_path")
 

◆ pre_collector_path() [2/2]

pre_collector_path	(		self,
			path )

Definition at line 846 of file framework.py.

    def pre_collector_path(self, path):
        """
        """
        self._set_default_collection_attribute("pre_collector_path", path)
 

◆ reset_database()

reset_database	(		self,
			apply_to_default_collection = True )

Keyword Arguments:
    apply_to_default_collection (bool): Should we also reset the default collection?

Remove everything in the database_chain of this Calibration, including the default central database tag automatically
included from `basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`. This will NOT affect the
database chain of any `Collection` other than the default one. You can prevent the default Collection from having its chain
reset by setting 'apply_to_default_collection' to False.

Definition at line 671 of file framework.py.

    def reset_database(self, apply_to_default_collection=True):
        """
        Keyword Arguments:
            apply_to_default_collection (bool): Should we also reset the default collection?
 
        Remove everything in the database_chain of this Calibration, including the default central database tag automatically
        included from `basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`. This will NOT affect the
        database chain of any `Collection` other than the default one. You can prevent the default Collection from having its chain
        reset by setting 'apply_to_default_collection' to False.
        """
        self.database_chain = []
        if self.default_collection_name in self.collections and apply_to_default_collection:
            self.collections[self.default_collection_name].reset_database()
 

◆ run()

run ( self )

Main logic of the Calibration object.
Will be run in a new Thread by calling the start() method.

Reimplemented from CalibrationBase.

Definition at line 1006 of file framework.py.

    def run(self):
        """
        Main logic of the Calibration object.
        Will be run in a new Thread by calling the start() method.
        """
        with CAFDB(self._db_path, read_only=True) as db:
            initial_state = db.get_calibration_value(self.name, "checkpoint")
            initial_iteration = db.get_calibration_value(self.name, "iteration")
        B2INFO(f"Initial status of {self.name} found to be state={initial_state}, iteration={initial_iteration}")
        self.machine = CalibrationMachine(self,
                                          iov_to_calibrate=self.iov,
                                          initial_state=initial_state,
                                          iteration=initial_iteration)
        
        self.state = initial_state
        self.machine.root_dir = Path(os.getcwd(), self.name)
        self.machine.collector_backend = self.backend
 
        # Before we start running, let's clean up any iteration directories from iterations above our initial one.
        # Should prevent confusion between attempts if we fail again.
        all_iteration_paths = find_int_dirs(self.machine.root_dir)
        for iteration_path in all_iteration_paths:
            if int(iteration_path.name) > initial_iteration:
                shutil.rmtree(iteration_path)
 
        while self.state != self.end_state and self.state != self.fail_state:
            if self.state == "init":
                try:
                    B2INFO(f"Attempting collector submission for calibration {self.name}.")
                    self.machine.submit_collector()
                except Exception as err:
                    B2FATAL(str(err))
 
                self._poll_collector()
 
            # If we failed take us to the final fail state
            if self.state == "collector_failed":
                self.machine.fail_fully()
                return
 
            # It's possible that we might raise an error while attempting to run due
            # to some problems e.g. Missing collector output files
            # We catch the error and exit with failed state so the CAF will stop
            try:
                B2INFO(f"Attempting to run algorithms for calibration {self.name}.")
                self.machine.run_algorithms()
            except MachineError as err:
                B2ERROR(str(err))
                self.machine.fail()
 
            # If we failed take us to the final fail state
            if self.machine.state == "algorithms_failed":
                self.machine.fail_fully()
                return
 

◆ state() [1/2]

state ( self )

The current major state of the calibration in the database file. The machine may have a different state.

Definition at line 1077 of file framework.py.

    def state(self):
        """
        The current major state of the calibration in the database file. The machine may have a different state.
        """
        with CAFDB(self._db_path, read_only=True) as db:
            state = db.get_calibration_value(self.name, "state")
        return state
 

◆ state() [2/2]

state	(		self,
			state )

Definition at line 1086 of file framework.py.

    def state(self, state):
        """
        """
        B2DEBUG(29, f"Setting {self.name} to state {state}.")
        with CAFDB(self._db_path) as db:
            db.update_calibration_value(self.name, "state", str(state))
            if state in self.checkpoint_states:
                db.update_calibration_value(self.name, "checkpoint", str(state))
        B2DEBUG(29, f"{self.name} set to {state}.")
 

◆ strategies() [1/2]

strategies ( self )

The `caf.strategies.AlgorithmStrategy` or `list` of them used when running the algorithm(s).

Definition at line 969 of file framework.py.

    def strategies(self):
        """
        The `caf.strategies.AlgorithmStrategy` or `list` of them used when running the algorithm(s).
        """
        return [alg.strategy for alg in self.algorithms]
 

◆ strategies() [2/2]

strategies	(		self,
			strategy )

Definition at line 977 of file framework.py.

    def strategies(self, strategy):
        """
        """
        if strategy:
            for alg in self.algorithms:
                alg.strategy = strategy
        else:
            B2ERROR("Something evaluated as False passed in as a strategy.")
 

◆ use_central_database()

use_central_database	(	self,
		global_tag,
		apply_to_default_collection = True )

Parameters:
    global_tag (str): The central database global tag to use for this calibration.

Keyword Arguments:
    apply_to_default_collection (bool): Should we also call use_central_database on the default collection (if it exists)

Using this allows you to append a central database to the database chain for this calibration.
The default database chain is just the central one from
`basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`.
To turn off central database completely or use a custom tag as the base, you should call `Calibration.reset_database`
and start adding databases with `Calibration.use_local_database` and `Calibration.use_central_database`.

Note that the database chain attached to the `Calibration` will only affect the default `Collection` (if it exists),
and the algorithm processes. So calling:

>> cal.use_central_database("global_tag")

will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
assigned to

>> cal.collections['default'].database_chain

But calling

>> cal.use_central_database(file_path, payload_dir, False)

will add the database to the Algorithm processes, but leave the default Collection database chain untouched.
So if you have multiple Collections in this Calibration *their database chains are separate*.
To specify an additional `CentralDatabase` for a different collection, you will have to call:

>> cal.collections['OtherCollection'].use_central_database("global_tag")

Definition at line 685 of file framework.py.

    def use_central_database(self, global_tag, apply_to_default_collection=True):
        """
        Parameters:
            global_tag (str): The central database global tag to use for this calibration.
 
        Keyword Arguments:
            apply_to_default_collection (bool): Should we also call use_central_database on the default collection (if it exists)
 
        Using this allows you to append a central database to the database chain for this calibration.
        The default database chain is just the central one from
        `basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`.
        To turn off central database completely or use a custom tag as the base, you should call `Calibration.reset_database`
        and start adding databases with `Calibration.use_local_database` and `Calibration.use_central_database`.
 
        Note that the database chain attached to the `Calibration` will only affect the default `Collection` (if it exists),
        and the algorithm processes. So calling:
 
        >> cal.use_central_database("global_tag")
 
        will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
        assigned to
 
        >> cal.collections['default'].database_chain
 
        But calling
 
        >> cal.use_central_database(file_path, payload_dir, False)
 
        will add the database to the Algorithm processes, but leave the default Collection database chain untouched.
        So if you have multiple Collections in this Calibration *their database chains are separate*.
        To specify an additional `CentralDatabase` for a different collection, you will have to call:
 
        >> cal.collections['OtherCollection'].use_central_database("global_tag")
        """
        central_db = CentralDatabase(global_tag)
        self.database_chain.append(central_db)
        if self.default_collection_name in self.collections and apply_to_default_collection:
            self.collections[self.default_collection_name].use_central_database(global_tag)
 

◆ use_local_database()

use_local_database	(	self,
		filename,
		directory = "",
		apply_to_default_collection = True )

Parameters:
    filename (str): The path to the database.txt of the local database

Keyword Argumemts:
    directory (str): The path to the payloads directory for this local database.
    apply_to_default_collection (bool): Should we also call use_local_database on the default collection (if it exists)

Append a local database to the chain for this calibration.
You can call this function multiple times and each database will be added to the chain IN ORDER.
The databases are applied to this calibration ONLY.
The Local and Central databases applied via these functions are applied to the algorithm processes and optionally
the default `Collection` job as a database chain.
There are other databases applied to the processes later, checked by basf2 in this order:

1) Local Database from previous iteration of this Calibration.
2) Local Database chain from output of previous dependent Calibrations.
3) This chain of Local and Central databases where the last added is checked first.

Note that this function on the `Calibration` object will only affect the default `Collection` if it exists and if
'apply_to_default_collection' remains True. So calling:

>> cal.use_local_database(file_path, payload_dir)

will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
assigned to

>> cal.collections['default'].database_chain

But calling

>> cal.use_local_database(file_path, payload_dir, False)

will add the database to the Algorithm processes, but leave the default Collection database chain untouched.

If you have multiple Collections in this Calibration *their database chains are separate*.
To specify an additional `LocalDatabase` for a different collection, you will have to call:

>> cal.collections['OtherCollection'].use_local_database(file_path, payload_dir)

Definition at line 724 of file framework.py.

    def use_local_database(self, filename, directory="", apply_to_default_collection=True):
        """
        Parameters:
            filename (str): The path to the database.txt of the local database
 
        Keyword Argumemts:
            directory (str): The path to the payloads directory for this local database.
            apply_to_default_collection (bool): Should we also call use_local_database on the default collection (if it exists)
 
        Append a local database to the chain for this calibration.
        You can call this function multiple times and each database will be added to the chain IN ORDER.
        The databases are applied to this calibration ONLY.
        The Local and Central databases applied via these functions are applied to the algorithm processes and optionally
        the default `Collection` job as a database chain.
        There are other databases applied to the processes later, checked by basf2 in this order:
 
        1) Local Database from previous iteration of this Calibration.
        2) Local Database chain from output of previous dependent Calibrations.
        3) This chain of Local and Central databases where the last added is checked first.
 
        Note that this function on the `Calibration` object will only affect the default `Collection` if it exists and if
        'apply_to_default_collection' remains True. So calling:
 
        >> cal.use_local_database(file_path, payload_dir)
 
        will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
        assigned to
 
        >> cal.collections['default'].database_chain
 
        But calling
 
        >> cal.use_local_database(file_path, payload_dir, False)
 
        will add the database to the Algorithm processes, but leave the default Collection database chain untouched.
 
        If you have multiple Collections in this Calibration *their database chains are separate*.
        To specify an additional `LocalDatabase` for a different collection, you will have to call:
 
        >> cal.collections['OtherCollection'].use_local_database(file_path, payload_dir)
 
        """
        local_db = LocalDatabase(filename, directory)
        self.database_chain.append(local_db)
        if self.default_collection_name in self.collections and apply_to_default_collection:
            self.collections[self.default_collection_name].use_local_database(filename, directory)
 

Member Data Documentation

◆ _algorithms

list _algorithms = []

protected

Internal calibration algorithms stored for this calibration.

Definition at line 552 of file framework.py.

◆ _db_path

_db_path = None

protected

Location of a SQLite database that will save the state of the calibration so that it can be restarted from failure.

Definition at line 614 of file framework.py.

◆ alg_output_dir

str alg_output_dir = "algorithm_output"

static

Subdirectory name for algorithm output.

Definition at line 529 of file framework.py.

◆ algorithms

algorithms = algorithms

Algorithm classes that will be run by this Calibration.

You should set this attribute to either a single CalibrationAlgorithm C++ class, or a list of them if you want to run multiple CalibrationAlgorithms using one CalibrationCollectorModule.

Definition at line 572 of file framework.py.

◆ algorithms_runner

algorithms_runner = runners.SeqAlgorithmsRunner

The class that runs all the algorithms in this Calibration using their assigned :py:class:caf.strategies.AlgorithmStrategy.

Plugin your own runner class to change how your calibration will run the list of algorithms.

Definition at line 600 of file framework.py.

◆ backend

backend = None

The backend <backends.Backend> we'll use for our collector submission in this calibration.

If None it will be set by the CAF used to run this Calibration (recommended!).

Definition at line 603 of file framework.py.

◆ checkpoint_states

list checkpoint_states = ["init", "collector_completed", "completed"]

static

Checkpoint states which we are allowed to restart from.

Definition at line 531 of file framework.py.

◆ collections

dict collections = {}

Collections stored for this calibration.

Definition at line 550 of file framework.py.

◆ collector_full_update_interval

int collector_full_update_interval = 30

While checking if the collector is finished we don't bother wastefully checking every subjob's status.

We exit once we find the first subjob that isn't ready. But after this interval has elapsed we do a full :py:meth:caf.backends.Job.update_status call and print the fraction of SubJobs completed.

Definition at line 608 of file framework.py.

◆ database_chain

list database_chain = database_chain

The database chain that is applied to the algorithms.

This is often updated at the same time as the database chain for the default Collection.

Definition at line 591 of file framework.py.

◆ default_collection_name

default_collection_name = "default"

static

Default collection name.

Definition at line 533 of file framework.py.

◆ dependencies

list dependencies = []

inherited

List of calibration objects, where each one is a dependency of this one.

Definition at line 349 of file framework.py.

◆ end_state

str end_state = "completed"

staticinherited

The name of the successful completion state.

The :py:class:CAF will use this as the state to decide when the Calibration is completed.

Definition at line 335 of file framework.py.

◆ fail_state

str fail_state = "failed"

staticinherited

The name of the failure state.

The :py:class:CAF will use this as the state to decide when the Calibration failed.

Definition at line 338 of file framework.py.

◆ future_dependencies

list future_dependencies = []

inherited

List of calibration objects that depend on this one.

Definition at line 347 of file framework.py.

◆ heartbeat

heartbeat = 3

This calibration's sleep time before rechecking to see if it can move state.

Definition at line 610 of file framework.py.

◆ ignored_runs

ignored_runs = None

List of ExpRun that will be ignored by this Calibration.

This runs will not have Collector jobs run on them (if possible). And the algorithm execution will exclude them from a ExpRun list. However, the algorithm execution may merge IoVs of final payloads to cover the 'gaps' caused by these runs. You should pay attention to what the AlgorithmStrategy you choose will do in these cases.

Definition at line 582 of file framework.py.

◆ iov

iov = None

inherited

IoV which will be calibrated.

This is set by the CAF itself when calling CAF.run()

Definition at line 364 of file framework.py.

◆ jobs_to_submit

list jobs_to_submit = []

inherited

A simple list of jobs that this Calibration wants submitted at some point.

Definition at line 371 of file framework.py.

◆ machine

machine = None

The caf.state_machines.CalibrationMachine that we will run to process this calibration start to finish.

Definition at line 612 of file framework.py.

◆ max_iterations

max_iterations = None

Variable to define the maximum number of iterations for this calibration specifically.

It overrides the CAF calibration_defaults value if set.

Definition at line 577 of file framework.py.

◆ moves

list moves = ["submit_collector", "complete", "run_algorithms", "iterate", "fail_fully"]

static

Allowed transitions that we will use to progress.

Definition at line 527 of file framework.py.

◆ name

name = name

inherited

Name of calibration object.

This must be unique when adding into the py:class:CAF.

Definition at line 345 of file framework.py.

◆ output_database_dir

str output_database_dir = ""

inherited

The directory where we'll store the local database payloads from this calibration.

Definition at line 366 of file framework.py.

◆ results

dict results = {}

Output results of algorithms for each iteration.

Definition at line 574 of file framework.py.

◆ save_payloads

bool save_payloads = True

inherited

Marks this Calibration as one which has payloads that should be copied and uploaded.

Defaults to True, and should only be False if this is an intermediate Calibration who's payloads are never needed.

Definition at line 369 of file framework.py.

◆ state

str state = initial_state

state

Definition at line 1020 of file framework.py.

◆ strategies

strategies = strategies.SingleIOV

The strategy that the algorithm(s) will be run against.

Assign a list of strategies the same length as the number of algorithms, or assign a single strategy to apply it to all algorithms in this Calibration. You can see the choices in :py:mod:caf.strategies.

Definition at line 587 of file framework.py.

The documentation for this class was generated from the following file:

calibration/scripts/caf/framework.py

Public Member Functions

Public Attributes

Static Public Attributes

Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ _() [1/3]

◆ _() [2/3]

◆ _() [3/3]

◆ __repr__()

◆ _apply_calibration_defaults()

◆ _get_default_collection_attribute()

◆ _poll_collector()

◆ _set_default_collection_attribute()

◆ add_collection()

◆ algorithms() [1/2]

◆ algorithms() [2/2]

◆ backend_args() [1/2]

◆ backend_args() [2/2]

◆ collector() [1/2]

◆ collector() [2/2]

◆ dependencies_met()

◆ depends_on()

◆ failed_dependencies()

◆ files_to_iovs() [1/2]

◆ files_to_iovs() [2/2]

◆ input_files() [1/2]

◆ input_files() [2/2]

◆ is_valid()

◆ iteration() [1/2]

◆ iteration() [2/2]

◆ max_collector_jobs() [1/2]

◆ max_collector_jobs() [2/2]

◆ max_files_per_collector_job() [1/2]

◆ max_files_per_collector_job() [2/2]

◆ output_patterns() [1/2]

◆ output_patterns() [2/2]

◆ pre_algorithms() [1/2]

◆ pre_algorithms() [2/2]

◆ pre_collector_path() [1/2]

◆ pre_collector_path() [2/2]

◆ reset_database()

◆ run()

◆ state() [1/2]

◆ state() [2/2]

◆ strategies() [1/2]

◆ strategies() [2/2]

◆ use_central_database()

◆ use_local_database()

Member Data Documentation

◆ _algorithms

◆ _db_path

◆ alg_output_dir

◆ algorithms

◆ algorithms_runner

◆ backend

◆ checkpoint_states

◆ collections

◆ collector_full_update_interval

◆ database_chain

◆ default_collection_name

◆ dependencies

◆ end_state

◆ fail_state

◆ future_dependencies

◆ heartbeat

◆ ignored_runs

◆ iov

◆ jobs_to_submit

◆ machine

◆ max_iterations

◆ moves

◆ name

◆ output_database_dir

◆ results

◆ save_payloads

◆ state

◆ init()

◆ repr()