Calibration Class Reference

Inheritance diagram for Calibration:

Collaboration diagram for Calibration:

Public Member Functions
def	__init__ (self, name, collector=None, algorithms=None, input_files=None, pre_collector_path=None, database_chain=None, output_patterns=None, max_files_per_collector_job=None, max_collector_jobs=None, backend_args=None)
def	add_collection (self, name, collection)
def	is_valid (self)
def	reset_database (self, apply_to_default_collection=True)
def	use_central_database (self, global_tag, apply_to_default_collection=True)
def	use_local_database (self, filename, directory="", apply_to_default_collection=True)
def	collector (self)
def	collector (self, collector)
def	input_files (self)
def	input_files (self, files)
def	files_to_iovs (self)
def	files_to_iovs (self, file_map)
def	pre_collector_path (self)
def	pre_collector_path (self, path)
def	output_patterns (self)
def	output_patterns (self, patterns)
def	max_files_per_collector_job (self)
def	max_files_per_collector_job (self, max_files)
def	max_collector_jobs (self)
def	max_collector_jobs (self, max_jobs)
def	backend_args (self)
def	backend_args (self, args)
def	algorithms (self)
def	algorithms (self, value)
def	pre_algorithms (self)
def	pre_algorithms (self, func)
def	strategies (self)
def	strategies (self, strategy)
def	__repr__ (self)
def	run (self)
def	state (self)
def	state (self, state)
def	iteration (self)
def	iteration (self, iteration)
def	depends_on (self, calibration)
def	dependencies_met (self)
def	failed_dependencies (self)

Public Attributes
	collections
	Collections stored for this calibration.
	algorithms
	Algorithm classes that wil be run by this Calibration. More...
	results
	Output results of algorithms for each iteration.
	max_iterations
	Variable to define the maximum number of iterations for this calibration specifically. More...
	ignored_runs
	List of ExpRun that will be ignored by this Calibration. More...
	strategies
	The strategy that the algorithm(s) will be run against. More...
	database_chain
	The database chain that is applied to the algorithms. More...
	algorithms_runner
	The class that runs all the algorithms in this Calibration using their assigned :py:class:caf.strategies.AlgorithmStrategy. More...
	backend
	The backend <backends.Backend> we'll use for our collector submission in this calibration. More...
	collector_full_update_interval
	While checking if the collector is finished we don't bother wastefully checking every subjob's status. More...
	heartbeat
	This calibration's sleep time before rechecking to see if it can move state.
	machine
	The caf.state_machines.CalibrationMachine that we will run to process this calibration start to finish.
	state
	name
	Name of calibration object. More...
	future_dependencies
	List of calibration objects that depend on this one.
	dependencies
	List of calibration objects, where each one is a dependency of this one.
	files_to_iovs
	File -> Iov dictionary, should be : {absolute_file_path:iov} : Where iov is a :py:class:IoV <caf.utils.IoV> object. More...
	input_files
	Files used for collection procedure.
	iov
	IoV which will be calibrated. More...
	output_database_dir
	The directory where we'll store the local database payloads from this calibration.
	save_payloads
	Marks this Calibration as one which has payloads that should be copied and uploaded. More...
	jobs_to_submit
	A simple list of jobs that this Calibration wants submitted at some point.

Static Public Attributes
list	moves = ["submit_collector", "complete", "run_algorithms", "iterate", "fail_fully"]
	Allowed transitions that we will use to progress.
string	alg_output_dir = "algorithm_output"
	Subdirectory name for algorithm output.
list	checkpoint_states = ["init", "collector_completed", "completed"]
	Checkpoint states which we are allowed to restart from.
string	default_collection_name = "default"
	Default collection name.
string	end_state = "completed"
	The name of the successful completion state. More...
string	fail_state = "failed"
	The name of the failure state. More...

Private Member Functions
def	_get_default_collection_attribute (self, attr)
def	_set_default_collection_attribute (self, attr, value)
def	_ (self, value)
def	_ (self, values)
def	_ (self, values)
def	_poll_collector (self)
def	_apply_calibration_defaults (self, defaults)

Private Attributes
	_algorithms
	Internal calibration algorithms stored for this calibration.
	_db_path
	Location of a SQLite database that will save the state of the calibration so that it can be restarted from failure.

Detailed Description

Every Calibration object must have at least one collector at least one algorithm.
You have the option to add in your collector/algorithm by argument here, or add them
later by changing the properties.

If you plan to use multiple `Collection` objects I recommend that you only set the name here and add the Collections
separately via `add_collection()`.

Parameters:
    name (str): Name of this calibration. It should be unique for use in the `CAF`
Keyword Arguments:
    collector (str, `basf2.Module`): Should be set to a CalibrationCollectorModule() or a string with the module name.
    algorithms (list, ``ROOT.Belle2.CalibrationAlgorithm``): The algorithm(s) to use for this `Calibration`.
    input_files (str, list[str]): Input files for use by this Calibration. May contain wildcards useable by `glob.glob`

A Calibration won't be valid in the `CAF` until it has all of these four attributes set. For example:

>>> cal = Calibration('TestCalibration1')
>>> col1 = register_module('CaTest')
>>> cal.add_collection('TestColl', col1)

or equivalently

>>> cal = Calibration('TestCalibration1', 'CaTest')

If you want to run a basf2 :py:class:`path <basf2.Path>` before your collector module when running over data

>>> cal.pre_collector_path = my_basf2_path

You don't have to put a RootInput module in this pre-collection path, but you can if
you need some special parameters. If you want to process sroot files the you have to explicitly add
SeqRootInput to your pre-collection path.
The inputFileNames parameter of (Seq)RootInput will be set by the CAF automatically for you.


You can use optional arguments to pass in some/all during initialisation of the `Calibration` class

>>> cal = Calibration( 'TestCalibration1', 'CaTest', [alg1,alg2], ['/path/to/file.root'])

you can change the input file list later on, before running with `CAF`

>>> cal.input_files = ['path/to/*.root', 'other/path/to/file2.root']

If you have multiple collections from calling `add_collection()` then you should instead set the pre_collector_path,
input_files, database chain etc from there. See `Collection`.

Adding the CalibrationAlgorithm(s) is easy

>>> alg1 = TestAlgo()
>>> cal.algorithms = alg1

Or equivalently

>>> cal.algorithms = [alg1]

Or for multiple algorithms for one collector

>>> alg2 = TestAlgo()
>>> cal.algorithms = [alg1, alg2]

Note that when you set the algorithms, they are automatically wrapped and stored as a Python class
`Algorithm`. To access the C++ algorithm clas underneath directly do:

>>> cal.algorithms[i].algorithm

If you have a setup function that you want to run before each of the algorithms, set that with

>>> cal.pre_algorithms = my_function_object

If you want a different setup for each algorithm use a list with the same number of elements
as your algorithm list.

>>> cal.pre_algorithms = [my_function1, my_function2, ...]

You can also specify the dependencies of the calibration on others

>>> cal.depends_on(cal2)

By doing this, the `CAF` will respect the ordering of the calibrations and will pass the
calibration constants created by earlier completed calibrations to dependent ones.

Definition at line 425 of file framework.py.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
			name,
			collector = None,
			algorithms = None,
			input_files = None,
			pre_collector_path = None,
			database_chain = None,
			output_patterns = None,
			max_files_per_collector_job = None,
			max_collector_jobs = None,
			backend_args = None
	)

Definition at line 516 of file framework.py.

Member Function Documentation

_() [1/3]

def _ (   self,   value  )

private

Alternate algorithms setter for lists and tuples of CalibrationAlgorithms.

Definition at line 896 of file framework.py.

_() [2/3]

def _ (   self,   values  )

private

Alternate pre_algorithms setter for lists and tuples of functions, should be one per algorithm.

Definition at line 930 of file framework.py.

_() [3/3]

def _ (   self,   values  )

private

Alternate strategies setter for lists and tuples of functions, should be one per algorithm.

Definition at line 963 of file framework.py.

__repr__()

def __repr__

(

self

)

Definition at line 976 of file framework.py.

_apply_calibration_defaults()

def _apply_calibration_defaults (   self,   defaults  )

privateinherited

We pass in default calibration options from the `CAF` instance here if called.
Won't overwrite any options already set.

Definition at line 412 of file framework.py.

_poll_collector()

def _poll_collector (   self )

private

Definition at line 1037 of file framework.py.

add_collection()

def add_collection	(		self,
			name,
			collection
	)

Parameters:
    name (str): Unique name of this `Collection` in the Calibration.
    collection (`Collection`): `Collection` object to use.

Adds a new `Collection` object to the `Calibration`. Any valid Collection will be used in the Calibration.
A default Collection is automatically added but isn't valid and won't run unless you have assigned a collector
+ input files.
You can ignore the default one and only add your own custom Collections. You can configure the default from the
Calibration(...) arguments or after creating the Calibration object via directly setting the cal.collector, cal.input_files
attributes.

Definition at line 597 of file framework.py.

algorithms() [1/2]

def algorithms

(

self

)

Definition at line 877 of file framework.py.

algorithms() [2/2]

def algorithms	(		self,
			value
	)

Definition at line 884 of file framework.py.

backend_args() [1/2]

def backend_args

(

self

)

Definition at line 865 of file framework.py.

backend_args() [2/2]

def backend_args	(		self,
			args
	)

Definition at line 871 of file framework.py.

collector() [1/2]

def collector

(

self

)

Definition at line 772 of file framework.py.

collector() [2/2]

def collector	(		self,
			collector
	)

Definition at line 778 of file framework.py.

dependencies_met()

def dependencies_met (   self )

inherited

Checks if all of the Calibrations that this one depends on have reached a successful end state.

Definition at line 396 of file framework.py.

depends_on()

def depends_on (   self,   calibration  )

inherited

Parameters:
    calibration (`CalibrationBase`): The Calibration object which will produce constants that this one depends on.

Adds dependency of this calibration on another i.e. This calibration
will not run until the dependency has completed, and the constants produced
will be used via the database chain.

You can define multiple dependencies for a single calibration simply
by calling this multiple times. Be careful when adding the calibration into
the `CAF` not to add a circular/cyclic dependency. If you do the sort will return an
empty order and the `CAF` processing  will fail.

This function appens to the `CalibrationBase.dependencies` and `CalibrationBase.future_dependencies` attributes of this
`CalibrationBase` and the input one respectively. This prevents us having to do too much recalculation later on.

Definition at line 368 of file framework.py.

failed_dependencies()

def failed_dependencies (   self )

inherited

Returns the list of calibrations in our dependency list that have failed.

Definition at line 402 of file framework.py.

files_to_iovs() [1/2]

def files_to_iovs

(

self

)

Definition at line 805 of file framework.py.

files_to_iovs() [2/2]

def files_to_iovs	(		self,
			file_map
	)

Definition at line 811 of file framework.py.

input_files() [1/2]

def input_files

(

self

)

Definition at line 793 of file framework.py.

input_files() [2/2]

def input_files	(		self,
			files
	)

Definition at line 799 of file framework.py.

is_valid()

def is_valid

(

self

)

A full calibration consists of a collector AND an associated algorithm AND input_files.

Returns False if:
    1) We are missing any of the above.
    2) There are multiple Collections and the Collectors have mis-matched granularities.
    3) Any of our Collectors have granularities that don't match what our Strategy can use.

Reimplemented from CalibrationBase.

Definition at line 616 of file framework.py.

iteration() [1/2]

def iteration

(

self

)

Retrieves the current iteration number in the database file.

Returns:
    int: The current iteration number

Definition at line 1073 of file framework.py.

iteration() [2/2]

def iteration	(		self,
			iteration
	)

Definition at line 1085 of file framework.py.

max_collector_jobs() [1/2]

def max_collector_jobs

(

self

)

Definition at line 853 of file framework.py.

max_collector_jobs() [2/2]

def max_collector_jobs	(		self,
			max_jobs
	)

Definition at line 859 of file framework.py.

max_files_per_collector_job() [1/2]

def max_files_per_collector_job

(

self

)

Definition at line 841 of file framework.py.

max_files_per_collector_job() [2/2]

def max_files_per_collector_job	(		self,
			max_files
	)

Definition at line 847 of file framework.py.

output_patterns() [1/2]

def output_patterns

(

self

)

Definition at line 829 of file framework.py.

output_patterns() [2/2]

def output_patterns	(		self,
			patterns
	)

Definition at line 835 of file framework.py.

pre_algorithms() [1/2]

def pre_algorithms

(

self

)

Callback run prior to each algorithm iteration.

Definition at line 911 of file framework.py.

pre_algorithms() [2/2]

def pre_algorithms	(		self,
			func
	)

Definition at line 919 of file framework.py.

pre_collector_path() [1/2]

def pre_collector_path

(

self

)

Definition at line 817 of file framework.py.

pre_collector_path() [2/2]

def pre_collector_path	(		self,
			path
	)

Definition at line 823 of file framework.py.

reset_database()

def reset_database	(		self,
			apply_to_default_collection = True
	)

Keyword Arguments:
    apply_to_default_collection (bool): Should we also reset the default collection?

Remove everything in the database_chain of this Calibration, including the default central database tag automatically
included from `basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`. This will NOT affect the
database chain of any `Collection` other than the default one. You can prevent the default Collection from having its chain
reset by setting 'apply_to_default_collection' to False.

Definition at line 652 of file framework.py.

run()

def run

(

self

)

Main logic of the Calibration object.
Will be run in a new Thread by calling the start() method.

Reimplemented from CalibrationBase.

Definition at line 981 of file framework.py.

state() [1/2]

def state

(

self

)

The current major state of the calibration in the database file. The machine may have a different state.

Definition at line 1053 of file framework.py.

state() [2/2]

def state	(		self,
			state
	)

Definition at line 1062 of file framework.py.

strategies() [1/2]

def strategies

(

self

)

The `caf.strategies.AlgorithmStrategy` or `list` of them used when running the algorithm(s).

Definition at line 944 of file framework.py.

strategies() [2/2]

def strategies	(		self,
			strategy
	)

Definition at line 952 of file framework.py.

use_central_database()

def use_central_database	(		self,
			global_tag,
			apply_to_default_collection = True
	)

Parameters:
    global_tag (str): The central database global tag to use for this calibration.

Keyword Arguments:
    apply_to_default_collection (bool): Should we also call use_central_database on the default collection (if it exists)

Using this allows you to append a central database to the database chain for this calibration.
The default database chain is just the central one from
`basf2.conditions.default_globaltags <ConditionsConfiguration.default_globaltags>`.
To turn off central database completely or use a custom tag as the base, you should call `Calibration.reset_database`
and start adding databases with `Calibration.use_local_database` and `Calibration.use_central_database`.

Note that the database chain attached to the `Calibration` will only affect the default `Collection` (if it exists),
and the algorithm processes. So calling:

>> cal.use_central_database("global_tag")

will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
assigned to

>> cal.collections['default'].database_chain

But calling

>> cal.use_central_database(file_path, payload_dir, False)

will add the database to the Algorithm processes, but leave the default Collection database chain untouched.
So if you have multiple Collections in this Calibration *their database chains are separate*.
To specify an additional `CentralDatabase` for a different collection, you will have to call:

>> cal.collections['OtherCollection'].use_central_database("global_tag")

Definition at line 666 of file framework.py.

use_local_database()

def use_local_database	(		self,
			filename,
			directory = "",
			apply_to_default_collection = True
	)

Parameters:
    filename (str): The path to the database.txt of the local database

Keyword Argumemts:
    directory (str): The path to the payloads directory for this local database.
    apply_to_default_collection (bool): Should we also call use_local_database on the default collection (if it exists)

Append a local database to the chain for this calibration.
You can call this function multiple times and each database will be added to the chain IN ORDER.
The databases are applied to this calibration ONLY.
The Local and Central databases applied via these functions are applied to the algorithm processes and optionally
the default `Collection` job as a database chain.
There are other databases applied to the processes later, checked by basf2 in this order:

1) Local Database from previous iteration of this Calibration.
2) Local Database chain from output of previous dependent Calibrations.
3) This chain of Local and Central databases where the last added is checked first.

Note that this function on the `Calibration` object will only affect the default `Collection` if it exists and if
'apply_to_default_collection' remains True. So calling:

>> cal.use_local_database(file_path, payload_dir)

will modify the database chain used by all the algorithms assigned to this `Calibration`, and modifies the database chain
assigned to

>> cal.collections['default'].database_chain

But calling

>> cal.use_local_database(file_path, payload_dir, False)

will add the database to the Algorithm processes, but leave the default Collection database chain untouched.

If you have multiple Collections in this Calibration *their database chains are separate*.
To specify an additional `LocalDatabase` for a different collection, you will have to call:

>> cal.collections['OtherCollection'].use_local_database(file_path, payload_dir)

Definition at line 705 of file framework.py.

Member Data Documentation

algorithms

algorithms

Algorithm classes that wil be run by this Calibration.

You should set this attribute to either a single CalibrationAlgorithm C++ class, or a list of them if you want to run multiple CalibrationAlgorithms using one CalibrationCollectorModule.

Definition at line 553 of file framework.py.

algorithms_runner

algorithms_runner

The class that runs all the algorithms in this Calibration using their assigned :py:class:caf.strategies.AlgorithmStrategy.

Plugin your own runner class to change how your calibration will run the list of algorithms.

Definition at line 581 of file framework.py.

backend

backend

The backend <backends.Backend> we'll use for our collector submission in this calibration.

If None it will be set by the CAF used to run this Calibration (recommended!).

Definition at line 584 of file framework.py.

collector_full_update_interval

collector_full_update_interval

While checking if the collector is finished we don't bother wastefully checking every subjob's status.

We exit once we find the first subjob that isn't ready. But after this interval has elapsed we do a full :py:meth:caf.backends.Job.update_status call and print the fraction of SubJobs completed.

Definition at line 589 of file framework.py.

database_chain

database_chain

The database chain that is applied to the algorithms.

This is often updated at the same time as the database chain for the default Collection.

Definition at line 572 of file framework.py.

end_state

string end_state = "completed"

staticinherited

The name of the successful completion state.

The :py:class:CAF will use this as the state to decide when the Calibration is completed.

Definition at line 316 of file framework.py.

fail_state

string fail_state = "failed"

staticinherited

The name of the failure state.

The :py:class:CAF will use this as the state to decide when the Calibration failed.

Definition at line 319 of file framework.py.

files_to_iovs

files_to_iovs

inherited

File -> Iov dictionary, should be :

{absolute_file_path:iov}

: Where iov is a :py:class:IoV <caf.utils.IoV> object.

Will be filled during CAF.run() if empty. To improve performance you can fill this yourself before calling CAF.run()

Definition at line 337 of file framework.py.

ignored_runs

ignored_runs

List of ExpRun that will be ignored by this Calibration.

This runs will not have Collector jobs run on them (if possible). And the algorithm execution will exclude them from a ExpRun list. However, the algorithm execution may merge IoVs of final payloads to cover the 'gaps' caused by these runs. You should pay attention to what the AlgorithmStrategy you choose will do in these cases.

Definition at line 563 of file framework.py.

iov

iov

inherited

IoV which will be calibrated.

This is set by the CAF itself when calling CAF.run()

Definition at line 345 of file framework.py.

max_iterations

max_iterations

Variable to define the maximum number of iterations for this calibration specifically.

It overrides tha CAF calibration_defaults value if set.

Definition at line 558 of file framework.py.

name

name

inherited

Name of calibration object.

This must be unique when adding into the py:class:CAF.

Definition at line 326 of file framework.py.

save_payloads

save_payloads

inherited

Marks this Calibration as one which has payloads that should be copied and uploaded.

Defaults to True, and should only be False if this is an intermediate Calibration who's payloads are never needed.

Definition at line 350 of file framework.py.

strategies

strategies

The strategy that the algorithm(s) will be run against.

Assign a list of strategies the same length as the number of algorithms, or assign a single strategy to apply it to all algorithms in this Calibration. You can see the choices in :py:mod:caf.strategies.

Definition at line 568 of file framework.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/framework.py