Inheritance diagram for SubJob:

Public Member Functions
	__init__ (self, job, subjob_id, input_files=None)

	output_dir (self)

	working_dir (self)

	name (self)

	status (self)

	status (self, status)

	subjobs (self)

	job_dict (self)

	__getattr__ (self, attribute)

	__repr__ (self)

	ready (self)

	update_status (self)

	create_subjob (self, i, input_files=None, args=None)

	dump_to_json (self, file_path)

	from_json (cls, file_path)

	dump_input_data (self)

	copy_input_sandbox_files_to_working_dir (self)

	check_input_data_files (self)

	full_command (self)

	append_current_basf2_setup_cmds (self)

Public Attributes
	id = subjob_id
	Id of Subjob.

	parent = job
	Job() instance of parent to this SubJob.

	splitter = None
	The `SubjobSplitter` used to create subjobs if necessary.

list	input_sandbox_files = []
	Files to be copied directly into the working directory (`pathlib.Path`).

list	output_patterns = []
	Files that we produce during the job and want to be returned.

list	cmd = []
	Command and arguments as a list that will be run by the job on the backend.

list	args = []
	The arguments that will be applied to the `cmd` (These are ignored by SubJobs as they have their own arguments)

list	input_files = []
	Input files to job (`str`), a list of these is copied to the working directory.

list	setup_cmds = []
	Bash commands to run before the main self.cmd (mainly used for batch system setup)

dict	backend_args = {}
	Config dictionary for the backend to use when submitting the job.

	result = None
	The result object of this Job.

Static Public Attributes
dict	statuses = {"init": 0, "submitted": 1, "running": 2, "failed": 3, "completed": 4}
	Allowed Job status dictionary.

list	exit_statuses = ["failed", "completed"]
	Job statuses that correspond to the Job being finished (successfully or not)

Protected Member Functions
	_get_overall_status_from_subjobs (self)

Protected Attributes
str	_status = "init"
	The actual status of the overall `Job`.

Detailed Description

This mini-class simply holds basic information about which subjob you are
and a reference to the parent Job object to be able to access the main data there.
Rather than replicating all of the parent job's configuration again.

Definition at line 678 of file backends.py.

Constructor & Destructor Documentation

◆ init()

__init__	(	self,
		job,
		subjob_id,
		input_files = None )

Definition at line 685 of file backends.py.

    def __init__(self, job, subjob_id, input_files=None):
        """
        """
        
        self.id = subjob_id
        
        self.parent = job
        
        if not input_files:
            input_files = []
        self.input_files = input_files
        
        self.result = None
        
        self._status = "init"
        
        self.args = []
 

Member Function Documentation

◆ getattr()

__getattr__	(		self,
			attribute )

Since a SubJob uses attributes from the parent Job, everything simply accesses the Job attributes
unless otherwise specified.

Definition at line 760 of file backends.py.

    def __getattr__(self, attribute):
        """
        Since a SubJob uses attributes from the parent Job, everything simply accesses the Job attributes
        unless otherwise specified.
        """
        return getattr(self.parent, attribute)
 

◆ repr()

__repr__ ( self )

Definition at line 767 of file backends.py.

    def __repr__(self):
        """
        """
        return f"SubJob({self.name})"
 
 

◆ _get_overall_status_from_subjobs()

_get_overall_status_from_subjobs ( self )

protectedinherited

Definition at line 463 of file backends.py.

    def _get_overall_status_from_subjobs(self):
        """
        """
        subjob_statuses = [subjob.status for subjob in self.subjobs.values()]
        status_level = min([self.statuses[status] for status in subjob_statuses])
        for status, level in self.statuses.items():
            if level == status_level:
                return status
 

◆ append_current_basf2_setup_cmds()

append_current_basf2_setup_cmds ( self )

inherited

This adds simple setup commands like ``source /path/to/tools/b2setup`` to your `Job`.
It should detect if you are using a local release or CVMFS and append the correct commands
so that the job will have the same basf2 release environment. It should also detect
if a local release is not compiled with the ``opt`` option.

Note that this *doesn't mean that every environment variable is inherited* from the submitting
process environment.

Definition at line 640 of file backends.py.

    def append_current_basf2_setup_cmds(self):
        """
        This adds simple setup commands like ``source /path/to/tools/b2setup`` to your `Job`.
        It should detect if you are using a local release or CVMFS and append the correct commands
        so that the job will have the same basf2 release environment. It should also detect
        if a local release is not compiled with the ``opt`` option.
 
        Note that this *doesn't mean that every environment variable is inherited* from the submitting
        process environment.
        """
        def append_environment_variable(cmds, envvar):
            """
            Append a command for setting an environment variable.
            """
            if envvar in os.environ:
                cmds.append(f"""if [ -z "${{{envvar}}}" ]; then""")
                cmds.append(f"  export {envvar}={os.environ[envvar]}")
                cmds.append("fi")
 
        if "BELLE2_TOOLS" not in os.environ:
            raise BackendError("No BELLE2_TOOLS found in environment")
        # Export all the environment variables defined via _backend_job_envvars
        for envvar in _backend_job_envvars:
            append_environment_variable(self.setup_cmds, envvar)
        if "BELLE2_RELEASE" in os.environ:
            self.setup_cmds.append(f"source {os.environ['BELLE2_TOOLS']}/b2setup {os.environ['BELLE2_RELEASE']}")
        elif 'BELLE2_LOCAL_DIR' in os.environ:
            self.setup_cmds.append("export BELLE2_NO_TOOLS_CHECK=\"TRUE\"")
            self.setup_cmds.append(f"BACKEND_B2SETUP={os.environ['BELLE2_TOOLS']}/b2setup")
            self.setup_cmds.append(f"BACKEND_BELLE2_RELEASE_LOC={os.environ['BELLE2_LOCAL_DIR']}")
            self.setup_cmds.append(f"BACKEND_BELLE2_OPTION={os.environ['BELLE2_OPTION']}")
            self.setup_cmds.append("pushd $BACKEND_BELLE2_RELEASE_LOC > /dev/null")
            self.setup_cmds.append("source $BACKEND_B2SETUP")
            # b2code-option has to be executed only after the source of the tools.
            self.setup_cmds.append("b2code-option $BACKEND_BELLE2_OPTION")
            self.setup_cmds.append("popd > /dev/null")
 
 

◆ check_input_data_files()

check_input_data_files ( self )

inherited

Check the input files and make sure that there aren't any duplicates.
Also check if the files actually exist if possible.

Definition at line 598 of file backends.py.

    def check_input_data_files(self):
        """
        Check the input files and make sure that there aren't any duplicates.
        Also check if the files actually exist if possible.
        """
        existing_input_files = []  # We use a list instead of set to avoid losing any ordering of files
        for file_path in self.input_files:
            file_uri = parse_file_uri(file_path)
            if file_uri.scheme == "file":
                p = Path(file_uri.path)
                if p.is_file():
                    if file_uri.geturl() not in existing_input_files:
                        existing_input_files.append(file_uri.geturl())
                    else:
                        B2WARNING(f"Requested input file path {file_path} was already added, skipping it.")
                else:
                    B2WARNING(f"Requested input file path {file_path} does not exist, skipping it.")
            else:
                B2DEBUG(29, f"{file_path} is not a local file URI. Skipping checking if file exists")
                if file_path not in existing_input_files:
                    existing_input_files.append(file_path)
                else:
                    B2WARNING(f"Requested input file path {file_path} was already added, skipping it.")
        if self.input_files and not existing_input_files:
            B2WARNING(f"No valid input file paths found for {self.name}, but some were requested.")
 
        # Replace the Job's input files with the ones that exist + duplicates removed
        self.input_files = existing_input_files
 

◆ copy_input_sandbox_files_to_working_dir()

copy_input_sandbox_files_to_working_dir ( self )

inherited

Get all of the requested files for the input sandbox and copy them to the working directory.
Files like the submit.sh or input_data.json are not part of this process.

Definition at line 587 of file backends.py.

    def copy_input_sandbox_files_to_working_dir(self):
        """
        Get all of the requested files for the input sandbox and copy them to the working directory.
        Files like the submit.sh or input_data.json are not part of this process.
        """
        for file_path in self.input_sandbox_files:
            if file_path.is_dir():
                shutil.copytree(file_path, Path(self.working_dir, file_path.name))
            else:
                shutil.copy(file_path, self.working_dir)
 

◆ create_subjob()

create_subjob	(	self,
		i,
		input_files = None,
		args = None )

inherited

Creates a subjob Job object that references that parent Job.
Returns the SubJob object at the end.

Definition at line 435 of file backends.py.

    def create_subjob(self, i, input_files=None, args=None):
        """
        Creates a subjob Job object that references that parent Job.
        Returns the SubJob object at the end.
        """
        if i not in self.subjobs:
            B2INFO(f"Creating {self}.Subjob({i})")
            subjob = SubJob(self, i, input_files)
            if args:
                subjob.args = args
            self.subjobs[i] = subjob
            return subjob
        else:
            B2WARNING(f"{self} already contains SubJob({i})! This will not be created.")
 

◆ dump_input_data()

dump_input_data ( self )

inherited

Dumps the `Job.input_files` attribute to a JSON file. input_files should be a list of
string URI objects.

Definition at line 579 of file backends.py.

    def dump_input_data(self):
        """
        Dumps the `Job.input_files` attribute to a JSON file. input_files should be a list of
        string URI objects.
        """
        with open(Path(self.working_dir, _input_data_file_path), mode="w") as input_data_file:
            json.dump(self.input_files, input_data_file, indent=2)
 

◆ dump_to_json()

dump_to_json	(		self,
			file_path )

inherited

Dumps the Job object configuration to a JSON file so that it can be read in again later.

Parameters:
  file_path(`basf2.Path`): The filepath we'll dump to

Definition at line 538 of file backends.py.

    def dump_to_json(self, file_path):
        """
        Dumps the Job object configuration to a JSON file so that it can be read in again later.
 
        Parameters:
          file_path(`basf2.Path`): The filepath we'll dump to
        """
        # \cond false positive doxygen warning about job_dict
        with open(file_path, mode="w") as job_file:
            json.dump(self.job_dict, job_file, indent=2)
        # \endcond
 

◆ from_json()

from_json	(		cls,
			file_path )

inherited

Definition at line 551 of file backends.py.

    def from_json(cls, file_path):
        """
        """
        with open(file_path) as job_file:
            job_dict = json.load(job_file)
        return cls(job_dict["name"], job_dict=job_dict)
 

◆ full_command()

full_command ( self )

inherited

Returns:
    str: The full command that this job will run including any arguments.

Definition at line 628 of file backends.py.

    def full_command(self):
        """
        Returns:
            str: The full command that this job will run including any arguments.
        """
        all_components = self.cmd[:]
        all_components.extend(self.args)
        # We do a convert to string just in case arguments were generated as different types
        full_command = " ".join(map(str, all_components))
        B2DEBUG(29, f"Full command of {self} is '{full_command}'")
        return full_command
 

◆ job_dict()

job_dict ( self )

Returns:
    dict: A JSON serialisable representation of the `SubJob`. `Path <basf2.Path>` objects are converted to
    `string` via ``Path.as_posix()``. Since Subjobs inherit most of the parent job's config
    we only output the input files and arguments that are specific to this subjob and no other details.

Reimplemented from Job.

Definition at line 747 of file backends.py.

    def job_dict(self):
        """
        Returns:
            dict: A JSON serialisable representation of the `SubJob`. `Path <basf2.Path>` objects are converted to
            `string` via ``Path.as_posix()``. Since Subjobs inherit most of the parent job's config
            we only output the input files and arguments that are specific to this subjob and no other details.
        """
        job_dict = {}
        job_dict["id"] = self.id
        job_dict["input_files"] = self.input_files
        job_dict["args"] = self.args
        return job_dict
 

◆ name()

name ( self )

Getter for name of SubJob. Accesses the parent Job name to infer this.

Reimplemented from Job.

Definition at line 716 of file backends.py.

    def name(self):
        """Getter for name of SubJob. Accesses the parent Job name to infer this."""
        return "_".join((self.parent.name, str(self.id)))
 

◆ output_dir()

output_dir ( self )

Getter for output_dir of SubJob. Accesses the parent Job output_dir to infer this.

Reimplemented from Job.

Definition at line 705 of file backends.py.

    def output_dir(self):
        """
        Getter for output_dir of SubJob. Accesses the parent Job output_dir to infer this."""
        return Path(self.parent.output_dir, str(self.id))
 

◆ ready()

ready ( self )

inherited

Returns whether or not the Job has finished. If the job has subjobs then it will return true when they are all finished.
It will return False as soon as it hits the first failure. Meaning that you cannot guarantee that all subjobs will have
their status updated when calling this method. Instead use :py:meth:`update_status` to update all statuses if necessary.

Definition at line 412 of file backends.py.

    def ready(self):
        """
        Returns whether or not the Job has finished. If the job has subjobs then it will return true when they are all finished.
        It will return False as soon as it hits the first failure. Meaning that you cannot guarantee that all subjobs will have
        their status updated when calling this method. Instead use :py:meth:`update_status` to update all statuses if necessary.
        """
        if not self.result:
            B2DEBUG(29, f"You requested the ready() status for {self} but there is no result object set, returning False.")
            return False
        else:
            return self.result.ready()
 

◆ status() [1/2]

status ( self )

Returns the status of this SubJob.

Reimplemented from Job.

Definition at line 721 of file backends.py.

    def status(self):
        """
        Returns the status of this SubJob.
        """
        return self._status
 

◆ status() [2/2]

status	(		self,
			status )

Sets the status of this Job.

Reimplemented from Job.

Definition at line 728 of file backends.py.

    def status(self, status):
        """
        Sets the status of this Job.
        """
        # Print an error only if the job failed.
        if status == "failed":
            B2ERROR(f"Setting {self.name} status to failed")
        else:
            B2INFO(f"Setting {self.name} status to {status}")
        self._status = status
 

◆ subjobs()

subjobs ( self )

A subjob cannot have subjobs. Always return empty list.

Reimplemented from Job.

Definition at line 740 of file backends.py.

    def subjobs(self):
        """
        A subjob cannot have subjobs. Always return empty list.
        """
        return []
 

◆ update_status()

update_status ( self )

inherited

Calls :py:meth:`update_status` on the job's result. The result object should update all of the subjobs (if there are any)
in the best way for the type of result object/backend.

Definition at line 424 of file backends.py.

    def update_status(self):
        """
        Calls :py:meth:`update_status` on the job's result. The result object should update all of the subjobs (if there are any)
        in the best way for the type of result object/backend.
        """
        if not self.result:
            B2DEBUG(29, f"You requested update_status() for {self} but there is no result object set yet. Probably not submitted.")
        else:
            self.result.update_status()
        return self.status
 

◆ working_dir()

working_dir ( self )

Getter for working_dir of SubJob. Accesses the parent Job working_dir to infer this.

Reimplemented from Job.

Definition at line 711 of file backends.py.

    def working_dir(self):
        """Getter for working_dir of SubJob. Accesses the parent Job working_dir to infer this."""
        return Path(self.parent.working_dir, str(self.id))
 

Member Data Documentation

◆ _status

str _status = "init"

protectedinherited

The actual status of the overall Job.

The property handles querying for the subjob status to set this

Definition at line 404 of file backends.py.

◆ args

args = []

inherited

The arguments that will be applied to the cmd (These are ignored by SubJobs as they have their own arguments)

Definition at line 376 of file backends.py.

◆ backend_args

dict backend_args = {}

inherited

Config dictionary for the backend to use when submitting the job.

Saves us from having multiple attributes that may or may not be used.

Definition at line 383 of file backends.py.

◆ cmd

list cmd = []

inherited

Command and arguments as a list that will be run by the job on the backend.

Definition at line 374 of file backends.py.

◆ exit_statuses

list exit_statuses = ["failed", "completed"]

staticinherited

Job statuses that correspond to the Job being finished (successfully or not)

Definition at line 355 of file backends.py.

◆ id

id = subjob_id

Id of Subjob.

Definition at line 689 of file backends.py.

◆ input_files

input_files = []

inherited

Input files to job (str), a list of these is copied to the working directory.

Definition at line 378 of file backends.py.

◆ input_sandbox_files

list input_sandbox_files = []

inherited

Files to be copied directly into the working directory (pathlib.Path).

Not the input root files, those should be in Job.input_files.

Definition at line 366 of file backends.py.

◆ output_patterns

list output_patterns = []

inherited

Files that we produce during the job and want to be returned.

Can use wildcard (*)

Definition at line 372 of file backends.py.

◆ parent

parent = job

Job() instance of parent to this SubJob.

Definition at line 691 of file backends.py.

◆ result

result = None

inherited

The result object of this Job.

Only filled once the job is submitted to a backend since the backend creates a special result class depending on its type.

Definition at line 402 of file backends.py.

◆ setup_cmds

setup_cmds = []

inherited

Bash commands to run before the main self.cmd (mainly used for batch system setup)

Definition at line 380 of file backends.py.

◆ splitter

splitter = None

inherited

The SubjobSplitter used to create subjobs if necessary.

Definition at line 363 of file backends.py.

◆ statuses

dict statuses = {"init": 0, "submitted": 1, "running": 2, "failed": 3, "completed": 4}

staticinherited

Allowed Job status dictionary.

The key is the status name and the value is its level. The lowest level out of all subjobs is the one that is the overall status of the overall job.

Definition at line 352 of file backends.py.

The documentation for this class was generated from the following file:

calibration/scripts/caf/backends.py

Public Member Functions

Public Attributes

Static Public Attributes

Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ __getattr__()

◆ __repr__()

◆ _get_overall_status_from_subjobs()

◆ append_current_basf2_setup_cmds()

◆ check_input_data_files()

◆ copy_input_sandbox_files_to_working_dir()

◆ create_subjob()

◆ dump_input_data()

◆ dump_to_json()

◆ from_json()

◆ full_command()

◆ job_dict()

◆ name()

◆ output_dir()

◆ ready()

◆ status() [1/2]

◆ status() [2/2]

◆ subjobs()

◆ update_status()

◆ working_dir()

Member Data Documentation

◆ _status

◆ args

◆ backend_args

◆ cmd

◆ exit_statuses

◆ id

◆ input_files

◆ input_sandbox_files

◆ output_patterns

◆ parent

◆ result

◆ setup_cmds

◆ splitter

◆ statuses

◆ init()

◆ getattr()

◆ repr()