Job Class Reference

Inheritance diagram for Job:

Public Member Functions
def	__init__ (self, name, job_dict=None)
def	__repr__ (self)
def	ready (self)
def	update_status (self)
def	create_subjob (self, i, input_files=None, args=None)
def	status (self)
def	status (self, status)
def	output_dir (self)
def	output_dir (self, value)
def	working_dir (self)
def	working_dir (self, value)
def	input_sandbox_files (self)
def	input_sandbox_files (self, value)
def	input_files (self)
def	input_files (self, value)
def	max_subjobs (self)
def	max_subjobs (self, value)
def	max_files_per_subjob (self)
def	max_files_per_subjob (self, value)
def	dump_to_json (self, file_path)
def	from_json (cls, file_path)
def	job_dict (self)
def	dump_input_data (self)
def	copy_input_sandbox_files_to_working_dir (self)
def	check_input_data_files (self)
def	full_command (self)
def	append_current_basf2_setup_cmds (self)

Public Attributes
	name
	Job object's name. More...
	splitter
	The SubjobSplitter used to create subjobs if necessary.
	input_sandbox_files
	Files to be copied directly into the working directory (pathlib.Path). More...
	working_dir
	Working directory of the job (pathlib.Path). More...
	output_dir
	Output directory (pathlib.Path), where we will download our output_files to. More...
	output_patterns
	Files that we produce during the job and want to be returned. More...
	cmd
	Command and arguments as a list that wil be run by the job on the backend.
	args
	The arguments that will be applied to the cmd (These are ignored by SubJobs as they have their own arguments)
	input_files
	Input files to job (str), a list of these is copied to the working directory.
	setup_cmds
	Bash commands to run before the main self.cmd (mainly used for batch system setup)
	backend_args
	Config dictionary for the backend to use when submitting the job. More...
	subjobs
	dict of subjobs assigned to this job
	result
	The result object of this Job. More...
	status
	Not a real attribute, it's a property.

Static Public Attributes
dictionary	statuses = {"init": 0, "submitted": 1, "running": 2, "failed": 3, "completed": 4}
	Allowed Job status dictionary. More...
list	exit_statuses = ["failed", "completed"]
	Job statuses that correspond to the Job being finished (successfully or not)

Private Member Functions
def	_get_overall_status_from_subjobs (self)

Private Attributes
	_status
	The actual status of the overall Job. More...
	_output_dir
	_working_dir
	_input_sandbox_files
	_input_files

Detailed Description

This generic Job object is used to tell a Backend what to do.
This object basically holds necessary information about a process you want to submit to a `Backend`.
It should *not* do anything that is backend specific, just hold the configuration for a job to be
successfully submitted and monitored using a backend. The result attribute is where backend
specific job monitoring goes.

Parameters:
    name (str): Simply a name to describe the Job, not used for any critical purpose in the CAF

.. warning:: It is recommended to always use absolute paths for files when submitting a `Job`.

Definition at line 318 of file backends.py.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
			name,
			job_dict = None
	)

Definition at line 339 of file backends.py.

Member Function Documentation

__repr__()

def __repr__

(

self

)

Representation of Job class (what happens when you print a Job() instance).

Reimplemented in SubJob.

Definition at line 390 of file backends.py.

append_current_basf2_setup_cmds()

def append_current_basf2_setup_cmds

(

self

)

This adds simple setup commands like ``source /path/to/tools/b2setup`` to your `Job`.
It should detect if you are using a local release or CVMFS and append the correct commands
so that the job will have the same basf2 release environment. It should also detect
if a local release is not compiled with the ``opt`` option.

Note that this *doesn't mean that every environment variable is inherited* from the submitting
process environment.

Definition at line 614 of file backends.py.

check_input_data_files()

def check_input_data_files

(

self

)

Check the input files and make sure that there aren't any duplicates.
Also check if the files actually exist if possible.

Definition at line 572 of file backends.py.

copy_input_sandbox_files_to_working_dir()

def copy_input_sandbox_files_to_working_dir

(

self

)

Get all of the requested files for the input sandbox and copy them to the working directory.
Files like the submit.sh or input_data.json are not part of this process.

Definition at line 561 of file backends.py.

create_subjob()

def create_subjob	(		self,
			i,
			input_files = None,
			args = None
	)

Creates a subjob Job object that references that parent Job.
Returns the SubJob object at the end.

Definition at line 419 of file backends.py.

dump_input_data()

def dump_input_data

(

self

)

Dumps the `Job.input_files` attribute to a JSON file. input_files should be a list of
string URI objects.

Definition at line 553 of file backends.py.

dump_to_json()

def dump_to_json	(		self,
			file_path
	)

Dumps the Job object configuration to a JSON file so that it can be read in again later.

Parameters:
  file_path(`basf2.Path`): The filepath we'll dump to

Definition at line 516 of file backends.py.

full_command()

def full_command

(

self

)

Returns:
    str: The full command that this job will run including any arguments.

Definition at line 602 of file backends.py.

job_dict()

def job_dict

(

self

)

Returns:
    dict: A JSON serialisable representation of the `Job` and its `SubJob` objects.
    `Path <basf2.Path>` objects are converted to string via ``Path.as_posix()``.

Reimplemented in SubJob.

Definition at line 533 of file backends.py.

ready()

def ready

(

self

)

Returns whether or not the Job has finished. If the job has subjobs then it will return true when they are all finished.
It will return False as soon as it hits the first failure. Meaning that you cannot guarantee that all subjobs will have
their status updated when calling this method. Instead use :py:meth:`update_status` to update all statuses if necessary.

Definition at line 396 of file backends.py.

status() [1/2]

def status

(

self

)

Returns the status of this Job. If the job has subjobs then it will return the overall status equal to the lowest
subjob status in the hierarchy of statuses in `Job.statuses`.

Reimplemented in SubJob.

Definition at line 435 of file backends.py.

status() [2/2]

def status	(		self,
			status
	)

Sets the status of this Job.

Reimplemented in SubJob.

Definition at line 455 of file backends.py.

update_status()

def update_status

(

self

)

Calls :py:meth:`update_status` on the job's result. The result object should update all of the subjobs (if there are any)
in the best way for the type of result object/backend.

Definition at line 408 of file backends.py.

Member Data Documentation

_status

_status

private

The actual status of the overall Job.

The property handles querying for the subjob status to set this

Definition at line 388 of file backends.py.

backend_args

backend_args

Config dictionary for the backend to use when submitting the job.

Saves us from having multiple attributes that may or may not be used.

Definition at line 367 of file backends.py.

input_sandbox_files

input_sandbox_files

Files to be copied directly into the working directory (pathlib.Path).

Not the input root files, those should be in Job.input_files.

Definition at line 350 of file backends.py.

name

name

Job object's name.

Only descriptive, not necessarily unique.

Definition at line 343 of file backends.py.

output_dir

output_dir

Output directory (pathlib.Path), where we will download our output_files to.

Default is '.'

Definition at line 354 of file backends.py.

output_patterns

output_patterns

Files that we produce during the job and want to be returned.

Can use wildcard (*)

Definition at line 356 of file backends.py.

result

result

The result object of this Job.

Only filled once the job is submitted to a backend since the backend creates a special result class depending on its type.

Definition at line 386 of file backends.py.

statuses

dictionary statuses = {"init": 0, "submitted": 1, "running": 2, "failed": 3, "completed": 4}

static

Allowed Job status dictionary.

The key is the status name and the value is its level. The lowest level out of all subjobs is the one that is the overal status of the overall job.

Definition at line 334 of file backends.py.

working_dir

working_dir

Working directory of the job (pathlib.Path).

Default is '.', mostly used in Local() backend

Definition at line 352 of file backends.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/backends.py