LSF.LSFResult Class Reference

Inheritance diagram for LSF.LSFResult:

Public Member Functions
	__init__ (self, job, job_id)
	update_status (self)
	ready (self)
	get_exit_code_from_file (self)

Public Attributes
	job_id = job_id
	job id given by LSF
	job = job
	Job object for result.
	time_to_wait_for_exit_code_file = timedelta(minutes=5)
	After our first attempt to view the exit code file once the job is 'finished', how long should we wait for it to exist before timing out?
	exit_code_file_initial_time = None
	Time we started waiting for the exit code file to appear.

Static Public Attributes
dict	backend_code_to_status
	LSF statuses mapped to Job statuses.

Protected Member Functions
	_update_result_status (self, bjobs_output)
	_get_status_from_output (self, output)

Protected Attributes
bool	_is_ready = False
	Quicker way to know if it's ready once it has already been found.

Detailed Description

Simple class to help monitor status of jobs submitted by LSF Backend.

You pass in a `Job` object and job id from a bsub command.
When you call the `ready` method it runs bjobs to see whether or not the job has finished.

Definition at line 1677 of file backends.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
			job,
			job_id )

Pass in the job object and the job id to allow the result to do monitoring and perform
post processing of the job.

Definition at line 1693 of file backends.py.

        def __init__(self, job, job_id):
            """
            Pass in the job object and the job id to allow the result to do monitoring and perform
            post processing of the job.
            """
            super().__init__(job)
            
            self.job_id = job_id
 

Member Function Documentation

_get_status_from_output()

_get_status_from_output ( self, output )

protected

Get status from output

Definition at line 1756 of file backends.py.

        def _get_status_from_output(self, output):
            """
            Get status from output
            """
            if output["JOBS"] and "ERROR" in output["JOBS"][0]:
                if output["JOBS"][0]["ERROR"] == f"Job <{self.job_id}> is not found":
                    raise KeyError(f"No job record in the 'output' argument had the 'JOBID'=={self.job_id}")
                else:
                    raise BackendError(f"Unidentified Error during status check for {self.job}: {output}")
            else:
                for job_info in output["JOBS"]:
                    if job_info["JOBID"] == self.job_id:
                        return job_info["STAT"]
                else:
                    raise KeyError(f"No job record in the 'output' argument had the 'JOBID'=={self.job_id}")
 
    @classmethod

_update_result_status()

_update_result_status ( self, bjobs_output )

protected

Parameters:
    bjobs_output (dict): The JSON output of a previous call to bjobs which we can reuse to find the
        status of this job. Obviously you should only be passing a JSON dict that contains the 'stat' and
        'id' information, otherwise it is useless.

Definition at line 1715 of file backends.py.

        def _update_result_status(self, bjobs_output):
            """
            Parameters:
                bjobs_output (dict): The JSON output of a previous call to bjobs which we can reuse to find the
                    status of this job. Obviously you should only be passing a JSON dict that contains the 'stat' and
                    'id' information, otherwise it is useless.
 
            """
            try:
                backend_status = self._get_status_from_output(bjobs_output)
            except KeyError:
                # If this happens then maybe the job id wasn't in the bjobs_output argument because it finished.
                # Instead of failing immediately we try re-running bjobs here explicitly and then fail if it still isn't there.
                bjobs_output = LSF.bjobs(output_fields=["stat", "id"], job_id=str(self.job_id))
                try:
                    backend_status = self._get_status_from_output(bjobs_output)
                except KeyError:
                    # If this happened, maybe we're looking at an old finished job. We could fall back to bhist, but it's
                    # slow and terrible. Instead let's try looking for the exit code file.
                    try:
                        exit_code = self.get_exit_code_from_file()
                    except FileNotFoundError:
                        waiting_time = datetime.now() - self.exit_code_file_initial_time
                        if self.time_to_wait_for_exit_code_file > waiting_time:
                            B2ERROR(f"Exit code file for {self.job} missing and we can't wait longer. Setting exit code to 1.")
                            exit_code = 1
                        else:
                            B2WARNING(f"Exit code file for {self.job} missing, will wait longer.")
                            return
                    if exit_code:
                        backend_status = "EXIT"
                    else:
                        backend_status = "FINISHED"
            try:
                new_job_status = self.backend_code_to_status[backend_status]
            except KeyError as err:
                raise BackendError(f"Unidentified backend status found for {self.job}: {backend_status}") from err
 
            if new_job_status != self.job.status:
                self.job.status = new_job_status
 

get_exit_code_from_file()

get_exit_code_from_file ( self )

inherited

Read the exit code file to discover the exit status of the job command. Useful fallback if the job is no longer
known to the job database (batch system purged it for example). Since some backends may take time to download
the output files of the job back to the working directory we use a time limit on how long to wait.

Definition at line 908 of file backends.py.

    def get_exit_code_from_file(self):
        """
        Read the exit code file to discover the exit status of the job command. Useful fallback if the job is no longer
        known to the job database (batch system purged it for example). Since some backends may take time to download
        the output files of the job back to the working directory we use a time limit on how long to wait.
        """
        if not self.exit_code_file_initial_time:
            self.exit_code_file_initial_time = datetime.now()
        exit_code_path = Path(self.job.working_dir, Backend.exit_code_file)
        with open(exit_code_path) as f:
            exit_code = int(f.read().strip())
            B2DEBUG(29, f"Exit code from file for {self.job} was {exit_code}")
            return exit_code
 
 

ready()

ready ( self )

inherited

Returns whether or not this job result is known to be ready. Doesn't actually change the job status. Just changes
the 'readiness' based on the known job status.

Definition at line 887 of file backends.py.

    def ready(self):
        """
        Returns whether or not this job result is known to be ready. Doesn't actually change the job status. Just changes
        the 'readiness' based on the known job status.
        """
        B2DEBUG(29, f"Calling {self.job}.result.ready()")
        if self._is_ready:
            return True
        elif self.job.status in self.job.exit_statuses:
            self._is_ready = True
            return True
        else:
            return False
 

update_status()

update_status

(

self

)

Update the job's (or subjobs') status by calling bjobs.

Reimplemented from Result.

Definition at line 1702 of file backends.py.

        def update_status(self):
            """
            Update the job's (or subjobs') status by calling bjobs.
            """
            B2DEBUG(29, f"Calling {self.job.name}.result.update_status()")
            # Get all jobs info and reuse it for each status update to minimise tie spent on this updating.
            bjobs_output = LSF.bjobs(output_fields=["stat", "id"])
            if self.job.subjobs:
                for subjob in self.job.subjobs.values():
                    subjob.result._update_result_status(bjobs_output)
            else:
                self._update_result_status(bjobs_output)
 

Member Data Documentation

_is_ready

bool _is_ready = False

protectedinherited

Quicker way to know if it's ready once it has already been found.

Saves a lot of calls to batch system commands.

Definition at line 880 of file backends.py.

backend_code_to_status

dict backend_code_to_status

static

Initial value:

=  {"RUN": "running",
                                  "DONE": "completed",
                                  "FINISHED": "completed",
                                  "EXIT": "failed",
                                  "PEND": "submitted"
                                  }

LSF statuses mapped to Job statuses.

Definition at line 1686 of file backends.py.

exit_code_file_initial_time

exit_code_file_initial_time = None

inherited

Time we started waiting for the exit code file to appear.

Definition at line 885 of file backends.py.

job

job = job

inherited

Job object for result.

Definition at line 878 of file backends.py.

job_id

job_id = job_id

job id given by LSF

Definition at line 1700 of file backends.py.

time_to_wait_for_exit_code_file

time_to_wait_for_exit_code_file = timedelta(minutes=5)

inherited

After our first attempt to view the exit code file once the job is 'finished', how long should we wait for it to exist before timing out?

Definition at line 883 of file backends.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/backends.py