HTCondor.HTCondorResult Class Reference

Inheritance diagram for HTCondor.HTCondorResult:

Public Member Functions
	__init__ (self, job, job_id)
	update_status (self)
	ready (self)
	get_exit_code_from_file (self)

Public Attributes
	job_id = job_id
	job id given by HTCondor
	job = job
	Job object for result.
	time_to_wait_for_exit_code_file = timedelta(minutes=5)
	After our first attempt to view the exit code file once the job is 'finished', how long should we wait for it to exist before timing out?
	exit_code_file_initial_time = None
	Time we started waiting for the exit code file to appear.

Static Public Attributes
dict	backend_code_to_status
	HTCondor statuses mapped to Job statuses.

Protected Member Functions
	_update_result_status (self, condor_q_output)

Protected Attributes
bool	_is_ready = False
	Quicker way to know if it's ready once it has already been found.

Detailed Description

Simple class to help monitor status of jobs submitted by HTCondor Backend.

You pass in a `Job` object and job id from a condor_submit command.
When you call the `ready` method it runs condor_q and, if needed, ``condor_history``
to see whether or not the job has finished.

Definition at line 2022 of file backends.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
			job,
			job_id )

Pass in the job object and the job id to allow the result to do monitoring and perform
post processing of the job.

Definition at line 2041 of file backends.py.

        def __init__(self, job, job_id):
            """
            Pass in the job object and the job id to allow the result to do monitoring and perform
            post processing of the job.
            """
            super().__init__(job)
            
            self.job_id = job_id
 

Member Function Documentation

_update_result_status()

_update_result_status ( self, condor_q_output )

protected

In order to be slightly more efficient we pass in a previous call to condor_q to see if it can work.
If it is there we update the job's status. If not we are forced to start calling condor_q and, if needed,
``condor_history``, etc.

Parameters:
    condor_q_output (dict): The JSON output of a previous call to `HTCondor.condor_q` which we can reuse to find the
        status of this job if it was active when that command ran.

Definition at line 2063 of file backends.py.

        def _update_result_status(self, condor_q_output):
            """
            In order to be slightly more efficient we pass in a previous call to condor_q to see if it can work.
            If it is there we update the job's status. If not we are forced to start calling condor_q and, if needed,
            ``condor_history``, etc.
 
            Parameters:
                condor_q_output (dict): The JSON output of a previous call to `HTCondor.condor_q` which we can reuse to find the
                    status of this job if it was active when that command ran.
            """
            B2DEBUG(29, f"Calling {self.job}.result._update_result_status()")
            jobs_info = []
            for job_record in condor_q_output["JOBS"]:
                job_id = job_record["GlobalJobId"].split("#")[1]
                if job_id == self.job_id:
                    B2DEBUG(29, f"Found {self.job_id} in condor_q_output.")
                    jobs_info.append(job_record)
 
            # Let's look for the exit code file where we expect it
            if not jobs_info:
                try:
                    exit_code = self.get_exit_code_from_file()
                except FileNotFoundError:
                    waiting_time = datetime.now() - self.exit_code_file_initial_time
                    if self.time_to_wait_for_exit_code_file > waiting_time:
                        B2ERROR(f"Exit code file for {self.job} missing and we can't wait longer. Setting exit code to 1.")
                        exit_code = 1
                    else:
                        B2WARNING(f"Exit code file for {self.job} missing, will wait longer.")
                        return
                if exit_code:
                    jobs_info = [{"JobStatus": 6, "HoldReason": None}]  # Set to failed
                else:
                    jobs_info = [{"JobStatus": 4, "HoldReason": None}]  # Set to completed
 
            # If this job wasn't in the passed in condor_q output, let's try our own with the specific job_id
            if not jobs_info:
                jobs_info = HTCondor.condor_q(job_id=self.job_id, class_ads=["JobStatus", "HoldReason"])["JOBS"]
 
            # If no job information is returned then the job already left the queue
            # check in the history to see if it succeeded or failed
            if not jobs_info:
                try:
                    jobs_info = HTCondor.condor_history(job_id=self.job_id, class_ads=["JobStatus", "HoldReason"])["JOBS"]
                except KeyError:
                    hold_reason = "No Reason Known"
 
            # Still no record of it after waiting for the exit code file?
            if not jobs_info:
                jobs_info = [{"JobStatus": 6, "HoldReason": None}]  # Set to failed
 
            job_info = jobs_info[0]
            backend_status = job_info["JobStatus"]
            # if job is held (backend_status = 5) then report why then keep waiting
            if backend_status == 5:
                hold_reason = job_info.get("HoldReason", None)
                B2WARNING(f"{self.job} on hold because of {hold_reason}. Keep waiting.")
                backend_status = 2
            try:
                new_job_status = self.backend_code_to_status[backend_status]
            except KeyError as err:
                raise BackendError(f"Unidentified backend status found for {self.job}: {backend_status}") from err
            if new_job_status != self.job.status:
                self.job.status = new_job_status
 
    @classmethod

get_exit_code_from_file()

get_exit_code_from_file ( self )

inherited

Read the exit code file to discover the exit status of the job command. Useful fallback if the job is no longer
known to the job database (batch system purged it for example). Since some backends may take time to download
the output files of the job back to the working directory we use a time limit on how long to wait.

Definition at line 909 of file backends.py.

    def get_exit_code_from_file(self):
        """
        Read the exit code file to discover the exit status of the job command. Useful fallback if the job is no longer
        known to the job database (batch system purged it for example). Since some backends may take time to download
        the output files of the job back to the working directory we use a time limit on how long to wait.
        """
        if not self.exit_code_file_initial_time:
            self.exit_code_file_initial_time = datetime.now()
        exit_code_path = Path(self.job.working_dir, Backend.exit_code_file)
        with open(exit_code_path) as f:
            exit_code = int(f.read().strip())
            B2DEBUG(29, f"Exit code from file for {self.job} was {exit_code}")
            return exit_code
 
 

ready()

ready ( self )

inherited

Returns whether or not this job result is known to be ready. Doesn't actually change the job status. Just changes
the 'readiness' based on the known job status.

Definition at line 888 of file backends.py.

    def ready(self):
        """
        Returns whether or not this job result is known to be ready. Doesn't actually change the job status. Just changes
        the 'readiness' based on the known job status.
        """
        B2DEBUG(29, f"Calling {self.job}.result.ready()")
        if self._is_ready:
            return True
        elif self.job.status in self.job.exit_statuses:
            self._is_ready = True
            return True
        else:
            return False
 

update_status()

update_status

(

self

)

Update the job's (or subjobs') status by calling condor_q.

Reimplemented from Result.

Definition at line 2050 of file backends.py.

        def update_status(self):
            """
            Update the job's (or subjobs') status by calling condor_q.
            """
            B2DEBUG(29, f"Calling {self.job.name}.result.update_status()")
            # Get all jobs info and reuse it for each status update to minimise tie spent on this updating.
            condor_q_output = HTCondor.condor_q()
            if self.job.subjobs:
                for subjob in self.job.subjobs.values():
                    subjob.result._update_result_status(condor_q_output)
            else:
                self._update_result_status(condor_q_output)
 

Member Data Documentation

_is_ready

bool _is_ready = False

protectedinherited

Quicker way to know if it's ready once it has already been found.

Saves a lot of calls to batch system commands.

Definition at line 881 of file backends.py.

backend_code_to_status

dict backend_code_to_status

static

Initial value:

=  {0: "submitted",
                                  1: "submitted",
                                  2: "running",
                                  3: "failed",
                                  4: "completed",
                                  5: "submitted",
                                  6: "failed"
                                  }

HTCondor statuses mapped to Job statuses.

Definition at line 2032 of file backends.py.

exit_code_file_initial_time

exit_code_file_initial_time = None

inherited

Time we started waiting for the exit code file to appear.

Definition at line 886 of file backends.py.

job

job = job

inherited

Job object for result.

Definition at line 879 of file backends.py.

job_id

job_id = job_id

job id given by HTCondor

Definition at line 2048 of file backends.py.

time_to_wait_for_exit_code_file

time_to_wait_for_exit_code_file = timedelta(minutes=5)

inherited

After our first attempt to view the exit code file once the job is 'finished', how long should we wait for it to exist before timing out?

Definition at line 884 of file backends.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/backends.py