HTCondor Class Reference

Inheritance diagram for HTCondor:

Classes
class	HTCondorResult

Public Member Functions
	get_batch_submit_script_path (self, job)
	can_submit (self, njobs=1)
	condor_q (cls, class_ads=None, job_id="", username="")
	condor_history (cls, class_ads=None, job_id="", username="")
	submit (self, job, check_can_submit=True, jobs_per_check=100)
	get_submit_script_path (self, job)

Public Attributes
int	global_job_limit = self.default_global_job_limit
	The active job limit.
int	sleep_between_submission_checks = self.default_sleep_between_submission_checks
	Seconds we wait before checking if we can submit a list of jobs.
dict	backend_args = {**self.default_backend_args, **backend_args}
	The backend args that will be applied to jobs unless the job specifies them itself.

Static Public Attributes
str	batch_submit_script = "submit.sub"
	HTCondor batch script (different to the wrapper script of Backend.submit_script)
list	default_class_ads = ["GlobalJobId", "JobStatus", "Owner"]
	Default ClassAd attributes to return from commands like condor_q.
list	submission_cmds = []
	Shell command to submit a script, should be implemented in the derived class.
int	default_global_job_limit = 1000
	Default global limit on the total number of submitted/running jobs that the user can have.
int	default_sleep_between_submission_checks = 30
	Default time betweeon re-checking if the active jobs is below the global job limit.
str	submit_script = "submit.sh"
	Default submission script name.
str	exit_code_file = "__BACKEND_CMD_EXIT_STATUS__"
	Default exit code file name.
dict	default_backend_args = {}
	Default backend_args.

Protected Member Functions
	_make_submit_file (self, job, submit_file_path)
	_add_batch_directives (self, job, batch_file)
	_create_cmd (self, script_path)
	_submit_to_batch (cls, cmd)
	_create_job_result (cls, job, job_id)
	_create_parent_job_result (cls, parent)
	_ (self, job, check_can_submit=True, jobs_per_check=100)
	_ (self, job, check_can_submit=True, jobs_per_check=100)
	_ (self, jobs, check_can_submit=True, jobs_per_check=100)
	_add_wrapper_script_setup (self, job, batch_file)
	_add_wrapper_script_teardown (self, job, batch_file)

Static Protected Member Functions
	_add_setup (job, batch_file)

Detailed Description

Backend for submitting calibration processes to a HTCondor batch system.

Definition at line 1924 of file backends.py.

Member Function Documentation

_() [1/3]

_ ( self, job, check_can_submit = True, jobs_per_check = 100 )

protectedinherited

Submit method of Batch backend for a `SubJob`. Should take `SubJob` object, create needed directories,
create batch script, and send it off with the batch submission command.
It should apply the correct options (default and user requested).

Should set a Result object as an attribute of the job.

Definition at line 1211 of file backends.py.

    def _(self, job, check_can_submit=True, jobs_per_check=100):
        """
        Submit method of Batch backend for a `SubJob`. Should take `SubJob` object, create needed directories,
        create batch script, and send it off with the batch submission command.
        It should apply the correct options (default and user requested).
 
        Should set a Result object as an attribute of the job.
        """
        # Make sure the output directory of the job is created, commented out due to permission issues
        # job.output_dir.mkdir(parents=True, exist_ok=True)
        # Make sure the working directory of the job is created
        job.working_dir.mkdir(parents=True, exist_ok=True)
        job.copy_input_sandbox_files_to_working_dir()
        job.dump_input_data()
        # Make submission file if needed
        batch_submit_script_path = self.get_batch_submit_script_path(job)
        self._make_submit_file(job, batch_submit_script_path)
        # Get the bash file we will actually run, might be the same file
        script_path = self.get_submit_script_path(job)
        # Construct the batch submission script (with directives if that is supported)
        with open(script_path, mode="w") as batch_file:
            self._add_batch_directives(job, batch_file)
            self._add_wrapper_script_setup(job, batch_file)
            self._add_setup(job, batch_file)
            print(job.full_command, file=batch_file)
            self._add_wrapper_script_teardown(job, batch_file)
        os.chmod(script_path, 0o755)
        B2INFO(f"Submitting {job}")
        # Do the actual batch submission
        cmd = self._create_cmd(batch_submit_script_path)
        output = self._submit_to_batch(cmd)
        self._create_job_result(job, output)
        job.status = "submitted"
        B2INFO(f"{job} submitted")
 

_() [2/3]

_ ( self, job, check_can_submit = True, jobs_per_check = 100 )

protectedinherited

Submit method of Batch backend. Should take job object, create needed directories, create batch script,
and send it off with the batch submission command, applying the correct options (default and user requested.)

Should set a Result object as an attribute of the job.

Definition at line 1247 of file backends.py.

    def _(self, job, check_can_submit=True, jobs_per_check=100):
        """
        Submit method of Batch backend. Should take job object, create needed directories, create batch script,
        and send it off with the batch submission command, applying the correct options (default and user requested.)
 
        Should set a Result object as an attribute of the job.
        """
        # Make sure the output directory of the job is created, commented out due to permissions issue
        # job.output_dir.mkdir(parents=True, exist_ok=True)
        # Make sure the working directory of the job is created
        job.working_dir.mkdir(parents=True, exist_ok=True)
        # Check if we have any valid input files
        job.check_input_data_files()
        # Add any required backend args that are missing (I'm a bit hesitant to actually merge with job.backend_args)
        # just in case you want to resubmit the same job with different backend settings later.
        # job_backend_args = {**self.backend_args, **job.backend_args}
 
        # If there's no splitter then we just submit the Job with no SubJobs
        if not job.splitter:
            # Get all of the requested files for the input sandbox and copy them to the working directory
            job.copy_input_sandbox_files_to_working_dir()
            job.dump_input_data()
            # Make submission file if needed
            batch_submit_script_path = self.get_batch_submit_script_path(job)
            self._make_submit_file(job, batch_submit_script_path)
            # Get the bash file we will actually run
            script_path = self.get_submit_script_path(job)
            # Construct the batch submission script (with directives if that is supported)
            with open(script_path, mode="w") as batch_file:
                self._add_batch_directives(job, batch_file)
                self._add_wrapper_script_setup(job, batch_file)
                self._add_setup(job, batch_file)
                print(job.full_command, file=batch_file)
                self._add_wrapper_script_teardown(job, batch_file)
            os.chmod(script_path, 0o755)
            B2INFO(f"Submitting {job}")
            # Do the actual batch submission
            cmd = self._create_cmd(batch_submit_script_path)
            output = self._submit_to_batch(cmd)
            self._create_job_result(job, output)
            job.status = "submitted"
            B2INFO(f"{job} submitted")
        else:
            # Create subjobs according to the splitter's logic
            job.splitter.create_subjobs(job)
            # Submit the subjobs
            self.submit(list(job.subjobs.values()))
            # After submitting subjobs, make a Job.result for the parent Job object, used to call ready() on
            self._create_parent_job_result(job)
 

_() [3/3]

_ ( self, jobs, check_can_submit = True, jobs_per_check = 100 )

protectedinherited

Submit method of Batch Backend that takes a list of jobs instead of just one and submits each one.

Definition at line 1298 of file backends.py.

    def _(self, jobs, check_can_submit=True, jobs_per_check=100):
        """
        Submit method of Batch Backend that takes a list of jobs instead of just one and submits each one.
        """
        B2INFO(f"Submitting a list of {len(jobs)} jobs to a Batch backend")
        # Technically this could be a list of Jobs or SubJobs. And if it is a list of Jobs then it might not
        # be necessary to check if we can submit right now. We could do it later during the submission of the
        # SubJob list. However in the interest of simpler code we just do the check here, and re-check again
        # if a SubJob list comes through this function. Slightly inefficient, but much simpler logic.
 
        # The first thing to do is make sure that we are iterating through the jobs list in chunks that are
        # equal to or smaller than the global limit. Otherwise nothing will ever submit.
 
        if jobs_per_check > self.global_job_limit:
            B2INFO(f"jobs_per_check (={jobs_per_check}) but this is higher than the global job "
                   f"limit for this backend (={self.global_job_limit}). Will instead use the "
                   " value of the global job limit.")
            jobs_per_check = self.global_job_limit
 
        # We group the jobs list into chunks of length jobs_per_check
        for jobs_to_submit in grouper(jobs_per_check, jobs):
            # Wait until we are allowed to submit
            while not self.can_submit(njobs=len(jobs_to_submit)):
                B2INFO("Too many jobs are currently in the batch system globally. Waiting until submission can continue...")
                time.sleep(self.sleep_between_submission_checks)
            else:
                # We loop here since we have already checked if the number of jobs is low enough, we don't want to hit this
                # function again unless one of the jobs has subjobs.
                B2INFO(f"Submitting the next {len(jobs_to_submit)} jobs...")
                for job in jobs_to_submit:
                    self.submit(job, check_can_submit, jobs_per_check)
        B2INFO(f"All {len(jobs)} requested jobs submitted")
 

_add_batch_directives()

_add_batch_directives ( self, job, batch_file )

protected

For HTCondor leave empty as the directives are already included in the submit file.

Reimplemented from Batch.

Definition at line 1971 of file backends.py.

    def _add_batch_directives(self, job, batch_file):
        """
        For HTCondor leave empty as the directives are already included in the submit file.
        """
        print('#!/bin/bash', file=batch_file)
 

_add_setup()

_add_setup ( job, batch_file )

staticprotectedinherited

Adds setup lines to the shell script file.

Definition at line 806 of file backends.py.

    def _add_setup(job, batch_file):
        """
        Adds setup lines to the shell script file.
        """
        for line in job.setup_cmds:
            print(line, file=batch_file)
 

_add_wrapper_script_setup()

_add_wrapper_script_setup ( self, job, batch_file )

protectedinherited

Adds lines to the submitted script that help with job monitoring/setup. Mostly here so that we can insert
`trap` statements for Ctrl-C situations.

Definition at line 813 of file backends.py.

    def _add_wrapper_script_setup(self, job, batch_file):
        """
        Adds lines to the submitted script that help with job monitoring/setup. Mostly here so that we can insert
        `trap` statements for Ctrl-C situations.
        """
        start_wrapper = f"""# ---
# trap ctrl-c and call ctrl_c()
trap '(ctrl_c 130)' SIGINT
trap '(ctrl_c 143)' SIGTERM
 
function write_exit_code() {{
  echo "Writing $1 to exit status file"
  echo "$1" > {self.exit_code_file}
  exit $1
}}
 
function ctrl_c() {{
  trap '' SIGINT SIGTERM
  echo "** Trapped Ctrl-C **"
  echo "$1" > {self.exit_code_file}
  exit $1
}}
# ---"""
        print(start_wrapper, file=batch_file)
 

_add_wrapper_script_teardown()

_add_wrapper_script_teardown ( self, job, batch_file )

protectedinherited

Adds lines to the submitted script that help with job monitoring/teardown. Mostly here so that we can insert
an exit code of the job cmd being written out to a file. Which means that we can know if the command was
successful or not even if the backend server/monitoring database purges the data about our job i.e. If PBS
removes job information too quickly we may never know if a job succeeded or failed without some kind of exit
file.

Definition at line 838 of file backends.py.

    def _add_wrapper_script_teardown(self, job, batch_file):
        """
        Adds lines to the submitted script that help with job monitoring/teardown. Mostly here so that we can insert
        an exit code of the job cmd being written out to a file. Which means that we can know if the command was
        successful or not even if the backend server/monitoring database purges the data about our job i.e. If PBS
        removes job information too quickly we may never know if a job succeeded or failed without some kind of exit
        file.
        """
        end_wrapper = """# ---
write_exit_code $?"""
        print(end_wrapper, file=batch_file)
 

_create_cmd()

_create_cmd ( self, script_path )

protected

Reimplemented from Batch.

Definition at line 1977 of file backends.py.

    def _create_cmd(self, script_path):
        """
        """
        submission_cmd = self.submission_cmds[:]
        submission_cmd.append(script_path.as_posix())
        return submission_cmd
 

_create_job_result()

_create_job_result ( cls, job, job_id )

protected

Reimplemented from Batch.

Definition at line 2121 of file backends.py.

    def _create_job_result(cls, job, job_id):
        """
        """
        B2INFO(f"Job ID of {job} recorded as: {job_id}")
        job.result = cls.HTCondorResult(job, job_id)
 

_create_parent_job_result()

_create_parent_job_result ( cls, parent )

protected

We want to be able to call `ready()` on the top level `Job.result`. So this method needs to exist
so that a Job.result object actually exists. It will be mostly empty and simply updates subjob
statuses and allows the use of ready().

Reimplemented from Backend.

Definition at line 2128 of file backends.py.

    def _create_parent_job_result(cls, parent):
        parent.result = cls.HTCondorResult(parent, None)
 

_make_submit_file()

_make_submit_file ( self, job, submit_file_path )

protected

Fill HTCondor submission file.

Reimplemented from Batch.

Definition at line 1945 of file backends.py.

    def _make_submit_file(self, job, submit_file_path):
        """
        Fill HTCondor submission file.
        """
        # Find all files/directories in the working directory to copy on the worker node
 
        files_to_transfer = [i.as_posix() for i in job.working_dir.iterdir()]
 
        job_backend_args = {**self.backend_args, **job.backend_args}  # Merge the two dictionaries, with the job having priority
 
        with open(submit_file_path, "w") as submit_file:
            print(f'executable = {self.get_submit_script_path(job)}', file=submit_file)
            print(f'log = {Path(job.output_dir, "htcondor.log").as_posix()}', file=submit_file)
            print(f'output = {Path(job.working_dir, _STDOUT_FILE).as_posix()}', file=submit_file)
            print(f'error = {Path(job.working_dir, _STDERR_FILE).as_posix()}', file=submit_file)
            print('transfer_input_files = ', ','.join(files_to_transfer), file=submit_file)
            print(f'universe = {job_backend_args["universe"]}', file=submit_file)
            print(f'getenv = {job_backend_args["getenv"]}', file=submit_file)
            print(f'request_memory = {job_backend_args["request_memory"]}', file=submit_file)
            print('should_transfer_files = Yes', file=submit_file)
            print('when_to_transfer_output = ON_EXIT', file=submit_file)
            # Any other lines in the backend args that we don't deal with explicitly but maybe someone wants to insert something
            for line in job_backend_args["extra_lines"]:
                print(line, file=submit_file)
            print('queue', file=submit_file)
 

_submit_to_batch()

_submit_to_batch ( cls, cmd )

protected

Do the actual batch submission command and collect the output to find out the job id for later monitoring.

Reimplemented from Batch.

Definition at line 1991 of file backends.py.

    def _submit_to_batch(cls, cmd):
        """
        Do the actual batch submission command and collect the output to find out the job id for later monitoring.
        """
        job_dir = Path(cmd[-1]).parent.as_posix()
        sub_out = ""
        attempt = 0
        sleep_time = 30
 
        while attempt < 3:
            try:
                sub_out = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True, cwd=job_dir)
                break
            except subprocess.CalledProcessError as e:
                attempt += 1
                if attempt == 3:
                    B2ERROR(f"Error during condor_submit: {str(e)} occurred more than 3 times.")
                    raise e
                else:
                    B2ERROR(f"Error during condor_submit: {str(e)}, sleeping for {sleep_time} seconds.")
                    time.sleep(30)
        return re.search(r"(\d+\.\d+) - \d+\.\d+", sub_out).groups()[0]
 

can_submit()

can_submit	(		self,
			njobs = 1 )

Checks the global number of jobs in HTCondor right now (submitted or running) for this user.
Returns True if the number is lower that the limit, False if it is higher.

Parameters:
    njobs (int): The number of jobs that we want to submit before checking again. Lets us check if we
        are sufficiently below the limit in order to (somewhat) safely submit. It is slightly dangerous to
        assume that it is safe to submit too many jobs since there might be other processes also submitting jobs.
        So njobs really shouldn't be abused when you might be getting close to the limit i.e. keep it <=250
        and check again before submitting more.

Reimplemented from Batch.

Definition at line 2131 of file backends.py.

    def can_submit(self, njobs=1):
        """
        Checks the global number of jobs in HTCondor right now (submitted or running) for this user.
        Returns True if the number is lower that the limit, False if it is higher.
 
        Parameters:
            njobs (int): The number of jobs that we want to submit before checking again. Lets us check if we
                are sufficiently below the limit in order to (somewhat) safely submit. It is slightly dangerous to
                assume that it is safe to submit too many jobs since there might be other processes also submitting jobs.
                So njobs really shouldn't be abused when you might be getting close to the limit i.e. keep it <=250
                and check again before submitting more.
        """
        B2DEBUG(29, "Calling HTCondor().can_submit()")
        jobs_info = self.condor_q()
        total_jobs = jobs_info["NJOBS"]
        B2INFO(f"Total jobs active in the HTCondor system is currently {total_jobs}")
        if (total_jobs + njobs) > self.global_job_limit:
            B2INFO(f"Since the global limit is {self.global_job_limit} we cannot submit {njobs} jobs until some complete.")
            return False
        else:
            B2INFO("There is enough space to submit more jobs.")
            return True
 

condor_history()

condor_history	(		cls,
			class_ads = None,
			job_id = "",
			username = "" )

Simplistic interface to the ``condor_history`` command. lets you request information about all jobs matching the filters
``job_id`` and ``username``. Note that setting job_id negates username so it is ignored.
The result is a JSON dictionary filled by output of the ``-json`` ``condor_history`` option.

Parameters:
    class_ads (list[str]): A list of condor_history ClassAds that you would like information about.
        By default we give {cls.default_class_ads}, increasing the amount of class_ads increase the time taken
        by the condor_q call.
    job_id (str): String representation of the Job ID given by condor_submit during submission.
        If this argument is given then the output of this function will be only information about this job.
        If this argument is not given, then all jobs matching the other filters will be returned.
    username (str): By default we return information about only the current user's jobs. By giving
        a username you can access the job information of a specific user's jobs. By giving ``username='all'`` you will
        receive job information from all known user jobs matching the other filters. This is limited to 10000 records
        and isn't recommended.

Returns:
    dict: JSON dictionary of the form:

    .. code-block:: python

      {
        "NJOBS":<number of records returned by command>,
        "JOBS":[
                {
                 <ClassAd: value>, ...
                }, ...
               ]
      }

Definition at line 2222 of file backends.py.

    def condor_history(cls, class_ads=None, job_id="", username=""):
        """
        Simplistic interface to the ``condor_history`` command. lets you request information about all jobs matching the filters
        ``job_id`` and ``username``. Note that setting job_id negates username so it is ignored.
        The result is a JSON dictionary filled by output of the ``-json`` ``condor_history`` option.
 
        Parameters:
            class_ads (list[str]): A list of condor_history ClassAds that you would like information about.
                By default we give {cls.default_class_ads}, increasing the amount of class_ads increase the time taken
                by the condor_q call.
            job_id (str): String representation of the Job ID given by condor_submit during submission.
                If this argument is given then the output of this function will be only information about this job.
                If this argument is not given, then all jobs matching the other filters will be returned.
            username (str): By default we return information about only the current user's jobs. By giving
                a username you can access the job information of a specific user's jobs. By giving ``username='all'`` you will
                receive job information from all known user jobs matching the other filters. This is limited to 10000 records
                and isn't recommended.
 
        Returns:
            dict: JSON dictionary of the form:
 
            .. code-block:: python
 
              {
                "NJOBS":<number of records returned by command>,
                "JOBS":[
                        {
                         <ClassAd: value>, ...
                        }, ...
                       ]
              }
        """
        B2DEBUG(29, f"Calling HTCondor.condor_history(class_ads={class_ads}, job_id={job_id}, username={username})")
        if not class_ads:
            class_ads = cls.default_class_ads
        # Output fields should be comma separated.
        field_list_cmd = ",".join(class_ads)
        cmd_list = ["condor_history", "-json", "-attributes", field_list_cmd]
        # If job_id is set then we ignore all other filters
        if job_id:
            cmd_list.append(job_id)
        else:
            if not username:
                username = os.environ["USER"]
            # If the username is set to all it is a special case
            if username != "all":
                cmd_list.append(username)
        # We get a JSON serialisable summary from condor_q. But we will alter it slightly to be more similar to other backends
        cmd = " ".join(cmd_list)
        B2DEBUG(29, f"Calling subprocess with command = '{cmd}'")
        try:
            records = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True, shell=True)
        except BaseException:
            records = None
 
        if records:
            records = decode_json_string(records)
        else:
            records = []
 
        jobs_info = {"JOBS": records}
        jobs_info["NJOBS"] = len(jobs_info["JOBS"])  # Just to avoid having to len() it in the future
        return jobs_info
 
 

condor_q()

condor_q	(		cls,
			class_ads = None,
			job_id = "",
			username = "" )

Simplistic interface to the `condor_q` command. lets you request information about all jobs matching the filters
'job_id' and 'username'. Note that setting job_id negates username so it is ignored.
The result is the JSON dictionary returned by output of the ``-json`` condor_q option.

Parameters:
    class_ads (list[str]): A list of condor_q ClassAds that you would like information about.
        By default we give {cls.default_class_ads}, increasing the amount of class_ads increase the time taken
        by the condor_q call.
    job_id (str): String representation of the Job ID given by condor_submit during submission.
        If this argument is given then the output of this function will be only information about this job.
        If this argument is not given, then all jobs matching the other filters will be returned.
    username (str): By default we return information about only the current user's jobs. By giving
        a username you can access the job information of a specific user's jobs. By giving ``username='all'`` you will
        receive job information from all known user jobs matching the other filters. This may be a LOT of jobs
        so it isn't recommended.

Returns:
    dict: JSON dictionary of the form:

    .. code-block:: python

      {
        "NJOBS":<number of records returned by command>,
        "JOBS":[
                {
                 <ClassAd: value>, ...
                }, ...
               ]
      }

Definition at line 2155 of file backends.py.

    def condor_q(cls, class_ads=None, job_id="", username=""):
        """
        Simplistic interface to the `condor_q` command. lets you request information about all jobs matching the filters
        'job_id' and 'username'. Note that setting job_id negates username so it is ignored.
        The result is the JSON dictionary returned by output of the ``-json`` condor_q option.
 
        Parameters:
            class_ads (list[str]): A list of condor_q ClassAds that you would like information about.
                By default we give {cls.default_class_ads}, increasing the amount of class_ads increase the time taken
                by the condor_q call.
            job_id (str): String representation of the Job ID given by condor_submit during submission.
                If this argument is given then the output of this function will be only information about this job.
                If this argument is not given, then all jobs matching the other filters will be returned.
            username (str): By default we return information about only the current user's jobs. By giving
                a username you can access the job information of a specific user's jobs. By giving ``username='all'`` you will
                receive job information from all known user jobs matching the other filters. This may be a LOT of jobs
                so it isn't recommended.
 
        Returns:
            dict: JSON dictionary of the form:
 
            .. code-block:: python
 
              {
                "NJOBS":<number of records returned by command>,
                "JOBS":[
                        {
                         <ClassAd: value>, ...
                        }, ...
                       ]
              }
        """
        B2DEBUG(29, f"Calling HTCondor.condor_q(class_ads={class_ads}, job_id={job_id}, username={username})")
        if not class_ads:
            class_ads = cls.default_class_ads
        # Output fields should be comma separated.
        field_list_cmd = ",".join(class_ads)
        cmd_list = ["condor_q", "-json", "-attributes", field_list_cmd]
        # If job_id is set then we ignore all other filters
        if job_id:
            cmd_list.append(job_id)
        else:
            if not username:
                username = os.environ["USER"]
            # If the username is set to all it is a special case
            if username == "all":
                cmd_list.append("-allusers")
            else:
                cmd_list.append(username)
        # We get a JSON serialisable summary from condor_q. But we will alter it slightly to be more similar to other backends
        cmd = " ".join(cmd_list)
        B2DEBUG(29, f"Calling subprocess with command = '{cmd}'")
        # condor_q occasionally fails
        try:
            records = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True, shell=True)
        except BaseException:
            records = None
 
        if records:
            records = decode_json_string(records)
        else:
            records = []
        jobs_info = {"JOBS": records}
        jobs_info["NJOBS"] = len(jobs_info["JOBS"])  # Just to avoid having to len() it in the future
        return jobs_info
 

get_batch_submit_script_path()

get_batch_submit_script_path	(		self,
			job )

Construct the Path object of the .sub file that we will use to describe the job.

Reimplemented from Batch.

Definition at line 1984 of file backends.py.

    def get_batch_submit_script_path(self, job):
        """
        Construct the Path object of the .sub file that we will use to describe the job.
        """
        return Path(job.working_dir, self.batch_submit_script)
 

get_submit_script_path()

get_submit_script_path ( self, job )

inherited

Construct the Path object of the bash script file that we will submit. It will contain
the actual job command, wrapper commands, setup commands, and any batch directives

Definition at line 859 of file backends.py.

    def get_submit_script_path(self, job):
        """
        Construct the Path object of the bash script file that we will submit. It will contain
        the actual job command, wrapper commands, setup commands, and any batch directives
        """
        return Path(job.working_dir, self.submit_script)
 
 

submit()

submit ( self, job, check_can_submit = True, jobs_per_check = 100 )

inherited

Reimplemented from Backend.

Definition at line 1204 of file backends.py.

    def submit(self, job, check_can_submit=True, jobs_per_check=100):
        """
        """
        raise NotImplementedError("This is an abstract submit(job) method that shouldn't have been called. "
                                  "Did you submit a (Sub)Job?")
 

Member Data Documentation

backend_args

dict backend_args = {**self.default_backend_args, **backend_args}

inherited

The backend args that will be applied to jobs unless the job specifies them itself.

Definition at line 796 of file backends.py.

batch_submit_script

batch_submit_script = "submit.sub"

static

HTCondor batch script (different to the wrapper script of Backend.submit_script)

Definition at line 1929 of file backends.py.

default_backend_args

dict default_backend_args = {}

staticinherited

Default backend_args.

Definition at line 788 of file backends.py.

default_class_ads

list default_class_ads = ["GlobalJobId", "JobStatus", "Owner"]

static

Default ClassAd attributes to return from commands like condor_q.

Definition at line 1943 of file backends.py.

default_global_job_limit

int default_global_job_limit = 1000

staticinherited

Default global limit on the total number of submitted/running jobs that the user can have.

This limit will not affect the total number of jobs that are eventually submitted. But the jobs won't actually be submitted until this limit can be respected i.e. until the number of total jobs in the Batch system goes down. Since we actually submit in chunks of N jobs, before checking this limit value again, this value needs to be a little lower than the real batch system limit. Otherwise you could accidentally go over during the N job submission if other processes are checking and submitting concurrently. This is quite common for the first submission of jobs from parallel calibrations.

Note that if there are other jobs already submitted for your account, then these will count towards this limit.

Definition at line 1155 of file backends.py.

default_sleep_between_submission_checks

int default_sleep_between_submission_checks = 30

staticinherited

Default time betweeon re-checking if the active jobs is below the global job limit.

Definition at line 1157 of file backends.py.

exit_code_file

str exit_code_file = "__BACKEND_CMD_EXIT_STATUS__"

staticinherited

Default exit code file name.

Definition at line 786 of file backends.py.

global_job_limit

int global_job_limit = self.default_global_job_limit

inherited

The active job limit.

This is 'global' because we want to prevent us accidentally submitting too many jobs from all current and previous submission scripts.

Definition at line 1166 of file backends.py.

sleep_between_submission_checks

sleep_between_submission_checks = self.default_sleep_between_submission_checks

inherited

Seconds we wait before checking if we can submit a list of jobs.

Only relevant once we hit the global limit of active jobs, which is a lot usually.

Definition at line 1169 of file backends.py.

submission_cmds

list submission_cmds = []

staticinherited

Shell command to submit a script, should be implemented in the derived class.

Definition at line 1142 of file backends.py.

submit_script

submit_script = "submit.sh"

staticinherited

Default submission script name.

Definition at line 784 of file backends.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/backends.py