MaxSubjobsSplitter Class Reference

Inheritance diagram for MaxSubjobsSplitter:

Public Member Functions
	__init__ (self, *, arguments_generator=None, max_subjobs=1000)
	create_subjobs (self, job)
	assign_arguments (self, job)
	__repr__ (self)

Public Attributes
	max_subjobs = max_subjobs
	The maximum number of SubJob objects to be created, input files are split evenly between them.
	arguments_generator = arguments_generator
	The ArgumentsGenerator used when creating subjobs.

Detailed Description

Definition at line 241 of file backends.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
		*	,
			arguments_generator = None,
			max_subjobs = 1000 )

Parameters:
    max_subjobs (int): The maximum number ofsubjobs that will be created.

Definition at line 245 of file backends.py.

    def __init__(self, *, arguments_generator=None, max_subjobs=1000):
        """
        Parameters:
            max_subjobs (int): The maximum number ofsubjobs that will be created.
        """
        super().__init__(arguments_generator=arguments_generator)
        
        self.max_subjobs = max_subjobs
 

Member Function Documentation

__repr__()

__repr__ ( self )

inherited

Definition at line 204 of file backends.py.

    def __repr__(self):
        """
        """
        return f"{self.__class__.__name__}"
 
 

assign_arguments()

assign_arguments ( self, job )

inherited

Use the `arguments_generator` (if one exists) to assign the argument tuples to the
subjobs.

Definition at line 178 of file backends.py.

    def assign_arguments(self, job):
        """
        Use the `arguments_generator` (if one exists) to assign the argument tuples to the
        subjobs.
        """
        if self.arguments_generator:
            arg_gen = self.arguments_generator.generator
            generating = True
            for subjob in sorted(job.subjobs.values(), key=lambda sj: sj.id):
                if generating:
                    try:
                        args = arg_gen.send(subjob)
                    except StopIteration:
                        B2ERROR(f"StopIteration called when getting args for {subjob}, "
                                "setting all subsequent subjobs to have empty argument tuples.")
                        args = tuple()  # If our generator finishes before the subjobs we use empty argument tuples
                        generating = False
                else:
                    args = tuple()
                B2DEBUG(29, f"Arguments for {subjob}: {args}")
                subjob.args = args
            return
 
        B2INFO(f"No ArgumentsGenerator assigned to the {self} so subjobs of {job} "
               "won't automatically have arguments assigned.")
 

create_subjobs()

create_subjobs	(		self,
			job )

This function creates subjobs for the parent job passed in. It creates as many subjobs as required
by the number of input files up to the maximum set by `MaxSubjobsSplitter.max_subjobs`. If there are
more input files than `max_subjobs` it instead groups files by the minimum number per subjob in order to
respect the subjob limit e.g. If you have 11 input files and a maximum number of subjobs of 4, then it
will create 4 subjobs, 3 of them with 3 input files, and one with 2 input files.

Reimplemented from SubjobSplitter.

Definition at line 254 of file backends.py.

    def create_subjobs(self, job):
        """
        This function creates subjobs for the parent job passed in. It creates as many subjobs as required
        by the number of input files up to the maximum set by `MaxSubjobsSplitter.max_subjobs`. If there are
        more input files than `max_subjobs` it instead groups files by the minimum number per subjob in order to
        respect the subjob limit e.g. If you have 11 input files and a maximum number of subjobs of 4, then it
        will create 4 subjobs, 3 of them with 3 input files, and one with 2 input files.
        """
        if not job.input_files:
            B2WARNING(f"Subjob splitting by input files requested, but no input files exist for {job}. No subjobs created.")
            return
 
        # Since we will be popping from the left of the input file list, we construct a deque
        remaining_input_files = deque(job.input_files)
        # The initial number of available subjobs, will go down as we create them
        available_subjobs = self.max_subjobs
        subjob_i = 0
        while remaining_input_files:
            # How many files should we use for this subjob?
            num_input_files = ceil(len(remaining_input_files) / available_subjobs)
            # Pop them from the remaining files
            subjob_input_files = []
            for i in range(num_input_files):
                subjob_input_files.append(remaining_input_files.popleft())
            # Create the actual subjob
            job.create_subjob(subjob_i, input_files=subjob_input_files)
            subjob_i += 1
            available_subjobs -= 1
 
        self.assign_arguments(job)
        B2INFO(f"{self} created {subjob_i} Subjobs for {job}")
 
 

Member Data Documentation

arguments_generator

arguments_generator = arguments_generator

inherited

The ArgumentsGenerator used when creating subjobs.

Definition at line 170 of file backends.py.

max_subjobs

max_subjobs = max_subjobs

The maximum number of SubJob objects to be created, input files are split evenly between them.

Definition at line 252 of file backends.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/backends.py