MaxFilesSplitter Class Reference

Inheritance diagram for MaxFilesSplitter:

Public Member Functions
	__init__ (self, *, arguments_generator=None, max_files_per_subjob=1)
	create_subjobs (self, job)
	assign_arguments (self, job)
	__repr__ (self)

Public Attributes
	max_files_per_subjob = max_files_per_subjob
	The maximum number of input files that will be used for each SubJob created.
	arguments_generator = arguments_generator
	The ArgumentsGenerator used when creating subjobs.

Detailed Description

Definition at line 211 of file backends.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
		*	,
			arguments_generator = None,
			max_files_per_subjob = 1 )

Parameters:
    max_files_per_subjob (int): The maximum number of input files used per `SubJob` created.

Definition at line 216 of file backends.py.

    def __init__(self, *, arguments_generator=None, max_files_per_subjob=1):
        """
        Parameters:
            max_files_per_subjob (int): The maximum number of input files used per `SubJob` created.
        """
        super().__init__(arguments_generator=arguments_generator)
        
        self.max_files_per_subjob = max_files_per_subjob
 

Member Function Documentation

__repr__()

__repr__ ( self )

inherited

Definition at line 205 of file backends.py.

    def __repr__(self):
        """
        """
        return f"{self.__class__.__name__}"
 
 

assign_arguments()

assign_arguments ( self, job )

inherited

Use the `arguments_generator` (if one exists) to assign the argument tuples to the
subjobs.

Definition at line 179 of file backends.py.

    def assign_arguments(self, job):
        """
        Use the `arguments_generator` (if one exists) to assign the argument tuples to the
        subjobs.
        """
        if self.arguments_generator:
            arg_gen = self.arguments_generator.generator
            generating = True
            for subjob in sorted(job.subjobs.values(), key=lambda sj: sj.id):
                if generating:
                    try:
                        args = arg_gen.send(subjob)
                    except StopIteration:
                        B2ERROR(f"StopIteration called when getting args for {subjob}, "
                                "setting all subsequent subjobs to have empty argument tuples.")
                        args = tuple()  # If our generator finishes before the subjobs we use empty argument tuples
                        generating = False
                else:
                    args = tuple()
                B2DEBUG(29, f"Arguments for {subjob}: {args}")
                subjob.args = args
            return
 
        B2INFO(f"No ArgumentsGenerator assigned to the {self} so subjobs of {job} "
               "won't automatically have arguments assigned.")
 

create_subjobs()

create_subjobs	(		self,
			job )

This function creates subjobs for the parent job passed in. It creates as many subjobs as required
in order to prevent the number of input files per subjob going over the limit set by
`MaxFilesSplitter.max_files_per_subjob`.

Reimplemented from SubjobSplitter.

Definition at line 225 of file backends.py.

    def create_subjobs(self, job):
        """
        This function creates subjobs for the parent job passed in. It creates as many subjobs as required
        in order to prevent the number of input files per subjob going over the limit set by
        `MaxFilesSplitter.max_files_per_subjob`.
        """
        if not job.input_files:
            B2WARNING(f"Subjob splitting by input files requested, but no input files exist for {job}. No subjobs created.")
            return
 
        for i, subjob_input_files in enumerate(grouper(self.max_files_per_subjob, job.input_files)):
            job.create_subjob(i, input_files=subjob_input_files)
 
        self.assign_arguments(job)
 
        B2INFO(f"{self} created {i+1} Subjobs for {job}")
 
 

Member Data Documentation

arguments_generator

arguments_generator = arguments_generator

inherited

The ArgumentsGenerator used when creating subjobs.

Definition at line 171 of file backends.py.

max_files_per_subjob

max_files_per_subjob = max_files_per_subjob

The maximum number of input files that will be used for each SubJob created.

Definition at line 223 of file backends.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/backends.py