ArgumentsSplitter Class Reference

Inheritance diagram for ArgumentsSplitter:

Public Member Functions
	__init__ (self, *, arguments_generator=None, max_subjobs=None)
	create_subjobs (self, job)
	assign_arguments (self, job)
	__repr__ (self)

Public Attributes
	max_subjobs = max_subjobs
	If we try to create more than this many subjobs we throw an exception, if None then there is no maximum.
	arguments_generator = arguments_generator
	The ArgumentsGenerator used when creating subjobs.

Detailed Description

Creates SubJobs based on the given argument generator. The generator will be called until a `StopIteration` is issued.
Be VERY careful to not accidentally give an infinite generator! Otherwise it will simply create SubJobs until you run out
of memory. You can set the `ArgumentsSplitter.max_subjobs` parameter to try and prevent this and throw an exception.

This splitter is useful for MC production jobs where you don't have any input files, but you want to control the exp/run
numbers of subjobs. If you do have input files set for the parent `Job` objects, then the same input files will be
assigned to every `SubJob`.

Parameters:
    arguments_generator (ArgumentsGenerator): The standard ArgumentsGenerator that is used to assign arguments

Definition at line 290 of file backends.py.

Constructor & Destructor Documentation

__init__()

__init__	(		self,
		*	,
			arguments_generator = None,
			max_subjobs = None )

Definition at line 304 of file backends.py.

    def __init__(self, *, arguments_generator=None, max_subjobs=None):
        """
        """
        super().__init__(arguments_generator=arguments_generator)
        
        self.max_subjobs = max_subjobs
 

Member Function Documentation

__repr__()

__repr__ ( self )

inherited

Definition at line 205 of file backends.py.

    def __repr__(self):
        """
        """
        return f"{self.__class__.__name__}"
 
 

assign_arguments()

assign_arguments ( self, job )

inherited

Use the `arguments_generator` (if one exists) to assign the argument tuples to the
subjobs.

Definition at line 179 of file backends.py.

    def assign_arguments(self, job):
        """
        Use the `arguments_generator` (if one exists) to assign the argument tuples to the
        subjobs.
        """
        if self.arguments_generator:
            arg_gen = self.arguments_generator.generator
            generating = True
            for subjob in sorted(job.subjobs.values(), key=lambda sj: sj.id):
                if generating:
                    try:
                        args = arg_gen.send(subjob)
                    except StopIteration:
                        B2ERROR(f"StopIteration called when getting args for {subjob}, "
                                "setting all subsequent subjobs to have empty argument tuples.")
                        args = tuple()  # If our generator finishes before the subjobs we use empty argument tuples
                        generating = False
                else:
                    args = tuple()
                B2DEBUG(29, f"Arguments for {subjob}: {args}")
                subjob.args = args
            return
 
        B2INFO(f"No ArgumentsGenerator assigned to the {self} so subjobs of {job} "
               "won't automatically have arguments assigned.")
 

create_subjobs()

create_subjobs	(		self,
			job )

This function creates subjobs for the parent job passed in. It creates subjobs until the
`SubjobSplitter.arguments_generator` finishes.

If `ArgumentsSplitter.max_subjobs` is set, then it will throw an exception if more than this number of
subjobs are created.

Reimplemented from SubjobSplitter.

Definition at line 311 of file backends.py.

    def create_subjobs(self, job):
        """
        This function creates subjobs for the parent job passed in. It creates subjobs until the
        `SubjobSplitter.arguments_generator` finishes.
 
        If `ArgumentsSplitter.max_subjobs` is set, then it will throw an exception if more than this number of
        subjobs are created.
        """
        arg_gen = self.arguments_generator.generator
        for i in count():
            # Reached the maximum?
            if i >= self.max_subjobs:
                raise SplitterError(f"{self} tried to create more subjobs than the maximum (={self.max_subjobs}).")
            try:
                subjob = SubJob(job, i, job.input_files)  # We manually create it because we might not need it
                args = arg_gen.send(subjob)  # Might throw StopIteration
                B2INFO(f"Creating {job}.{subjob}")
                B2DEBUG(29, f"Arguments for {subjob}: {args}")
                subjob.args = args
                job.subjobs[i] = subjob
            except StopIteration:
                break
        B2INFO(f"{self} created {i+1} Subjobs for {job}")
 
 

Member Data Documentation

arguments_generator

arguments_generator = arguments_generator

inherited

The ArgumentsGenerator used when creating subjobs.

Definition at line 171 of file backends.py.

max_subjobs

max_subjobs = max_subjobs

If we try to create more than this many subjobs we throw an exception, if None then there is no maximum.

Definition at line 309 of file backends.py.

---

The documentation for this class was generated from the following file:

• calibration/scripts/caf/backends.py