Public Member Functions
def	__init__ (self)
	The default constructor. More...

def	adjust_path (self, str path)
	This method can be used if path names are different on submission and execution hosts. More...

def	available (self)
	The cluster should always be available to accept new jobs. More...

def	execute (self, Script job, options="", dry=False, tag="current")
	Takes a Script object and a string with options and runs it on the cluster, either with ROOT or with basf2, depending on the file type. More...

def	is_job_finished (self, Script job)
	Checks whether the '.done'-file has been created for a job. More...

def	terminate (self, Script job)
	Terminate a running job, not support with this backend so ignore the call.

Static Public Member Functions
def	is_supported ()

def	name ()

def	description ()

Public Attributes
	submit_command
	The command to submit a job. More...

	requirement_vmem
	required vmem by the job in GB, required on DESY NAF, otherwise jobs get killed due to memory consumption

	requirement_storage
	the storage IO in GB which can be performed by each job. More...

	queuename
	Queue best suitable for execution at DESY NAF.

	path
	The path, where the help files are being created Maybe there should be a special subfolder for them?

	logger
	Contains a reference to the logger-object from validate_basf2 Set up the logging functionality for the 'cluster execution'-Class, so we can log to validate_basf2.py's log what is going on in .execute and .is_finished.

	tools
	We need to set up the same environment on the cluster like on the local machine. More...

	b2setup
	The command for b2setup (and setoption)

	clusterlog
	The file object to which all cluster messages will be written.

Detailed Description

A class that provides the controls for running jobs on a (remote)
Sun Grid Engine cluster. It provides two methods:
- is_job_finished(job): Returns True or False, depending on whether the job
    has finished execution
- execute(job): Takes a job and executes it by sending it to the cluster

Definition at line 23 of file clustercontrolsge.py.

Constructor & Destructor Documentation

◆ init()

def __init__ ( self )

The default constructor.

Holds the current working directory, which is also the location of the shellscripts that are being sent to the cluster.
Initializes a logger which writes to validate_basf2.py's log.
Finds the revision of basf2 that will be set up on the cluster.

Definition at line 53 of file clustercontrolsge.py.

     def __init__(self):
         """!
         The default constructor.
         - Holds the current working directory, which is also the location of
           the shellscripts that are being sent to the cluster.
         - Initializes a logger which writes to validate_basf2.py's log.
         - Finds the revision of basf2 that will be set up on the cluster.
         """
  
         
         self.submit_command = (
             "qsub -cwd -l h_vmem={requirement_vmem}G,"
             "h_fsize={requirement_storage}G "
             "-oo {logfile} -q {queuename} -V"
         )
  
         
         self.requirement_vmem = 4
  
         
         self.requirement_storage = 50
  
         
         self.queuename = "short.q"
  
         
         self.path = os.getcwd()
  
         
         self.logger = logging.getLogger("validate_basf2")
  
         
  
         
         self.tools = self.adjust_path(os.environ["BELLE2_TOOLS"])
         belle2_release_dir = os.environ.get("BELLE2_RELEASE_DIR", None)
         belle2_local_dir = os.environ.get("BELLE2_LOCAL_DIR", None)
  
         
         self.b2setup = "b2setup"
         if belle2_release_dir is not None:
             self.b2setup += " " + belle2_release_dir.split("/")[-1]
         if belle2_local_dir is not None:
             self.b2setup = (
                 "MY_BELLE2_DIR="
                 + self.adjust_path(belle2_local_dir)
                 + " "
                 + self.b2setup
             )
         if os.environ.get("BELLE2_OPTION") != "debug":
             self.b2setup += "; b2code-option " + os.environ.get("BELLE2_OPTION")
  
         # Write to log which revision we are using
         self.logger.debug(f"Setting up the following release: {self.b2setup}")
  
         # Define the folder in which the log of the cluster messages will be
         # stored (same folder like the log for validate_basf2.py)
         clusterlog_dir = "./html/logs/__general__/"
         if not os.path.exists(clusterlog_dir):
             os.makedirs(clusterlog_dir)
  
         
         self.clusterlog = open(clusterlog_dir + "clusterlog.log", "w+")
  

Member Function Documentation

◆ adjust_path()

def adjust_path	(		self,
		str	path
	)

This method can be used if path names are different on submission and execution hosts.

Parameters

path	The past that needs to be adjusted

Returns: : The adjusted path

Definition at line 128 of file clustercontrolsge.py.

◆ available()

def available ( self )

The cluster should always be available to accept new jobs.

Returns: : Will always return True if the function can be called

Definition at line 139 of file clustercontrolsge.py.

◆ description()

def description ( )

static

Returns description of this job control

Definition at line 47 of file clustercontrolsge.py.

◆ execute()

def execute	(		self,
		Script	job,
			options = `""`,
			dry = `False`,
			tag = `"current"`
	)

Takes a Script object and a string with options and runs it on the cluster, either with ROOT or with basf2, depending on the file type.

Parameters

job	The steering file object that should be executed
options	Options that will be given to the basf2 command
dry	Whether to perform a dry run or not
tag	The folder within the results directory

Returns: : None

Definition at line 147 of file clustercontrolsge.py.

◆ is_job_finished()

def is_job_finished	(		self,
		Script	job
	)

Checks whether the '.done'-file has been created for a job.

If so, it returns True, else it returns False. Also deletes the .done-File once it has returned True.

Parameters

job	The job of which we want to know if it finished

Returns: : True if the job has finished, otherwise False

Definition at line 238 of file clustercontrolsge.py.

◆ is_supported()

def is_supported ( )

static

Check if qsub is available

Definition at line 33 of file clustercontrolsge.py.

◆ name()

def name ( void )

static

Returns name of this job contol

Definition at line 40 of file clustercontrolsge.py.

Member Data Documentation

◆ requirement_storage

requirement_storage

the storage IO in GB which can be performed by each job.

By default, this is 3GB at DESY which is to small for some validation scripts

Definition at line 77 of file clustercontrolsge.py.

◆ submit_command

submit_command

The command to submit a job.

'LOGFILE' will be replaced by the actual log file name

Definition at line 64 of file clustercontrolsge.py.

◆ tools

tools

We need to set up the same environment on the cluster like on the local machine.

The information can be extracted from $BELLE2_TOOLS, $BELLE2_RELEASE_DIR and $BELLE2_LOCAL_DIR

Path to the basf2 tools and central/local release

Definition at line 97 of file clustercontrolsge.py.

The documentation for this class was generated from the following file:

validation/scripts/clustercontrolsge.py

Public Member Functions

Static Public Member Functions

Public Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ adjust_path()

◆ available()

◆ description()

◆ execute()

◆ is_job_finished()

◆ is_supported()

◆ name()

Member Data Documentation

◆ requirement_storage

◆ submit_command

◆ tools

◆ init()