# 21.6.4. Batch submission¶

When a batch job is submitted from a work server at KEKCC, the job is scheduled by LSF (Platform Load Sharing Facility developed by IBM) which dispatches and executes the job on a calculation server. It is important to select an appropriate queue for your jobs.

In this lesson, we will go through some commands that are often used in analysis.

## Basic commands¶

It is important to know which queues can be used and what is the workload of the queue.

To display the information about all batch queues:

bqueues [-u $USER]  If no option is given, this returns the following information about all queues: queue name, queue priority, queue status, task statistics, and job state statistics. $ bqueues -u $USER QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP s 120 Open:Active 3000 600 - - 30885 28774 2111 0 l 100 Open:Active - 1000 - - 39959 29395 10564 0 h 100 Open:Active 1500 300 - - 1719 1416 303 0 p 100 Open:Active 1500 300 - - 577 0 577 0 sx 100 Open:Active - 200 - - 2185 1986 199 0 lx 100 Open:Active - 200 - - 2 0 2 0 hx 100 Open:Active 300 60 - - 0 0 0 0 px 100 Open:Active 300 100 - - 0 0 0 0 P1 100 Open:Active - - - - 0 0 0 0 Pmpi 100 Open:Active - - - - 0 0 0 0 b_b 100 Open:Active - 1000 - - 0 0 0 0 cmb_p 100 Open:Active - 300 - - 0 0 0 0 cmb_px 100 Open:Active - 100 - 10 0 0 0 0 a 100 Open:Active - 4 - - 11 3 8 0 dc_generic 100 Open:Active - - - - 0 0 0 0  Different queues have different settings. For analysis you can use s, l, or h. For short jobs with running time under 3 hours, the queue s is preferable. For jobs with execution time more than 3 hours, you might want to use the queue l which gives jobs up to 24 hours of running time. More information about LSF queues can be found here. This command also displays the current “Fairshare” values. Fairshare defines the priorities of jobs that are dispatched. bqueues -l [<queue_name>]  Here the square brackets […] indicate that the argument is optional and <…> indicates that the value should be filled in by you. Exercise Check your priorities on queue s. Solution bqueues -l s [| grep$USER]


Provide queue name after -l, and combine with grep command to get your information more quickly. If you have never used the batch queue before, it should be 0.333.

Every uses has the default value of 0.333 to start with. The more jobs you submit, the lower your Fairshare is.

Submit a job

With an example script as

#!/usr/bin/bash
echo "Hello world, this is script ${0}." >> batch_output.txt sleep 20 echo "Finished!" >> batch_output.txt  To submit a job to queue s bsub -q s "bash example.sh"  and check the output $ cat batch_output.txt
Hello world, this is script example.sh.
Finished!


Use the same method, you can submit Python or basf2 scripts to bqueues!

bsub -q <queue name> "basf2 <your_working_script>"


Note

Always test your script before submitting large scale jobs to batch system.

Display job status

To check the job status

bjobs [-q <queue name>] [<job_ID>]


Exercise

Submit a basf2 job to queue l, and then check the status of your jobs.

Hint

A simple basf2 job could be the following:

# Print all variables known to the variable manager
from variables import printVars
printVars()


Solution

Submission:

\$ bsub -q l "basf2 one_of_example.py"
Job <xxxxxxxx> is submitted to queue <l>.


To check the status, use one of the following:

bjobs -q l <xxxxxxxx>, bjobs <xxxxxxxx>, or just bjobs alone.

Cancel a job

To cancel jobs

bkill [<job_ID>]


Note

Use 0 to kill all jobs. Use this with caution.

Sometimes bjobs will still show the job after we tried to terminate it. In this case we can use the -r option to kill it by force. More information is given here.

## Optional¶

In some scenarios you might want to stop the submitted jobs and resume them later. For instance this might be due to scheduled maintenance of storage elements where the input data is located or the updating of analysis global tags that used in your jobs.

Suspend jobs

To suspend unfinished jobs

bstop <job_ID>


Note

Use -a to suspend all jobs.

Resume jobs

To resumes suspended jobs

bresume <job_ID>


Key points

• Submit a script to the short queue with bsub -q s "bash myscript.sh"

• Check job queues with bequeues

• Kill jobs with bkill <job id>

• Always test your scripts before large scale submissions!

Stuck? We can help!

If you get stuck or have any questions to the online book material, the #starterkit-workshop channel in our chat is full of nice people who will provide fast help.

Refer to Collaborative Tools. for other places to get help if you have specific or detailed questions about your own analysis.

Improving things!

If you know how to do it, we recommend you to report bugs and other requests with JIRA. Make sure to use the documentation-training component of the Belle II Software project.

If you just want to give very quick feedback, use the last box “Quick feedback”.

Please make sure to be as precise as possible to make it easier for us to fix things! So for example:

• typos (where?)

• missing bits of information (what?)

• bugs (what did you do? what goes wrong?)

• too hard exercises (which one?)

• etc.

If you are familiar with git and want to create your first pull request for the software, take a look at How to contribute. We’d be happy to have you on the team!

Quick feedback!

Author of this lesson

Chia-Ling Hsu