3.6.4. Batch submission#

When a batch job is submitted from a work server at KEKCC, the job is scheduled by LSF (Platform Load Sharing Facility developed by IBM) which dispatches and executes the job on a calculation server. It is important to select an appropriate queue for your jobs.

In this lesson, we will go through some commands that are often used in analysis.

Basic commands#

Displays information about batch queues

It is important to know which queues can be used and what is the workload of the queue.

To display the information about all batch queues:

bqueues [-u $USER]

If no option is given, this returns the following information about all queues: queue name, queue priority, queue status, task statistics, and job state statistics.

$ bqueues -u $USER
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP
s               120  Open:Active    3000  600    -    - 30885 28774  2111     0
l               100  Open:Active       - 1000    -    - 39959 29395 10564     0
h               100  Open:Active    1500  300    -    -  1719  1416   303     0
p               100  Open:Active    1500  300    -    -   577     0   577     0
sx              100  Open:Active       -  200    -    -  2185  1986   199     0
lx              100  Open:Active       -  200    -    -     2     0     2     0
hx              100  Open:Active     300   60    -    -     0     0     0     0
px              100  Open:Active     300  100    -    -     0     0     0     0
P1              100  Open:Active       -    -    -    -     0     0     0     0
Pmpi            100  Open:Active       -    -    -    -     0     0     0     0
b_b             100  Open:Active       - 1000    -    -     0     0     0     0
cmb_p           100  Open:Active       -  300    -    -     0     0     0     0
cmb_px          100  Open:Active       -  100    -   10     0     0     0     0
a               100  Open:Active       -    4    -    -    11     3     8     0
dc_generic      100  Open:Active       -    -    -    -     0     0     0     0

Different queues have different settings. For analysis you can use s, l, or h. For short jobs with a computing time (CPU time) of under 3 hours, the queue s is preferable. For jobs with execution time more than 3 hours, you might want to use the queue l which gives jobs up to 24 hours of computing time. More information about LSF queues can be found here.

This command also displays the current “Fairshare” values. Fairshare defines the priorities of jobs that are dispatched.

bqueues -l [<queue_name>]

Here the square brackets […] indicate that the argument is optional and <…> indicates that the value should be filled in by you.

Exercise

Check your priorities on queue s.

Solution

bqueues -l s [| grep $USER]

Provide queue name after -l, and combine with grep command to get your information more quickly. If you have never used the batch queue before, it should be 0.333.

Every uses has the default value of 0.333 to start with. The more jobs you submit, the lower your Fairshare is.

Submit a job

With an example script as

#!/usr/bin/bash
echo "Hello world, this is script ${0}." >> batch_output.txt
sleep 20
echo "Finished!" >> batch_output.txt

To submit a job to queue s

bsub -q s "bash example.sh"

and check the output

$ cat batch_output.txt
Hello world, this is script example.sh.
Finished!

Use the same method, you can submit Python or basf2 scripts to bqueues!

bsub -q <queue name> "basf2 <your_working_script>"

Note

Always test your script before submitting large scale jobs to batch system.

Display job status

To check the job status

bjobs [-q <queue name>] [<job_ID>]

Exercise

Submit a basf2 job to queue l, and then check the status of your jobs.

Hint

A simple basf2 job could be the following:

# Print all variables known to the variable manager
from variables import printVars
printVars()

Solution

Submission:

$ bsub -q l "basf2 one_of_example.py"
Job <xxxxxxxx> is submitted to queue <l>.

To check the status, use one of the following:

bjobs -q l <xxxxxxxx>, bjobs <xxxxxxxx>, or just bjobs alone.

Cancel a job

To cancel jobs

bkill [<job_ID>]

Note

Use 0 to kill all jobs. Use this with caution.

Sometimes bjobs will still show the job after we tried to terminate it. In this case we can use the -r option to kill it by force. More information is given here.

Optional#

In some scenarios you might want to stop the submitted jobs and resume them later. For instance this might be due to scheduled maintenance of storage elements where the input data is located or the updating of analysis global tags that used in your jobs.

Suspend jobs

To suspend unfinished jobs

bstop <job_ID>

Note

Use -a to suspend all jobs.

Resume jobs

To resumes suspended jobs

bresume <job_ID>

Key points

  • Submit a script to the short queue with bsub -q s "bash myscript.sh"

  • Check job queues with bequeues

  • Kill jobs with bkill <job id>

  • Always test your scripts before large scale submissions!

Stuck? We can help!

If you get stuck or have any questions to the online book material, the #starterkit-workshop channel in our chat is full of nice people who will provide fast help.

Refer to Collaborative Tools. for other places to get help if you have specific or detailed questions about your own analysis.

Improving things!

If you know how to do it, we recommend you to report bugs and other requests with GitLab. Make sure to use the documentation-training label of the basf2 project.

If you just want to give very quick feedback, use the last box “Quick feedback”.

Please make sure to be as precise as possible to make it easier for us to fix things! So for example:

  • typos (where?)

  • missing bits of information (what?)

  • bugs (what did you do? what goes wrong?)

  • too hard exercises (which one?)

  • etc.

If you are familiar with git and want to create your first merge request for the software, take a look at How to contribute. We’d be happy to have you on the team!

Quick feedback!

Author of this lesson

Chia-Ling Hsu