Batch submission
Contents
3.6.4. Batch submission#
When a batch job is submitted from a work server at KEKCC, the job is scheduled by LSF (Platform Load Sharing Facility developed by IBM) which dispatches and executes the job on a calculation server. It is important to select an appropriate queue for your jobs.
In this lesson, we will go through some commands that are often used in analysis.
Basic commands#
Displays information about batch queues
It is important to know which queues can be used and what is the workload of the queue.
To display the information about all batch queues:
bqueues [-u $USER]
If no option is given, this returns the following information about all queues: queue name, queue priority, queue status, task statistics, and job state statistics.
$ bqueues -u $USER
QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP
s 120 Open:Active 3000 600 - - 30885 28774 2111 0
l 100 Open:Active - 1000 - - 39959 29395 10564 0
h 100 Open:Active 1500 300 - - 1719 1416 303 0
p 100 Open:Active 1500 300 - - 577 0 577 0
sx 100 Open:Active - 200 - - 2185 1986 199 0
lx 100 Open:Active - 200 - - 2 0 2 0
hx 100 Open:Active 300 60 - - 0 0 0 0
px 100 Open:Active 300 100 - - 0 0 0 0
P1 100 Open:Active - - - - 0 0 0 0
Pmpi 100 Open:Active - - - - 0 0 0 0
b_b 100 Open:Active - 1000 - - 0 0 0 0
cmb_p 100 Open:Active - 300 - - 0 0 0 0
cmb_px 100 Open:Active - 100 - 10 0 0 0 0
a 100 Open:Active - 4 - - 11 3 8 0
dc_generic 100 Open:Active - - - - 0 0 0 0
Different queues have different settings. For analysis you can use s
,
l
, or h
. For short jobs with a computing time (CPU time) of under 3 hours, the queue
s
is preferable. For jobs with execution time more than 3 hours,
you might want to use the queue l
which gives jobs up to 24 hours of
computing time.
More information about LSF queues can be found
here.
This command also displays the current “Fairshare” values. Fairshare defines the priorities of jobs that are dispatched.
bqueues -l [<queue_name>]
Here the square brackets […] indicate that the argument is optional and <…> indicates that the value should be filled in by you.
Exercise
Check your priorities on queue s.
Solution
bqueues -l s [| grep $USER]
Provide queue name after -l
, and combine with grep
command to get your information more quickly.
If you have never used the batch queue before, it should be 0.333.
Every uses has the default value of 0.333 to start with. The more jobs you submit, the lower your Fairshare is.
Submit a job
With an example script as
#!/usr/bin/bash
echo "Hello world, this is script ${0}." >> batch_output.txt
sleep 20
echo "Finished!" >> batch_output.txt
To submit a job to queue s
bsub -q s "bash example.sh"
and check the output
$ cat batch_output.txt
Hello world, this is script example.sh.
Finished!
Use the same method, you can submit Python or basf2 scripts to bqueues!
bsub -q <queue name> "basf2 <your_working_script>"
Note
Always test your script before submitting large scale jobs to batch system.
Display job status
To check the job status
bjobs [-q <queue name>] [<job_ID>]
Exercise
Submit a
basf2
job to queuel
, and then check the status of your jobs.
Hint
A simple basf2
job could be the following:
Solution
Submission:
$ bsub -q l "basf2 one_of_example.py" Job <xxxxxxxx> is submitted to queue <l>.
To check the status, use one of the following:
bjobs -q l <xxxxxxxx>
,bjobs <xxxxxxxx>
, or justbjobs
alone.
Cancel a job
To cancel jobs
bkill [<job_ID>]
Note
Use 0
to kill all jobs. Use this with caution.
Sometimes bjobs
will still show the job after we tried to terminate it.
In this case we can use the -r
option to kill it by force.
More information is given here.
Optional#
In some scenarios you might want to stop the submitted jobs and resume them later. For instance this might be due to scheduled maintenance of storage elements where the input data is located or the updating of analysis global tags that used in your jobs.
Suspend jobs
To suspend unfinished jobs
bstop <job_ID>
Note
Use -a to suspend all jobs.
Resume jobs
To resumes suspended jobs
bresume <job_ID>
Key points
Submit a script to the short queue with
bsub -q s "bash myscript.sh"
Check job queues with
bequeues
Kill jobs with
bkill <job id>
Always test your scripts before large scale submissions!
Stuck? We can help!
If you get stuck or have any questions to the online book material, the #starterkit-workshop channel in our chat is full of nice people who will provide fast help.
Refer to Collaborative Tools. for other places to get help if you have specific or detailed questions about your own analysis.
Improving things!
If you know how to do it, we recommend you to report bugs and other requests
with GitLab.
Make sure to use the documentation-training
label of the basf2
project.
If you just want to give very quick feedback, use the last box “Quick feedback”.
Please make sure to be as precise as possible to make it easier for us to fix things! So for example:
typos (where?)
missing bits of information (what?)
bugs (what did you do? what goes wrong?)
too hard exercises (which one?)
etc.
If you are familiar with git and want to create your first pull request for the software, take a look at How to contribute. We’d be happy to have you on the team!
Quick feedback!
Author of this lesson
Chia-Ling Hsu