eResearch

PBS JOB ARRAY SUBMISSION

There are many times when users may need to execute their program (or similar programs) multiple times.

Depending on the number of program executions, this could be considered tedious and can consume a lot of time.  Additionally, when submitting each job individually, errors and typo’s often creep in and cause errors.  The HPC Scheduler (know as PBS) provides various functionally to make this task a lot simpler.  One of the easiest methods is to use “PBS Job Arrays” to submit multiple jobs to the scheduler at a single time.

One of the major benefits for using job arrays is that it basically submits the entire “job arrays” as a single job.  This reduces the load of the scheduler and allows significantly more jobs to be “queued” without affecting the performance of the HPC scheduler.

 

General PBS Parameters

  • The variable $PBS_O_WORKDIR indicates the directory where the PBS script file is located and launched from.
  • The “email user” section within the PBS submission has been purposely removed – as you don’t want to send lots of emails each time you run submit multiple jobs.
  • The Parameter “#PBS -J 1-<range_number>:<step_size>” allows you to define the range and step size (if no step size if provided, a step size of 1 will be used) of the array of jobs you wish to execute.  The range_number and step_size must be integer values.  Some examples include:
    • #PBS -J 1-10 – the jobs submitted will include { 1 2 3 4 5 6 7 8 9 10 }.
    • #PBS -J 1-10:2 – the jobs submitted will include { 1 3 5 7 9 }.
    • #PBS -J 5-100:5 – the jobs submitted will include { 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 }.
  • You can change the number of times you would like the program to execute by simple changing the “range_number” with a value of your choice.
  • Within the program, the PBS parameter uses the "range_number" values and assigns a different $PBS_ARRAY_INDEX value for each HPC job.

 

Example 1 – Executing The Same Program Multiple Times

The example script file and executables can be found at /apps/samples/job-arrays/multi-job/ and can be copied to your own directory for use and modification.

  • The program executable hello.exe is a simple “Hello World” program and can be replaced with your own executable file.
  • In this example, the PBS variable “$PBS_JOBNAME” will be set to Multi-Job.[$1], where $1 is the “Job Array” number.  You may wish to change the jobname to something different, but note that PBS has a jobname limit of 10 characters.
  • In this example, the PBS variable “$PBS_ARRAY_INDEX” will be one of the following values { 1 2 3 4 5 6 7 8 9 10 }.

The details for this example script “multi-submit.pbs” can be found below:

######  Select resources #####
#PBS -N Multi-Job
#PBS -l ncpus=1
#PBS -l mem=1g
#PBS -J 1-10

job=$PBS_JOBNAME.$PBS_ARRAY_INDEX

##### Change Directory Execute Program #####
cd $PBS_O_WORKDIR
./hello.exe > ${PBS_JOBNAME}.${PBS_ARRAY_INDEX}.txt 2> ${PBS_JOBNAME}.${PBS_ARRAY_INDEX}.err

To submit the PBS job Array job, simply issue the command:

qsub multi-submit.pbs
Example 2 – Multiple Jobs With Multiple Variables

One of the common uses for the HPC Scheduler job arrays is the submission of multiple jobs in an effort to conduct a parameter sweep of different input variables.  Unfortunately there is a limitation within the use of “job arrays”, such that you cannot use non-integer values or have a multi-dimensional array list of jobs.  To overcome this limitations, you can create a simple “flat” input file which lists all the variables you wish to use.

In this example, the following variables and values are used

  • x = { 1 2 3 4 5 6 7 8 9 10 }
  • y = { 0 0.5 1 1.5 2.0 }
  • z = {3 7 9 15}

Using the above input variables, the “input.txt” file would look like:

1 0 3
1 0 7
1 0 9
1 0 15
1 0.5 3
1 0.5 7
1 0.5 9
1 0.5 15
.
.
.
10 2.0 3
10 2.0 7
10 2.0 9
10 2.0 15

A simple bash script could be developed to create this file quickly.

#!/bin/bash

for ((x=1; x<=10; x++))
do
  for y in 0 0.5 1 1.5 2.0
  do
    for z in 3 7 9 15
     do
      printf "%d %f %d\n" $x $y $z
    done
   done
done

An example where the three variables are use in a simple “bash” script can be seen below:

###### Select resources #####
#PBS -N Test
#PBS -l ncpus=1
#PBS -l mem=8g
#PBS -J 1-200

##### Change to current working directory

cd $PBS_O_WORKDIR

##### Obtain Parameters from input.txt file using $PBS_ARRAY_INDEX as the line number #####

parameters=`sed -n "${PBS_ARRAY_INDEX} p" input.txt`

parameterArray=($parameters)

x=${parameterArray[0]}
y=${parameterArray[1]}
z=${parameterArray[2]}

##### Execute Program #####

printf "Values used are %d %f %d\n" $x $y $z

Both the input.txt and multi-var.pbs files can be found at /apps/samples/job-arrays/multi-var/ .  These files can be copied to your own directory for use and modification.

To submit the PBS job Array job, simply issue the command:

qsub multi-var.pbs