This chapter tells you how to submit jobs to LoadLeveler. In general, the information in this chapter applies both to serial jobs and to parallel jobs. For more specific information on parallel jobs, see Chapter 4. "Submitting and Managing Parallel Jobs"
Many LoadLeveler actions, such as submitting a job, can be done in either of the following ways:
Before you can submit a job or perform any other job related tasks, you need to build a job command file. A job command file describes the job you want to submit, and can include LoadLeveler keyword statements. For example, to specify a binary to be executed, you can use the executable keyword, which is described later in this section. To specify a shell script to be executed, the executable keyword can be used; if it is not used, LoadLeveler assumes that the job command file itself is the executable.
The job command file can include the following:
You can build a job command file either by using the Build a Job window on the GUI or by using a text editor.
The following general rules apply to job command files.
Figure 10 is an example of a simple serial job command file which is run from the current working directory. The job command file reads the input file, longjob.in1, from the current working directory and writes standard output and standard error files, longjob.out1 and longjob.err1, respectively, to the current working directory.
Figure 10. Serial Job Command File
# The name of this job command file is file.cmd. # The input file is longjob.in1 and the error file is # longjob.err1. The queue statement marks the end of # the job step. # # @ executable = longjob # @ input = longjob.in1 # @ output = longjob.out1 # @ error = longjob.err1 # @ queue |
To specify a stream of job steps, you need to list each job step in the job command file. You must specify one queue statement for each job step. Also, the executables for all job steps in the job command file must exist when you submit the job. All information in the first step is inherited by all succeeding steps.
LoadLeveler treats all job steps as independent job steps unless you use the dependency keyword. If you use the dependency keyword, LoadLeveler determines whether a job step should run based upon the exit status of the previously run job step.
For example, Figure 11 contains two separate job steps. Notice that step1 is the first job step to run and that step2 is a job step that runs only if step1 exits with the correct exit status.
Figure 11. Job Command File with Multiple Steps
# This job command file lists two job steps called "step1" # and "step2". "step2" only runs if "step1" completes # with exit status = 0. Each job step requires a new # queue statement. # # @ step_name = step1 # @ executable = executable1 # @ input = step1.in1 # @ output = step1.out1 # @ error = step2.err1 # @ queue # @ dependency = (step1 == 0) # @ step_name = step2 # @ executable = executable2 # @ input = step2.in1 # @ output = step2.out1 # @ error = step2.err1 # @ queue # |
In Figure 11, step1 is called the sustaining job step. step2 is called the dependent job step because whether or not it begins to run is dependent upon the exit status of step1. A single sustaining job step can have more than one dependent job steps and a dependent job step can also have job steps dependent upon it.
In Figure 11, each job step has its own executable, input, output, and error statements. Your job steps can have their own separate statements, or they can use those statements defined in a previous job step. For example, in Figure 12, step2 uses the executable statement defined in step1:
Figure 12. Job Command File with Multiple Steps and One Executable
# This job command file uses only one executable for # both job steps. # # @ step_name = step1 # @ executable = executable1 # @ input = step1.in1 # @ output = step1.out1 # @ error = step1.err1 # @ queue # @ dependency = (step1 == 0) # @ step_name = step2 # @ input = step2.in1 # @ output = step2.out1 # @ error = step2.err1 # @ queue |
See "Additional Job Command File Examples" for more information.
In addition to building job command files to submit serial jobs, you can also build job command files to submit parallel jobs. Before constructing parallel job command files, consult your LoadLeveler system administrator to see if your installation is configured for parallel batch job submission.
For more information on submitting parallel jobs, see Chapter 4. "Submitting and Managing Parallel Jobs"
After building a job command file, you can submit it for processing either to a machine in the LoadLeveler cluster or one outside of the cluster. (See "Querying Multiple LoadLeveler Clusters" for information on submitting a job to a machine outside the cluster.) You can submit a job command file either by using the GUI or the llsubmit command.
When you submit a job, LoadLeveler assigns the job a three part identifier and also sets environment variables for the job.
The identifier consists of the following:
LoadLeveler sets the following environment variables for the job:
For an example of submitting a job, see "Step 3: Submit a Job".
Submitting a Job Command File to be Routed to NQS Machines: When submitting a job command file to be routed to an NQS machine for processing, the job command file must contain the shell script to be submitted to the NQS node. NQS accepts only shell scripts; binaries are not allowed. All options in the command file pertaining to scheduling are used by LoadLeveler to schedule the job. When the job is dispatched to the node running the specified NQS class, the LoadLeveler options pertaining to the runtime environment are converted to NQS options and the job is submitted to the specified NQS queue. For more information on submitting jobs to NQS, see Figure 31 For more information on the llsubmit command, see llsubmit - Submit a Job.
Submitting a Job Command File Using a Submit-Only Machine: You can submit jobs from submit-only machines. Submit-only machines allow machines that do not run LoadLeveler daemons to submit jobs to the cluster. You can submit a job using either the submit-only version of the GUI or the llsubmit command.
To install submit-only LoadLeveler, follow the procedure in the LoadLeveler Installation Memo, or consult the appropriate README file.
In addition to allowing you to submit jobs, the submit-only feature allows you to cancel and query jobs from a submit-only machine.
This sections tells you how to edit a job command file, query the status of a job, place and release a hold on a job, cancel a job, change the priority of a job, checkpoint a step, and display machine status.
After you build a job command file, you can edit it using the editor of your choice. You may want to change the name of the executable or add or delete some statements.
Once you submit a job, you can query the status of the job to determine, for example, if it is still in the queue or if it is running. You also receive other job status related information such as the job ID and job owner. You can query the status of a LoadLeveler job either by using the GUI or the llq command. For an example of querying the status of a job, see "Step 4: Display the Status of a Job".
Querying the Status of a Job Running on an NQS Machine: If your job command file was routed to an NQS machine for processing, you can obtain its status by using either the GUI or the llq command. Keep in mind that a machine in the LoadLeveler cluster monitors the NQS machine where your job is running. The status you see on the GUI (or from llq) is generated by the machine in the LoadLeveler cluster. Since LoadLeveler only checks the NQS machine for status periodically, the status of the job on the NQS machine may change before LoadLeveler has an opportunity to update the GUI. If this happens, NQS will notify you, before LoadLeveler notifies you, regarding the status of the job.
Querying the Status of a Job Using a Submit-Only Machine: A submit-only machine, in addition to allowing you to submit and cancel jobs, allows you to query the status of jobs. You can query a job using either the submit-only version of the GUI or by using the llq command. For information on llq, see llq - Query Job Status.
This section applies only to those installations having more than one LoadLeveler cluster.
Using the LOADL_CONFIG environment variable, you can query, submit, or cancel jobs in multiple LoadLeveler clusters. The LOADL_CONFIG environment variable allows you to specify that the master configuration file be located in a directory other than the home directory of the loadl user ID. The file that LOADL_CONFIG points to must be in the /etc directory.
You need to set up your own master configuration file to point to the location of the LoadLeveler user ID, group ID, and configuration files. By default, the location of the master file is /etc/LoadL.cfg.
The following example explains how you can set up a machine to query multiple clusters:
You can configure /etc/LoadL.cfg to point to the "default" configuration files, and you can configure /etc/othercluster.cfg to point to the configuration files of another cluster which the user can select.
For example, you can enter the following query command:
$ llq
The above command uses the configuration from /etc/LoadL.cfg (this is determined by the LOADL_CONFIG environment variable). To send a query to the scheduler defined in the configuration file of /etc/othercluster.cfg, enter:
$ env LOADL_CONFIG=/etc/othercluster.cfg llq
Note that the machine from which you issue the llq command is considered as a submit-only machine by the other cluster.
You may place a hold on a job and thereby cause the job to remain in the queue until you release it.
There are two types of holds: a user hold and a system hold. Both you and your LoadLeveler administrator can place and release a user hold on a job. Only a LoadLeveler administrator, however, can place and release a system hold on a job.
You can place a hold on a job or release the hold either by using the GUI or the llhold command. For examples of holding and releasing jobs, see "Step 6: Hold a Job" and "Step 7: Release a Hold on a Job".
As a user or an administrator, you can also use the startdate keyword described in "startdate" to place a hold on a job. This keyword allows you to specify when you want to run a job.
You can cancel one of your jobs that is either running or waiting to run by using either the GUI or the llcancel command. You can use llcancel to cancel LoadLeveler jobs and jobs routed to NQS. Note that you can also cancel jobs from a submit-only machine.
Checkpointing is a method of periodically saving the state of a job so that, if for some reason, the job does not complete, it can be restarted from the saved state. For a detailed explanation of checkpointing, see "Step 13: Enable Checkpointing".
LoadLeveler uses the priority of a job to determine its position among a list of all jobs waiting to be dispatched. You can use the llprio command to change job priorities. See llprio - Change the User Priority of Submitted Job Steps for more information. This section discusses the different types of priorities and how LoadLeveler uses these priorities when considering jobs for dispatch.
Every job has a user priority associated with it. This priority, which can be specified by the user in the job command file, is a number between 0 and 100 inclusively. A job with a higher priority runs before a job with a lower priority (when both jobs are owned by the same user). The default user priority is 50. Note that this is not the UNIX nice priority.
Every job has a system priority associated with it. This priority is specified in LoadLeveler's configuration file using the SYSPRIO expression.
SYSPRIO is evaluated by LoadLeveler to determine the overall system priority of a job. A system priority value is assigned when the negotiator adds the new job to the queue of jobs eligible for dispatch.
The SYSPRIO expression can contain class, group, and user priorities, as shown in the following example:
SYSPRIO : (ClassSysprio * 100) + (UserSysprio * 10) + (GroupSysprio * 1) - (QDate)
For more information on the system priority expression, including all the variable you can use in this expression, see "Step 5: Prioritize the Queue Maintained by the Negotiator".
LoadLeveler schedules jobs based on the adjusted system priorty, which takes in account both system priority and user priority. Jobs with a higher adjusted system priority are scheduled ahead of jobs with a lower adjusted system priority. In determining which jobs to run first, LoadLeveler does the following:
For example, Table 3 represents the priorities assigned to jobs submitted by two users, Rich and
Joe. Two of the jobs belong to Joe, and three belong to Rich.
User Joe has two jobs (Joe1 and Joe2) in Class A with SYSPRIOs of 9 and 8
respectively. Since Joe2 has the higher user priority (20), and because
both of Joe's jobs are in the same class, Joe2's priority is swapped
with that of Joe1 when the adjusted system priority is calculated. This
results in Joe2 getting an adjusted system priority of 9, and Joe1 getting an
adjusted system priority of 8. Similarly, the Class A jobs belonging to
Rich (Rich1 and Rich3) also have their priorities swapped. The priority
of the job Rich2 does not change, since this job is in a different class
(Class B).
Table 3. How LoadLeveler Handles Job Priorities
| Job | User Priority | System Priority (SYSPRIO) | Class | Adjusted System Priority |
|---|---|---|---|---|
| Rich1 | 50 | 10 | A | 6 |
| Joe1 | 10 | 9 | A | 8 |
| Joe2 | 20 | 8 | A | 9 |
| Rich2 | 100 | 7 | B | 7 |
| Rich3 | 90 | 6 | A | 10 |
Throughout this book, the terms workstation, machine, and node refer to the machines in your cluster. See "Machines and Workstations" for information on the roles these machines can play.
You can perform the following types of tasks related to machines:
For an example of displaying machine status, see "Step 8: Display the Status of a Machine".
Usually, the system administrator is more concerned about the location of the central manager than the typical end user but you may also want to determine its location. One reason why you might want to locate the central manager is if you want to browse some configuration files that are stored on the same machine as the central manager.
Submit-only machines allow machines that are not part of the LoadLeveler cluster to submit jobs to the cluster for processing.
The section presents a series of simple tasks which a user might perform using commands. This section is meant for new users of LoadLeveler. More experienced users may want to continue on to "Additional Job Command File Examples".
Since you are not using the GUI, you have to build your job command file by using a text editor to create a script file. Into the file enter the name of the executable, other keywords designating such things as output locations for messages, and the necessary LoadLeveler statements, as shown in Figure 13:
Figure 13. Building a Job Command File
# This job command file is called longjob.cmd. The # executable is called longjob, the input file is longjob.in, # the output file is longjob.out, and the error file is # longjob.err. # # @ executable = longjob # @ input = longjob.in # @ output = longjob.out # @ error = longjob.err # @ queue |
You can optionally edit the job command file you created in step 1.
To submit the job command file that you created in step 1, use the llsubmit command:
llsubmit longjob.cmd
LoadLeveler responds by issuing a message similar to:
submit: The job "wizard.22" has been submitted.
where wizard is the name of the machine to which the job was submitted and 22 is the job identifier (ID). You may want to record the identifier for future use (although you can obtain this information later if necessary).
For more information on llsubmit, see llsubmit - Submit a Job
To display the status of the job you just submitted, use the llq command. This command returns information about all jobs in the LoadLeveler queue:
llq wizard.22
where wizard is the machine name to which you submitted the job, and 22 is the job ID. You can also query this job using the command llq wizard.22.0, where 0 is the step ID. For more information, see llq - Query Job Status.
You can change the user priority of a job that is in the queue or one that is running. This only affects jobs belonging to the same user and the same class. If you change the priority of a job in the queue, the job's priority increases or decreases in relation to your other jobs in the queue. If you change the priority of a job that is running, it does not affect the job while it is running. It only affects the job if the job re-enters the queue to be dispatched again. For more information, see "How Does a Job's Priority Affect Dispatching Order?".
To change the priority of a job, use the llprio command. To increase the priority of the job you submitted by a value of 10, enter:
llprio +10 wizard.22.0
For more information, see llprio - Change the User Priority of Submitted Job Steps.
To place a temporary hold on a job in a queue, use the llhold command. This command only takes effect if jobs are in the Idle or NotQueued state. To place a hold on wizard.22.0, enter:
llhold wizard.22.0
For more information, see llhold - Hold or Release a Submitted Job.
To release the hold you placed in step 6, use the llhold command:
llhold -r wizard.22.0
For more information, see llhold - Hold or Release a Submitted Job.
To display the status of the machine to which you submitted a job, use the llstatus command:
llstatus -l wizard
For more information, see llstatus - Query Machine Status.
To cancel wizard.22.0, use the llcancel command:
llcancel wizard.22.0
For more information, see llcancel - Cancel a Submitted Job.
Enter the llstatus command with the appropriate options to display the machine on which the central manager is running. For more information, see llstatus - Query Machine Status.
Public scheduling machines are those machines that participate in the scheduling of LoadLeveler jobs. The llstatus command can also be used to display the public scheduling machines.
"Serial Job Command File" gives you an example of a simple job command file. This section contains examples of building and submitting more complex job command files.
To run a program several times, varying the initial conditions each time, you could can multiple LoadLeveler scripts, each specifying a different input and output file as described in Figure 15. It would probably be more convenient to prepare different input files and submit the job only once, letting LoadLeveler generate the output files and do the multiple submissions for you.
Figure 14 illustrates the following:
Assume that you created five input files and each input file has different initial conditions for the program. The names of the input files are in the form longjob.in.x, where x is 0-4.
Submitting the LoadLeveler script shown in Figure 14 results in your program running five times, each time with a different input file. LoadLeveler generates the output file from the LoadLeveler job step IDs. This ensures that the results from the different submissions are not merged.
Figure 14. Job Command File with Varying input Statements
# @ executable = longjob # @ input = longjob.in.$(stepid) # @ output = longjob.out.$(jobid).$(stepid) # @ error = longjob.err.$(jobid).$(stepid) # @ queue # @ queue # @ queue # @ queue # @ queue |
To submit the job, type the command:
llsubmit longjob.cmd
LoadLeveler responds by issuing the following:
submit: The job "ll6.23" with 5 job steps has been submitted.
The following table shows you the standard input files, standard output
files, and standard error files for the five job steps:
| Job Step | Standard Input | Standard Output | Standard Error |
|---|---|---|---|
| ll6.23.0 | longjob.in.0 | longjob.out.23.0 | longjob.err.23.0 |
| ll6.23.1 | longjob.in.1 | longjob.out.23.1 | longjob.err.23.1 |
| ll6.23.2 | longjob.in.2 | longjob.out.23.2 | longjob.err.23.2 |
| ll6.23.3 | longjob.in.3 | longjob.out.23.3 | longjob.err.23.3 |
| ll6.23.4 | longjob.in.4 | longjob.out.23.4 | longjob.err.23.4 |
Figure 15 shows how you can use LoadLeveler variables in a job command file to assign different names to input and output files. This example assumes the following:
In Job Step 0:
In Job Step 1:
In Job Step 2:
Figure 15. Using LoadLeveler Variables in a Job Command File
# Job step 0 ============================================================
# The names of the output and error files created by this job step are:
#
# output: /u/rhclark/OSL/oslsslv_out/lltest1.122.0.out
# error : /u/rhclark/OSL/oslsslv_err/lltest1_122_0_err
#
# @ job_name = OSL
# @ step_name = step_0
# @ executable = oslsslv
# @ arguments = -maxmin=min -scale=yes -alg=dual
# @ environment = OSL_ENV1=20000; OSL_ENV2=500000
# @ requirements = (Arch == "R6000") && (OpSys == "AIX43")
# @ input = test01.mps.$(stepid)
# @ output = $(executable)_out/$(host).$(jobid).$(stepid).out
# @ error = $(executable)_err/$(host)_$(jobid)_$(stepid)_err
# @ queue
#
# Job step 1 ============================================================
# The names of the output and error files created by this job step are:
#
# output: /u/rhclark/OSL/oslsslv_out/lltest1.122.1.out
# error : /u/rhclark/OSL/oslsslv_err/lltest1_122_1_err
#
# @ step_name = step_1
# @ executable = ~rhclark/$(job_name)/oslsslv
# @ arguments = -maxmin=max -scale=no -alg=primal
# @ environment = OSL_ENV1=60000; OSL_ENV2=500000; \
OSL_ENV3=70000; OSL_ENV4=800000;
# @ input = ~rhclark/$(job_name)/test01.mps.$(stepid)
# @ output = ~rhclark/$(job_name)/$(base_executable)_out/$(hostname).$(cluster).$(process).out
# @ error = ~rhclark/$(job_name)/$(base_executable)_err/$(hostname)_$(cluster)_$(process)_err
# @ queue
#
# Job step 2 ============================================================
# The names of the output and error files created by this job step are:
#
# output: /u/rhclark/OSL/oslsslv_out/lltest1.122.2.out
# error : /u/rhclark/OSL/oslsslv_err/lltest1_122_2_err
#
# @ step_name = OSL
# @ dependency = (step_0 == 0) && (step_1 == 0)
# @ comment = oslsslv
# @ initialdir = /u/rhclark/$(step_name)
# @ arguments = -maxmin=min -scale=yes -alg=dual
# @ environment = OSL_ENV1=300000; OSL_ENV2=500000
# @ input = test01.mps.$(stepid)
# @ output = $(comment)_out/$(host).$(jobid).$(stepid).out
# @ error = $(comment)_err/$(host)_$(jobid)_$(stepid)_err
# @ queue
|
The name of the sample script shown in Figure 16 is run_spice_job. This script illustrates the following:
All file names that are not fully qualified are relative to the initial working directory /home/loadl/spice. LoadLeveler will send the job steps 0 and 1 of this job to a machine for that has a real memory of 64 MB or more for execution. Job step 2 most likely will be sent to a machine that has more that 128 MB of real memory and has the ESSL library installed since these preferences have been stated using the LoadLeveler preferences keyword. LoadLeveler will send job step 3 to the machine ll5.pok.ibm.com for execution because of the explicit requirement for this machine in the requirements statement.
Figure 16. Job Command File Used as the Executable
#!/bin/ksh
# @ job_name = spice_test
# @ account_no = 99999
# @ class = small
# @ arguments = spice3f5_input_$(stepid) spice3f5_output_$(stepid)
# @ input = spice2g6_input_$(stepid)
# @ output = $(job_name)_output_$(stepid)
# @ error = $(job_name)_error_$(stepid)
# @ initialdir = /home/loadl/spice
# @ requirements = ((Arch == "R6000") && (OpSys == "AIX43") && (Memory > 64))
# @ queue
# @ queue
# @ preferences = ((Memory > 128) && (Feature == "ESSL"))
# @ queue
# @ class = large
# @ arguments = spice3f5_input_benchmark1 spice3f5_output_benchmark1
# @ requirements = (Machine == "ll5.pok.ibm.com")
# @ input = spice2g6_input_benchmark1
# @ output = $(job_name)_output_benchmark1
# @ error = $(job_name)_error_benchmark1
# @ queue
OS_NAME=`uname`
case $OS_NAME in
AIX)
echo "Running $OS_NAME version of spice3f5" > $2
AIX_bin/spice3f5 < $1 >> $2 2>&1
echo "Running $OS_NAME version of spice2g6"
AIX_bin/spice2g6
;;
*)
echo "spice3f5 for $OS_NAME is not available" > $2
echo "spice2g6 for $OS_NAME is not available"
;;
esac
|
This section provides an alphabetical list of the keywords you can use in a LoadLeveler script. It also provides examples of statements that use these keywords. For most keywords, if you specify the keyword in a job step of a multi-step job, its value is inherited by all proceeding job steps. Exceptions to this are noted in the keyword description.
Supports centralized accounting. Allows you to specify an account number to associate with a job. This account number is stored with job resource information in local and global history files. It may also be validated before LoadLeveler allows a job to be submitted. For more information, see Chapter 7. "Gathering Job Accounting Data".
The syntax is:
account_no = string
where string is a text string that can consist of a combination of numbers and letters. For example, if the job accounting group charges for job time based upon the department to which you belong, your account number would be similar to:
account_no = dept34ca
Specifies the list of arguments to pass to your program when your job runs.
The syntax is:
arguments = arg1 arg2 ...
For example, if your job requires the numbers 5, 8, 9 as input, your arguments keyword would be similar to:
arguments = 5 8 9
Specifies whether you want to checkpoint your program.
The syntax is:
checkpoint = user_initiated | system_initiated | no
Specify user_initiated if you want to determine when the checkpoint is taken. User initiated checkpointing is available to both serial jobs and parallel POE jobs. (Checkpointing is not supported for parallel PVM jobs.) Serial jobs must use the LoadLeveler ckpt API call to request user initiated checkpointing. See "Serial Checkpointing API" for more information. POE jobs must use the Parallel Environment (PE) parallel checkpointing API to enable user initiated checkpointing. See IBM Parallel Environment for AIX: Operation and Use, Volume 1 for more information.
Specify system_initiated if you want LoadLeveler to automatically checkpoint your program at preset intervals. System initiated checkpointing is available only to serial jobs. To cause both user initiated and system initiated checkpoints to occur, specify system_initiated and have your program use the appropriate ckpt API call.
Specify no if you do not want your program to be checkpointed. This is the default.
To restart a program for which a checkpoint file exists, you must set the CHKPT_STATE environment variable to restart. For more information on environment variables associated with checkpointing, see "Set the Appropriate Environment Variables". For information on setting environment variables for a job, see "environment". Note that it is not necessary to set the restart job command language keyword for a checkpointing job. For more information, see "restart".
To use checkpointing, your program must be linked with the appropriate LoadLeveler libraries. See "Ensure all User's Jobs are Linked to Checkpointing Libraries" for more information. For more detailed information on checkpointing, see "Step 13: Enable Checkpointing".
Specifies the name of a job class defined locally in your cluster. If not specified, the default job class is assigned. You can use the llclass command to find out information on job classes.
The syntax is:
class = name
For example, if you are allowed to submit jobs belonging to a class called "largejobs", your class keyword would look like the following:
class = largejobs
Specifies text describing characteristics or distinguishing features of the job.
Specifies the hard limit and/or soft limit for the size of a core file. This is a per process limit.
The syntax is:
core_limit = hardlimit,softlimit
Some examples are:
core_limit = 125621,10kb core_limit = 5621kb,5000k core_limit = 2mb,1.5mb core_limit = 2.5mw core_limit = unlimited core_limit = rlim_infinity core_limit = copy
See "Limit Keywords" for more information on the values and units you can use with this keyword.
Specifies the hard limit and/or soft limit for the amount of CPU time that a submitted job step can use. This is a per process limit.
The syntax is:
cpu_limit = hardlimit,softlimit
For example:
cpu_limit = 12:56:21,12:50:00 cpu_limit = 56:21.5 cpu_limit = 1:03,21 cpu_limit = unlimited cpu_limit = rlim_infinity cpu_limit = copy
See "Limit Keywords" for more information on the values and units you can use with this keyword.
Specifies the hard limit and/or soft limit for the size of the data segment to be used by the job step. This is a per process limit.
The syntax is:
data_limit = hardlimit,softlimit
For example:
data_limit = ,125621 data_limit = 5621kb data_limit = 2mb data_limit = 2.5mw,2mb
See "Limit Keywords" for more information on the values and units you can use with this keyword.
Specifies the dependencies between job steps. A job dependency, if used in a given job step, must be explicitly specified for that step.
The syntax is:
dependency = expression
where the syntax for the expression is:
step_name operator value
where step_name (as described in "step_name") must be a previously defined job step and operator can be one of the following:
The value is usually a number which specifies the job return code to which the step_name is set. It can also be one of the following LoadLeveler defined job step return codes:
Examples: The following are examples of dependency statements:
Example 1: In the following example, the step that contains this dependency statement will run if the return code from step 1 is zero:
dependency = (step1 == 0)
Example 2: In the following example, step1 will run with the executable called myprogram1. Step2 will run only if LoadLeveler removes step1 from the system. If step2 does run, the executable called myprogram2 gets run.
# Beginning of step1 # @ step_name = step1 # @ executable = myprogram1 # @ ... # @ queue # Beginning of step2 # @ step_name = step2 # @ dependency = step1 == CC_REMOVED # @ executable = myprogram2 # @ ... # @ queue
Example 3: In the following example, step1 will run with the executable called myprogram1. Step2 will run if the return code of step1 equals zero. If the return code of step1 does not equal zero, step2 does not get executed. If step2 is not run, the dependency statement in step3 gets evaluated and it is determined that step2 did not run. Therefore, myprogram3 gets executed.
# Beginning of step1 # @ step_name = step1 # @ executable = myprogram1 # @ ... # @ queue # Beginning of step2 # @ step_name = step2 # @ dependency = step1 == 0 # @ executable = myprogram2 # @ ... # @ queue # Beginning of step3 # @ step_name = step3 # @ dependency = step2 == CC_NOTRUN # @ executable = myprogram3 # @ ... # @ queue
Example 4: In the following example, the step that contains step2 returns a non-negative value if successful. This step should take into account the fact that LoadLeveler uses a value of 1001 for CC_REMOVED and 1002 for CC_NOTRUN. This is done with the following dependency statement:
dependency = (step2 >= 0) && (step2 < CC_REMOVED)
Specifies your initial environment variables when your job step starts. Separate environment specifications with semicolons. An environment specification may be one of the following:
The syntax is:
environment = env1 ; env2 ; ...
For example:
environment = COPY_ALL; !env2;
Specifies the name of the file to use as standard error (stderr) when your job step runs. If you do no specify this keyword, the file /dev/null is used.
The syntax is:
error = filename
For example:
error = $(jobid).$(stepid).err
For serial jobs, executable identifies the name of the program to run. The program can be a shell script or a binary. For parallel jobs, executable can be a shell script or the following:
If you do not include this keyword and the job command file is a shell script, LoadLeveler uses the script file as the executable.
The syntax is:
executable = name
Examples:
Note that the executable statement automatically sets the $(base_executable) variable, which is the file name of the executable without the directory component. See Figure 15 for an example of using the $(base_executable) variable.
Specifies the hard limit and/or soft limit for the size of a file. This is a per process limit.
The syntax is:
file_limit = hardlimit,softlimit
For example:
file_limit = 120mb,100mb
See "Limit Keywords" for more information on the values and units you can use with this keyword.
Specifies the LoadLeveler group. If not specified, this defaults to the default group. The syntax is:
group = group_name
For example:
group = my_group_name
Specifies whether you want to place a hold on your job step when you submit it. There are three types of holds:
The syntax is:
hold = user|system|usersys
For example, to put a user hold on a job, the keyword statement would be:
hold = user
To remove the hold on the job, you can use either the GUI or the llhold -r command.
Maximum virtual image size, in kilobytes, to which your program will grow during execution. LoadLeveler tries to execute your job steps on a machine that has enough resources to support executing and checkpointing your job step. If your job command file has multiple job steps, the job steps will not necessarily run on the same machine, unless you explicitly request that they do.
If you do not specify the image size of your job command file, the image size is that of the executable. If you underestimate the image size of your job step, your job step may crash due to the inability to acquire more address space. If you overestimate the image size, LoadLeveler may have difficulty finding machines that have the required resources.
The syntax is:
image_size = number
For example, to set an image size of 11 KB, the keyword statement would be:
image_size = 11
The path name of the directory to use as the initial working directory during execution of the job step. If none is specified, the initial directory is the current working directory at the time you submitted the job. File names mentioned in the command file which do not begin with a / are relative to the initial directory. The initial directory must exist on the submitting machine as well as on the machine where the job runs.
The syntax is:
initialdir = pathname
For example:
initialdir = /var/home/mike/ll_work
Specifies the name of the file to use as standard input (stdin) when your job step runs. If not specified, the file /dev/null is used.
The syntax is:
input = filename
For example:
input = input.$(process)
Specifies the hard limit and/or soft limit for the CPU time to be used by all processes of a job step. For example, if a job step forks to produce multiple processes, the sum total of CPU consumed by all of the processes is added and controlled by this limit.
The syntax is:
job_cpu_limit = hardlimit,softlimit
For example:
job_cpu_limit = 12:56,12:50
See "Limit Keywords" for more information on the values and units you can use with this keyword.
Specifies the name of the job. This keyword must be specified in the first job step. If it is specified in other job steps in the job command file, it is ignored. You can name the job using any combination of letters and/or numbers.
The syntax is:
job_name = job_name
For example:
job_name = my_first_job
The job_name only appears in the long reports of the llq, llstatus, and llsummary commands, and in mail related to the job
Specifies the type of job step to process. Valid entries are:
Note that when you specify job_type=pvm3 or job_type=serial, you cannot specify the following keywords: node, tasks_per_node, total_tasks, network.LAPI, and network.MPI.
The syntax is:
job_type = string
For example:
job_type = pvm3
Specifies the maximum number of nodes requested for a parallel job, regardless of the number of processors contained in the node.
This keyword is equivalent to the maximum value you specify on the new node keyword. In any new job command files you create for non-PVM jobs, you should use the node keyword to request nodes/processors. The max_processors keyword should be used by existing job command files and new PVM job command files. Note that if you specify in a job command file both the max_processors keyword and the node keyword, the job is not submitted.
The syntax is:
max_processors = number
For example:
max_processors = 6
Specifies the minimum number of nodes requested for a parallel job, regardless of the number of processors contained in the node.
This keyword is equivalent to the minimum value you specify on the new node keyword. In any new job command files you create for non-PVM jobs, you should use the node keyword to request nodes/processors. The min_processors keyword should be used by existing job command files and new PVM job command files. Note that if you specify in a job command file both the min_processors keyword and the node keyword, the job is not submitted.
The syntax is:
min_processors = number
For example:
min_processors = 4
Specifies communication protocols, adapters, and their characteristics. You need to specify this keyword when you want a task of a parallel job step to request a specific adapter that is defined in the LoadLeveler administration file. You do not need to specify this keyword when you want a task to access a shared, default adapter via TCP/IP. (A default adapter is an adapter whose name matches a machine stanza name.)
Note that you cannot specify both the network statement and the Adapter requirement (or the Adapter preference) in a job command file. Also, the value of the network keyword applies only to the job step in which you specify the keyword. (That is, this keyword in not inherited by other job steps.)
The syntax is:
network.protocol = network_type[,usage][,mode]
Where:
Example 1: To use the MPI protocol with an SP switch adapter in User Space mode without sharing the adapter, enter the following:
network.MPI = css0,not_shared,US
Example 2: To use the MPI protocol with a shared SP switch adapter in IP mode, enter the following:
network.MPI = css0,IP
Because a shared adapter is the default, you do not need to specify shared.
Note that LoadLeveler can ensure that an adapter is dedicated (not shared) if you request the adapter in US mode, since any user who requests a user space adapter must do so using the network statement. However, if you request a dedicated adapter in IP mode, the adapter will only be dedicated if all other LoadLeveler users who request this adapter do so using the network statement.
Specifies the minimum and maximum number of nodes requested by a job step. You must specify at least one of these values. The value of the node keyword applies only to the job step in which you specify the keyword. (That is, this keyword in not inherited by other job steps.)
The syntax is:
node = [min][,max]
Where:
For example, to specify a range of six to twelve nodes, enter the following:
node = 6,12
To specify a maximum of seventeen nodes, enter the following:
node = ,17
When you use the node keyword together with the total_tasks keyword, the min and max values you specify on the node keyword must be equal, or you must specify only one value. For example:
node = 6
total_tasks = 12
For information of specifying the number of tasks you want to run on a node, see "tasks_per_node" and "total_tasks".
Specifies whether this job step shares nodes with other job steps.
The syntax is:
node_usage = shared | not_shared
Where:
Specifies when the user specified in the notify_user keyword is sent mail. The syntax is:
notification = always|error|start|never|complete
Where:
For example, if you want to be notified with mail only when your job step completes, your notification keyword would be:
notification = complete
When a LoadLeveler job ends, you may receive UNIX mail notification indicating the job exit status. For example, you could get the following mail message:
Your LoadLeveler job myjob1 exited with status 4.
The return code 4 is from the user's job. LoadLeveler retrieves the return code and returns it in the mail message, but it is not a LoadLeveler return code.
Specifies the user to whom mail is sent based on the notification keyword. The default is the submitting user and the submitting machine.
The syntax is:
notify_user = userID
For example, if you are the job step owner but you want a co-worker whose name and user ID is bob, to receive mail regarding the job step, your notify keyword would be:
notify_user = bob
Specifies the name of the file to use as standard output (stdout) when your job step runs. If not specified, the file /dev/null is used.
The syntax is:
output = filename
For example:
output = out.$(jobid)
Specifies the path that should be used when starting a PVM 3.3 slave process. This is used for PVM 3.3 only and is translated into the ep keyword as defined in the PVM 3.3 hosts file.
For example:
parallel_path = /home/userid/cmds/pvm3/$PVM_ARCH:$PVM_ROOT/lib/$PVM_ARCH
The parallel_path statement above has two components, separated by a colon. The first component points to the location of the user's programs. The second component points to the location of the pvmgs routine - required if the job uses PVM 3.3 group support - assuming PVM 3.3 is installed "normally". Note that your installation must install PVM 3.3 to include group support in order for you to use group support within LoadLeveler. $PVM_ARCH will be replaced by the architecture of the machine, as defined by PVM 3.3. This will specify the path to be searched for executables when the user's job issues a pvm_spawn() command.
$PVM_ARCH, and $PVM_ROOT are PVM environment variables. For more information, see the appropriate PVM 3.3 documentation.
Specifies the characteristics that you prefer be available on the machine that executes the job steps. LoadLeveler attempts to run the job steps on machines that meet your preferences. If such a machine is not available, LoadLeveler will then assign machines which meet only your requirements.
The values you can specify in a preferences statement are the same values you can specify in a requirements statement, with the exception of the Adapter requirement. See "requirements" for more information.
The syntax is:
preferences = Boolean_expression
Some examples are::
preferences = (Memory <=16) && (Arch == "R6000") preferences = Memory >= 64
Places one copy of the job step in the queue. This statement is required. The queue statement essentially marks the end of the job step. Note that you can specify statements between queue statements.
The syntax is:
queue
Specifies the requirements which a machine in the LoadLeveler cluster must meet to execute any job steps. You can specify multiple requirements on a single requirements statement.
The syntax is:
requirements = Boolean_expression
When strings are used as part of a Boolean expression that must be enclosed in double quotes. Sample requirement statements are included following the descriptions of the supported requirements.
The requirements supported are:
The pre-defined network types are:
Note that LoadLeveler converts the above network types to the network statement. For more information, see "Migrating Your Existing Adapter Requirements Statements".
Example 1: To specify a memory requirement and a machine architecture requirement, enter:
requirements = (Memory >=16) && (Arch == "R6000")
Example 2: To specify that your job requires multiple machines for a parallel job, enter:
requirements = (Machine == { "ll6" "ll5" "ll0" })
Example 3: You can set a machine equal to a job step name. This means that you want the job step to run on the same machine on which the previous job step ran. For example::
requirements = (Machine == machine.step_name)
where step_name is a step name previously defined in the job command file. The use of Machine == machine.step_name is limited to serial jobs.
For example:
# @ step_name = step1 # @ executable = c1 # @ output = $(executable).$(jobid).$(step_name).out # @ queue # @ step_name = step2 # @ dependency = (step1 == 0) # @ requirements = (Machine == machine.step1) # @ executable = c2 # @ output = $(executable).$(jobid).$(step_name).out # @ queue
Example 4: To specify a requirement for an SP switch adapter in IP mode, enter:
requirements = (Adapter == "hps_ip")
Example 5: To specify a requirement for a specific pool number, enter:
requirements = (Pool == 7)
Example 6: To specify a requirement that the job runs on LoadLeveler Version 2 Release 1 or any follow-on release, enter:
requirements = (LL_Version >= "2.1")
Note that the statement requirements = (LL_Version == "2.1") matches only the value 2.1.0.0.
Specifies whether LoadLeveler considers a job "restartable." The syntax is:
restart = yes|no
If restart=yes, which is the default, and the job is vacated from its executing machine before completing, the central manager requeues the job. It can start running again when a machine on which it can run becomes available. If restart=no, a vacated job is cancelled rather than requeued.
Note that this keyword is different from the restart state associated with checkpointing jobs. This state tells LoadLeveler to restart a job from an existing checkpoint file. (Checkpoint jobs are always considered "restartable.") For more information, see "Set the Appropriate Environment Variables".
Specifies the hard limit and/or soft limit for the resident set size.
The syntax is:
rss_limit = hardlimit,softlimit
For example:
rss_limit=120,100
Because no units are specified in the above example, bytes are assumed. See "Limit Keywords" for more information on the values and units you can use with this keyword.
Specifies the name of the shell to use for the job step. If not specified, the shell used in the owner's password file entry is used. If none is specified, the /bin/sh is used.
The syntax is:
shell = name
For example, if you wanted to use the Korn shell, the shell keyword would be:
shell = /bin/ksh
Specifies the hard limit and/or soft limit for the size of the stack that is created.
The syntax is:
stack_limit = hardlimit,softlimit
For example:
stack_limit = 120,100
Because no units are specified in the above example, bytes are assumed. See "Limit Keywords" for more information on the values and units you can use with this keyword.
Specifies when you want to run the job step. If not specified, the current date and time are used.
The syntax is:
startdate = date time
date is expressed as MM/DD/YY, and time is expressed as HH:mm(:ss).
For example, if you want the job to run on August 28th, 1999 at 1:30 PM, issue:
startdate = 08/28/99 13:30
If you specify a start date that is in the future, your job is kept in the Deferred state until that start date.
Specifies the name of the job step. You can name the job step using any combination of letters, numbers, underscores (_) and periods (.). You cannot, however, name it T or F, or use a number in the first position of the step name. The step name you use must be unique and can be used only once. If you don't specify a step name, by default the first job step is named the character string "0", the second is named the character string "1", and so on.
The syntax is:
step_name = step_name
For example:
step_name = step_3
Specifies the number of tasks of a parallel job you want to run per node. Use this keyword in conjunction with the node keyword. The value you specify on the node keyword can be a range or a single value.
The maximum number of tasks a job step can request is limited by the total_tasks keyword in the administration file (provided this keyword is specified). That is, the maximum must be less than any total_tasks value specified in a user, group, or class stanza.
The value of the tasks_per_node keyword applies only to the job step in which you specify the keyword. (That is, this keyword is not inherited by other job steps.)
Also, you cannot specify both the tasks_per_node keyword and the total_tasks keyword within a job step.
The syntax is:
tasks_per_node = number
Where number is the number of tasks you want to run per node. The default is one task per node.
For example, to specify a range of seven to 14 nodes, with four tasks running on each node, enter the following:
node = 7,14
tasks_per_node = 4
The above job step runs 28 to 56 tasks, depending on the number of nodes allocated to the job step.
Specifies the total number of tasks of a parallel job you want to run on all available nodes. Use this keyword in conjunction with the node keyword. The value you specify on the node keyword must be a single value rather than a range of values.
The maximum number of tasks a job step can request is limited by the total_tasks keyword in the administration file (provided this keyword is specified). That is, the maximum must be less than any total_tasks value specified in a user, group, or class stanza.
The value of the total_tasks keyword applies only to the job step in which you specify the keyword. (That is, this keyword is not inherited by other job steps.)
Also, you cannot specify both the total_tasks keyword and the tasks_per_node keyword within a job step.
The syntax is:
total_tasks = number
Where number is the total number of tasks you want to run.
For example, to run two tasks on each of 12 available nodes for a total of 24 tasks, enter the following:
node = 12
total_tasks = 24
If you specify an unequal distribution of tasks per node, LoadLeveler allocates the tasks on the nodes in a round-robin fashion. For example, if you have three nodes and five tasks, two tasks run on the first two nodes and one task runs on the third node.
Sets the initial priority of your job step. Priority only affects your job steps. It orders job steps you submitted with respect to other job steps submitted by you, not with respect to job steps submitted by other users.
The syntax is:
user_priority = number
where number is a number between 0 and 100, inclusive. A higher number indicates the job step will be selected before a job step with a lower number. The default priority is 50. Note that this is not the UNIX nice priority.
This priority guarantees the order the jobs are considered for dispatch. It does not guarantee the order in which they will run.
Sets the hard limit and/or soft limit for the elapsed time for which a job can run. In computing the elapsed time for a job, LoadLeveler considers the start time to be the time the job is dispatched.
If you are running the LoadLeveler Backfill scheduler, either users must set a wall clock limit in their job command file or the administrator must define a wall clock limit value for the class to which a job is assigned. In most cases, this wall clock limit value should not be unlimited. For more information, see "Choosing a Scheduler".
The syntax is:
wall_clock_limit = hardlimit,softlimit
An example is:
wall_clock_limit = 5:00,4:30
See "Limit Keywords" for more information on the values and units you can use with this keyword.
LoadLeveler has several variables you can use in a job command file. These variables are useful for distinguishing between output and error files.
You can refer to variables in mixed case, but you must specify them using the following syntax:
$(variable_name)
The following variables are available to you:
In addition, the following keywords are also available as variables. However, you must define them in the job command file. These keywords are described in detail in "Job Command File Keywords".
Note that for the $(comment) variable, the keyword definition must be a single string with no blanks. Also, the executable statement automatically sets the $(base_executable) variable, which is the file name of the executable without the directory component. See Figure 15 for an example of using the $(base_executable) variable.
The following job command file creates an output file called stance.78.out, where stance is the host and 78 is the jobid.
# @ executable = my_job # @ arguments = 5 # @ output = $(host).$(jobid).out # @ queue
The following job command file creates an output file called computel.step1.March05.
# @ comment = March05 # @ job_name = computel # @ step_name = step1 # @ executable = my_job # @ output = $(job_name).$(step_name).$(comment) # @ queue