Using and Administering

Interaction Between LoadLeveler and the Parallel API

This API does not give you access to any new LoadLeveler functions from Version 2 Release 1.0, or later releases.

Program applications which use the parallel APIs to interface with LoadLeveler are supported under a job type called parallel. When a user submits a job specifying the keyword job_type equal to parallel, the LoadLeveler API job control flow is as follows:

The negotiator selects nodes based on the resources you request. Once the nodes have been obtained, the negotiator contacts the schedd to start the job. The schedd marks the job pending and contacts the affected startds to start their starter processes.

One machine becomes the Master Starter. The Master Starter is one of the selected parallel nodes. After all starters are started and have completed inititialization, the Master Starter starts the executable specified in the job command file. The executable referred to as the Parallel Master uses this API to start tasks on remote nodes. A LOADLBATCH environment variable is set to YES so that the Parallel Master can distinguish between callers.

The Parallel Master must:

• Obtain the machine list through the ll_get_hostlist API.
• Start a task on all allocated machines through the ll_start_host API. It is mandatory that one and only one task be started on each machine. Each task is considered a Parallel Slave. Acquiring the task name, path and arguments is the responsibility of the Parallel Master. The user may pass this information through the arguments or environment keywords in the job command file.

When the Parallel Master starts, the job is marked Running. Once the Parallel Master and all tasks exit, the job is marked Complete.

Termination Paths

The Parallel Master is expected to cleanup and exit when:

• All of the Parallel Slaves have exited.
• A negative value is returned by either the ll_get_hostlist or ll_start_host subroutine.
• A SIGCONT, followed by a SIGTERM, is received. A possible reason for this is that LoadLeveler receives a job cancel request.

  The SIGTERM is also sent to all parallel tasks.

• A SIGCONT, followed by a SIGUSR1, is received. Reasons for this include:
  • The Parallel Master receives a VACATE or FLUSH request.
  • LoadLeveler receives a stop LoadLeveler daemons command.

  The SIGUSR1 is also sent to all parallel tasks.

A SIGKILL is issued to any process which does not exit within two minutes of receiving a termination signal.

Note that a SIGUSER1 indicates the job must terminate but will be restarted, while a SIGTERM indicates the job must terminate but will not be restarted.