Using and Administering

llstatus - Query Machine Status

Purpose

Returns status information about machines in the LoadLeveler cluster. It does not provide status on any NQS machine.

Syntax

llstatus [-?] [-H][-R][-F] [-v] [-l] [-f category_list] [-r category_list] [hostlist]

Flags

-?
Provides a short usage message.

-H
Provides extended help information.

-R
Lists all of the machine consumable resources associated with all of the machines in the LoadLeveler cluster (when specified alone). When a host list is specified, the option only displays machine consumable resources associated with the specified hosts. This option should not be used with any other option.

-F
Lists all of the floating consumable resources associated with the LoadLeveler cluster. This option should not be used with any other option.

-v
Outputs the name of the command, release number, service level, service level date, and operating system used to build the command.

-l
Specifies that a long listing be generated for each machine for which status is requested. If -l is not specified, the standard list, described below, is generated.

-f category_list
Is a blank-delimited list of categories you want to query. Each category you specify must be preceded by a percent sign. The category_list cannot contain duplicate entries. This flag allows you to create a customized version of the standard llstatus listing. The output fields produced by this flag all have a fixed length. The output is displayed in the order in which you specify the categories. category_list can be one or more of the following:
%a
Hardware architecture
%act
Number of jobs dispatched by the schedd on this machine
%cm
Custom Metric value
%cpu
Number of CPUs on this machine
%d
Available disk space in the LoadLeveler execute directory
%i
Number of seconds since last keyboard or mouse activity
%inq
Number of jobs in queue that were scheduled from this machine
%l
Berkeley one-minute load average
%m
Physical memory on this machine
%mt
Maximum number of tasks that can run simultaneously on this machine
%n
Machine name
%o
Operating system on this machine
%r
Number of jobs running on this machine
%sca
Availability of the schedd daemon
%scs
State of the schedd daemon
%sta
Availability of the startd daemon
%sts
State of the startd daemon
%v
Available swap space of this machine

-r category_list
Is a blank-delimited list of categories you want to query. Each category you specify must be preceded by a percent sign. The category_list cannot contain duplicate entries. This flag allows you to create a customized version of the standard llstatus listing. The output produced by this flag is considered raw, in that the fields can be variable in length. The output is displayed in the order in which you specify the formats. Output fields are separated by an exclamation point (!). category_list can be one or more of the categories listed under the -f flag.

hostlist
Is a blank-delimited list of machines for which status is requested.

Description

If no hostlist is specified, all machines are queried.

If you have more than a few machines configured for LoadLeveler, consider redirecting the output to a file when using the -l flag.

Each machine periodically updates the central manager with a snapshot of its situation. Since the information returned by using llstatus is a collection of such snapshots, all taken at varying times, the total picture may not be completely consistent.

Examples

This example requests a long status listing for machines named silver and gold:

llstatus -l silver gold

Results

In this section, the term "job step" refers to either a serial job step or a parallel task.

The Standard Listing: The standard listing is generated when you do not specify the -l option with the llstatus command. The following is sample output from the llstatus command, where there are two nodes in the cluster.

+--------------------------------------------------------------------------------+
|                                                                                |
|Name                      Schedd  InQ Act Startd Run LdAvg Idle Arch      OpSys |
|k10n09.ppd.pok.ibm.com    Avail     3   1 Run      1 2.72     0 R6000     AIX43 |
|k10n12.ppd.pok.ibm.com    Avail     0   0 Idle     0 0.00   365 R6000     AIX43 |
|                                                                                |
|R6000/AIX43            2 machines   3 jobs   1 running                          |
|Total Machines         2 machines   3 jobs   1 running                          |
|                                                                                |
|The Central Manager is defined on k10n09.ppd.pok.ibm.com                        |
|                                                                                |
|All machines on the machine_list are present.                                   |
+--------------------------------------------------------------------------------+

The standard listing includes the following fields:

Name
hostname of the machine.

Schedd
state of the schedd daemon, which can be one of the following:

Down
Drned (Drained)
Drning (Draining)
Avail (Available)

For a detailed explanation of these states, see The schedd Daemon.

InQ
number of job steps in the queue that were scheduled from this machine.

Act
number of job steps that the schedd has dispatched.

Startd
state of the startd daemon, which can be:

Busy
Down
Drned (Drained)
Drning (Draining)
Flush
Idle
None
Run (Running)
Suspnd (Suspend)

For a detailed explanation of these states, see The startd Daemon.

Run
The number of initiators used to run LoadLeveler jobs. One initiator is used for each serial job step. One initiator is used for each task of a parallel job step.

LdAvg
Berkeley one-minute load average on this machine.

Idle
The number of seconds since keyboard or mouse activity in a login session was detected. Highest number displayed is 9999.

Arch
The hardware architecture of the machine as listed in the configuration file.

OpSys
The operating system on this machine.

Consumable Resources Listing: The llstatus command, issued with the -R option, generates a listing of all of the consumable resources associated with all of the machines in the LoadLeveler cluster. When a host list is specified, this option will only display resources associated with the specified hosts. The following is sample output from this command:

   llstatus -R

+--------------------------------------------------------------------------------+
|                                                                                |
|Machine                        Consumable Resource(Available, Total)            |
|------------------------------ -------------------------------------------------|
|c163n11.ppd.pok.ibm.com        ConsumableCpus(2,4) resource_1(26,30)            |
|c163n12.ppd.pok.ibm.com        resource_1(10,15) res_2(15,24) spice2g6(13,13)   |
|ll6.pok.ibm.com                spice2g6(3,6) spice3f5(10,12)                    |
|ll7.pok.ibm.com                res_2(10,10) res_3(0,24) spice3f5(4,12)          |
|                                                                                |
|                                                                                |
+--------------------------------------------------------------------------------+

Floating Consumable Resources Listing: The llstatus command, issued with the -F option, generates a listing of all of the floating consumable resources associated with all of the machines in the LoadLeveler cluster. This option should not be specified with any other option. The following is sample output from this command:

   llstatus -F

+--------------------------------------------------------------------------------+
|                                                                                |
|Floating Resource              Available     Total                              |
|------------------------------ ------------- ---------------                    |
|EDA_licenses                   20            29                                 |
|Frame5                         15            20                                 |
|WorkBench6                     5             7                                  |
|XYZ_software                   6             6                                  |
|                                                                                |
|                                                                                |
+--------------------------------------------------------------------------------+

Customized, Formatted Standard Listing: A customized and formatted standard listing is generated when you specify llstatus with the -f option. The following is sample output from this command:

   llstatus -f %n %scs %inq %m %v %sts %l %o

+--------------------------------------------------------------------------------+
|Name             Schedd  InQ    Memory      FreeVMemory Startd  LdAvg  OpSys    |
|ll5.pok.ibm.com  Avail   0      128         22708       Run     0.23   AIX43    |
|ll6.pok.ibm.com  Avail   3      224         16732       Run     0.51   AIX43    |
|                                                                                |
|R6000/AIX43                 2 machines      3  jobs      3  running             |
|Total Machines              2 machines      3  jobs      3  running             |
|                                                                                |
|The Central Manager is defined on ll5.pok.ibm.com                               |
|                                                                                |
|All machines on the machine_list are present.                                   |
+--------------------------------------------------------------------------------+

Customized, Unformatted Standard Listing: A customized and unformatted (raw) standard listing is generated when you specify llstatus with the -r flag. Output fields are separated by an exclamation point (!). The following is sample output from this command:

llstatus -r %n %scs %inq %m %v %sts %l %o

+--------------------------------------------------------------------------------+
|ll5.pok.ibm.com!Avail!0!128!22688!Running!0.14!AIX43                            |
|ll6.pok.ibm.com!Avail!3!224!16668!Running!0.37!AIX43                            |
+--------------------------------------------------------------------------------+

The Long Listing: The long listing is generated when you specify the -l option with the llstatus command. Following the sample output is an explanation of all possible fields displayed by the llstatus command.

The following is sample output from the llstatus -l ll6 command:

+--------------------------------------------------------------------------------+
|================================================================================|
|Name                = ll6.pok.ibm.com                                           |
|Machine             = ll6.pok.ibm.com                                           |
|Arch                = R6000                                                     |
|OpSys               = AIX43                                                     |
|SYSPRIO             = (0 -  QDate)                                              |
|MACHPRIO            = (0 -  LoadAvg)                                            |
|VirtualMemory       = 16640                                                     |
|Disk                = 23000                                                     |
|KeyboardIdle        = 600                                                       |
|Tmp                 = 48868                                                     |
|LoadAvg             = 0.302991                                                  |
|ConfiguredClasses   = No_Class(2) osl(1) small(2) medium(1) POE(2)              |
|AvailableClasses    = No_Class(0) osl(1) small(2) medium(1) POE(2)              |
|DrainingClasses     =                                                           |
|DrainedClasses      =                                                           |
|Pool                = 1                                                         |
|Fabric Connectivity = 1                                                         |
|Adapter             = css0(switch,c166sn39.ppd.pok.ibm.com,9.114.72.167,38,4/4,8|
|                      READY) csss(striped,,,38,4/4,80M/80M,1,READY)             |
|                      en0(ethernet,c168n07.ppd.pok.ibm.com,9.114.72.103)        |
|Feature=                                                                        |
|Max_Starters        = 2                                                         |
|Memory              = 224                                                       |
|FreeRealMemory      = 83                                                        |
|PagesFreed          = 0                                                         |
|PagesScanned        = 0                                                         |
|PagesPagedIn        = 0                                                         |
|PagesPagedOut       = 0                                                         |
|ConsumableResources = ConsumableCpus(4,4) resA(26,26)                           |
|ConfigTimeStamp     = Wed Apr  8 09:05:36 1998                                  |
|Cpus                = 1                                                         |
|Speed               = 1.000000                                                  |
|Subnet              = 9.117.17                                                  |
|MasterMachPriority  = 0.000000                                                  |
|CustomMetric        = 1                                                         |
|StartdAvail         = 1                                                         |
|State               = Running                                                   |
|                                                                                |
+--------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------+
|EnteredCurrentState = Wed Apr  8 09:46:33 1998                                  |
|START               = T                                                         |
|SUSPEND             = F                                                         |
|CONTINUE            = T                                                         |
|VACATE              = F                                                         |
|KILL                = F                                                         |
|Machine Mode        = general                                                   |
|Running             = 2                                                         |
|ScheddAvail         = 1                                                         |
|ScheddState         = Avail                                                     |
|ScheddRunning       = 3                                                         |
|Pending             = 0                                                         |
|Starting            = 0                                                         |
|Idle                = 0                                                         |
|Unexpanded          = 0                                                         |
|Held                = 0                                                         |
|Removed             = 0                                                         |
|RemovedPending      = 0                                                         |
|Completed           = 0                                                         |
|TotalJobs           = 3                                                         |
|TimeStamp           = Wed Apr  8 09:47:45 1998                                  |
+--------------------------------------------------------------------------------+

The long listing includes these fields:

Name
hostname of the machine.

Running
The number of initiators used to run LoadLeveler jobs. One initiator is used for each serial job step. One initiator is used for each task of a parallel job step.

ScheddAvail
flag indicating if machine is running a schedd daemon (0=no, 1=yes).

StartdAvail
flag indicating if machine is running a startd daemon (0=no, 1=yes).

State
state of the startd daemon, which can be:

Busy
Down
Drain
Flush
Idle
None
Running
Suspend

For a detailed explanation of these states, see The startd Daemon.

OpSys
operating system on this machine.

Arch
hardware architecture of machine as listed in configuration file.

Machine
fully qualified name of the machine.

START
the expression, defined following C conventions in the configuration file, that evaluates to true or false (T/F). This determines whether jobs can be started on this machine.

SUSPEND
the expression, defined following C conventions in the configuration file, that evaluates to true or false (T/F). This determines whether running jobs should be suspended on this machine.

CONTINUE
the expression, defined following C conventions in the configuration file, that evaluates to true or false (T/F). This determines whether suspended jobs are continued on this machine.

VACATE
the expression, defined following C conventions in the configuration file, that evaluates to true or false (T/F). This determines whether suspended jobs are vacated on this machine.

KILL
the expression, defined following C conventions in the configuration file, that evaluates to true or false (T/F). This determines whether running jobs should be killed on this machine.

SYSPRIO
actual expression that determines overall system priority of the job, defined in the configuration file.

MACHPRIO
actual expression that determines machine priority, defined in the configuration file.

Machine Mode
the type of job this machine can run. This can be: batch, interactive, or general.

Virtual Memory
available swap space, in kilobytes, on this machine.

Entered Current State
date and time when machine state was set.

Disk
available space, in kilobytes (less 512KB) in LoadLeveler's execute directory on this machine.

Keyboard Idle
number of seconds since last keyboard or mouse activity.

LoadAvg
Berkely one-minute load average on machine.

AvailableClasses
set of currently available classes.

DrainingClasses
set of names of classes which are currently being drained on this machine.

DrainedClasses
set of names of classes which have been drained on this machine and are therefore unavailable.

ConfiguredClasses
set of all classes supported on this machine, both those in use and those not in use, as defined in the configuration file.

Pool
the identifier of the pool where this startd machine is located.

Adapter
Network adapter information associated with this machine. For a switch adapter, the format of this information is adapter_name(network_type, interface_name, interface_address, switch_node_number, available_adapter_windows/ total_adapter_windows, available_device_memory/total_device_memory, adapter_fabric_connectivity, adapter_state). For non-switch adapters, the format is adapter name (network_type, interface_name, interface_address).

Feature
set of all features on this machine.

Memory
physical memory, in megabytes, on this machine.

Max_Starters
maximum number of initiators that can be used simultaneously on this machine.

Config Time Stamp
date and time of last (re)configuration.

Cpus
number of CPUs on this machine.

Speed
speed associated with the machine.

MasterMachPriority
The machine priority for the parallel master node.

Subnet
The TCP/IP subnet that this machine resides on.

CustomMetric
The number that indicates the order of the machines for scheduling purposes.

ScheddRunning
The number of job steps submitted to this machine that are running somewhere in the LoadLeveler cluster.

Pending
The number of job steps in this state on this schedd machine.

Starting
The number of job steps in this state on this schedd machine.

Idle
The number of job steps in this state on this schedd machine.

Unexpanded
The number of job steps in this state on this schedd machine.

Held
The number of job steps in this state on this schedd machine.

Removed
The number of job steps in this state on this schedd machine.

Remove Pending
The number of job steps in this state on this schedd machine.

Completed
The number of job steps in this state on this schedd machine.

Total Jobs
The number of total job steps submitted to this schedd machine.

ScheddState
The state of the schedd on this schedd machine.

time stamp
The date and time the central manager last received a status update from this schedd machine.

FabricConnectivity
A boolean vector representing the current state of connectivity of this machine's switch adapter to the SP switch.

FreeRealMemory
Free real memory, in megabytes, on this machine. This value corresponds to the "fre" value of the vmstat command output, which is measured in page blocks.

PagesFreed
Pages freed per second. This value corresponds to the "fr" value of the vmstat command output.

PagesPaged In
Pages paged in from paging space per second. This value corresponds to the "pi" value of the vmstat command output.

PagesPagedOut
Pages paged out to paging space per second. This value corresponds to the "po" value of the vmstat command output.

PagesScanned
Pages scanned by the page-replacement algorithm per second. This value corresponds to the "sr" value of the vmstat command output.

ConsumableResources
Consumable resources associated with this machine. The format of this information is resource_name(available, total).


[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]