Your organization may have a policy of charging users or groups of users for the amount of resources that their jobs consume. You can do this using LoadLeveler's accounting feature. Using this feature, you can produce accounting reports that contain job resource information for completed serial and parallel jobs. You can also view job resource information on jobs that are continuing to run.
Information on completed serial and parallel jobs is gathered using the UNIX wait3 system call. Information on non-completed serial and parallel jobs is gathered in a platform-dependent manner by examining data from the UNIX process.
Accounting information on a completed serial job is determined by accumulating resources consumed by that job on the machine(s) that ran the job. Similarly, accounting information on completed parallel jobs is gathered by accumulating resources used on all of the nodes that ran the job.
You can also view resource consumption information on serial and parallel jobs that are still running by specifying the -x option of the llq command. In order to enable llq -x, you should specify the following keywords in the configuration file:
LoadLeveler can collect job resource usage information for every machine on which a job may run. A job may run on more than one machine because it is a parallel job or because the job is vacated from one machine and rescheduled to another machine.
To enable recording of resources by machine, you need to specify ACCT = A_ON A_DETAIL in the configuration file.
The machine's speed is part of the data collected. With this information, an installation can develop a charge back program which can charge more or less for resources consumed by a job on different machines. For more information on a machine's speed, refer to the machine stanza information. See "Step 1: Specify Machine Stanzas".
In addition to collecting job resource information based upon machines used, you can gather this information based upon an event or time that you specify. For example, you may want to collect accounting information at the end of every work shift or at the end of every week or month. To collect accounting information on all machines in this manner, use the llctl command with the capture parameter:
llctl -g capture eventname
eventname is any string of continuous characters (no white space is allowed) that defines the event about which you are collecting accounting data. For example, if you were collecting accounting data on the graveyard work shift, your command could be:
llctl -g capture graveyard
This command allows you to obtain a snapshot of the resources consumed by active jobs up to and including the moment when you issued the command. If you want to capture this type of information on a regular basis, you can set up a crontab entry to invoke this command regularly. For example:
# sample crontab for accounting # shift crontab 94/8/5 # # Set up three shifts, first, second, and graveyard shift. # Crontab entries indicate the end of shift. # #M H d m day command # 00 08 * * * /u/loadl/bin/llctl -g capture graveyard 00 16 * * * /u/loadl/bin/llctl -g capture first 00 00 * * * /u/loadl/bin/llctl -g capture second
For more information on the llctl command, refer to llctl - Control LoadLeveler Daemons. For more information on the collection of accounting records, see llq - Query Job Status.
If your installation is interested in keeping track of resources used on an account basis, you can require all users to specify an account number in their job command files. They can specify this account number with the account_no keyword which is explained in detail in "Job Command File Keywords".
LoadLeveler validates this account number by comparing it against a list of account numbers specified for the user in the user stanza in the administration file.
Account validation is under the control of the ACCT keyword in the configuration file. The routine which performs the validation is called llacctval. You can supply your own validation routine by specifying the ACCT_VALIDATION keyword in the configuration file. The following are passed as character string arguments to the validation routine:
The account validation routine must exit with a return code of zero if the validation succeeds. If it fails, the return code is a non-zero number.
LoadLeveler stores the accounting information that it collects in a file called history in the spool directory of the machine that initially scheduled this job, the schedd machine. Data on parallel jobs are also stored in the history files.
Resource information collected on the LoadLeveler job is constrained by the capabilities of the wait3 system call. Information for processes which fork child processes will include data for those child processes as long as the parent process waits for the child process to terminate. Complete data may not be collected for jobs which are not composed of simple parent/child processes. For example, if you have a LoadLeveler job which invokes an rsh command to execute a function on another machine, the resources consumed on the other machine will not be collected as part of the LoadLeveler accounting data.
LoadLeveler accounting uses the following types of files:
You can produce three types of reports using either the local or global history file. These reports are called the short, long, and extended versions. As their names imply, the short version of the report is a brief listing of the resources used by LoadLeveler jobs. The long version provides more comprehensive detail with summarized resource usage and the extended version of the report provides the comprehensive detail with detailed resource usage. If you do not specify a report type, you will receive the default short version.
The short report displays the number of jobs along with the total CPU usage according to user, class, group, and account number. The extended version of the report displays all of the data collected for every job. See the llsummary command, llsummary - Return Job Resource Information for Accounting, for examples of the short and extended versions of the report.
For information on the accounting Application Programming Interfaces, refer to Chapter 11. "LoadLeveler APIs".
The following sample scenario walks you through the process of collecting account data. You can perform all of the steps or just the ones that apply to your situation.
Edit the configuration file according to the following table:
| Edit this keyword: | To: | ||
|---|---|---|---|
| GLOBAL_HISTORY | Specify a directory in which to place the global history files. | ||
| ACCT | Turn accounting and account validation on and off and specify detailed accounting. | ||
| ACCT_VALIDATION | Specify the account validation routine. | ||
| |||
You can accomplish this step using either the llacctmrg command or the graphical user interface:
&TRIANGLE. A window appears prompting you to enter a directory name where the file will be placed. If no directory is specified, the directory specified with the GLOBAL_HISTORY keyword in the global configuration file is the default directory.
&TRIANGLE. The window closes and you return to the main window.
You can accomplish this step using either the llsummary command or the graphical user interface:
Note: If you want to receive an extended accounting report, select the extended cascading button.
&TRIANGLE. A window appears prompting you to enter the following information:
&TRIANGLE. The window closes and you return to the main window. The report appears in the Messages window if no output data file was specified.
&TRIANGLE. The Build a Job window appears.
&TRIANGLE. The window closes and you return to the main window.
To specify weights to associate with machines, specify the following keyword in a machine's machine stanza in the administration file:
Also, if you have in your cluster machines of differing speeds and you want LoadLeveler accounting information to be normalized for these differences, specify cpu_speed_scale=true in each machine's respective machine stanza.
For example, suppose you have a cluster of two machines, called A and B, where Machine B is three times as fast as Machine A. Machine A has speed=1.0, and Machine B has speed=3.0. Suppose a job runs for 12 CPU seconds on Machine A. The same job runs for 4 CPU seconds on Machine B. When you specify cpu_speed_scale=true, the accounting information collected on Machine B for that job shows the normalized value of 12 CPU seconds rather than the actual 4 CPU seconds.