IBM PE for AIX V2R4.0: Operation and Use, Vol. 1

Appendix B. POE Environment Variables and Command-Line Flags

This appendix contains tables which summarize the environment variables and command-line flags discussed throughout this book. You can set these variables and flags to influence the execution of parallel programs, and the operation of certain tools. The command-line flags temporarily override their associated environment variable. The tables divide the environment variables and flags by function.

Table 8 summarizes the environment variables and flags for controlling the Partition Manager. These environment variables and flags enable you to specify such things as an input or output host list file, and the method of node allocation. For a complete description of the variables and flags summarized in this table, see "Executing Parallel Programs".
Table 9 summarizes the environment variables and flags for Job Specifications. These environment variables and flags determine whether or not the Partition Manager should maintain the partition for multiple job steps, whether commands should be read from a file or STDIN, and how the partition should be loaded. For a complete description of the variables and flags summarized in this table, see "Executing Parallel Programs".
Table 10 summarizes the environment variables and flags for determining how I/O from the parallel tasks should be handled. These environment variables and flags set the input and output modes, and determine whether or not output is labeled by task id. For a complete description of the variables and flags summarized in this table, see "Managing Standard Input, Output, and Error".
Table 11 summarizes the environment variables and flags controlling VT trace collection. These environment variables and flags determine such things as which types of traces are collected, and how trace storage is handled. For a complete description of the variables and flags summarized in this table, see IBM Parallel Environment for AIX: Operation and Use, Volume 2, Tools Reference
Table 12 summarizes the environment variables and flags for collecting diagnostic information. These environment variables and flags enable you to generate diagnostic information that may be required by the IBM Support Center in resolving PE-related problems.
Table 13 summarizes the environment variables and flags for the Message Passing Interface. These environment variables and flags allow you to change message and memory sizes, as well as other message passing information.
Table 14 summarizes some miscellaneous environment variables and flags. These environment variables and flags provide control for the Program Marker Array, enable additional error checking, and let you set a dispatch priority class for execution.

You can use the POE command-line flags on the poe, pdbx, and pedb commands. You can also use some of these flags on program names when individually loading nodes from STDIN or a POE commands file. The flags you can use are mainly those having to do with VT trace collection. They are:

-infolevel or -ilevel
-ttempsize or -ttsize
-tmpdir
-samplefreq or -sfreq
-tbuffwrap or -tbwrap
-tbuffsize or -tbsize
-euidevelop

In the tables that follow, a check mark (&check.) denotes those flags you can use when individually loading nodes. For more information on individually loading nodes, refer to "Invoking an MPMD Program".

Table 8. POE Environment Variables/Command-Line Flags for Partition Manager Control

The Environment Variable/Command-Line Flag(s): Set: Possible Values: Default:
MP_ADAPTER_USE -adapter_use How the node's adapter should be used. If using LoadLeveler, the US communication subsystem library does not require dedicated use of the SP switch on the node. Adapter use will be defaulted, as in Table 4, but shared usage may be specified. If using the Resource Manager, this value is only used when POE is requesting non-specific nodes via the MP_RMPOOL or -rmpool setting. One of the following strings:

dedicated
Only a single program task can use the adapter.
shared
A number of tasks on the node can use the adapter.
Dedicated for US jobs, shared for IP jobs.
MP_AUTH (no associated command line flag) The type of user authorization. This value can be overridden with an entry in /etc/poe.limits. One of the following strings:

AIX
AIX based user security.
DFS
DFS/DCE bsed user security.
AIX
MP_CPU_USE -cpu_use How the node's CPU should be used. If using LoadLeveler, the US communication subsystem library does not require unique CPU use on the node. CPU use will be defaulted, as in Table 4, but multiple use may be specified. If using the Resource Manager, this value is only used when POE is requesting non-specific nodes via the MP_RMPOOL or -rmpool setting. One of the following strings:

unique
Only your program's task can use the CPU.
multiple
Your program may share the node with other users.
Unique for US jobs, multiple for IP jobs.
MP_EUIDEVICE -euidevice The adapter set to use for message passing - either Ethernet, FDDI, token-ring, or the IBM RS/6000 SP's high performance switch adapter. One of the following strings:

en0
Ethernet
fi0
FDDI
tr0
token-ring
css0
high performance switch
The adapter set used as the external network address.
MP_EUILIB -euilib The communication subsystem library implementation to use for communication - either the IP communication subsystem or the User Space (US) communication subsystem. Programs that use LAPI must set MP_EUILIB (or -euilib) to us. In order to use the US communication subsystem, you must have an SP system configured with its high performance switch feature.
One of the following strings:

ip
The IP communication subsystem.
us
The US communication subsystem.

Note: This specification is case-sensitive.

ip
MP_EUILIBPATH -euilibpath The path to the message passing and communication subsystem libraries. This only needs to be set if the libraries are moved. Any path specifier. /usr/lpp/ppe.poe/lib
MP_HOSTFILE -hostfile -hfile The name of a host list file for node allocation. Any file specifier or the word NULL. host.list in the current directory.
MP_PROCS -procs The number of program tasks. Any number from 1 to the maximum supported configuration. 1
MP_PULSE -pulse The interval (in seconds) at which POE checks the remote nodes to ensure that they are actively communicating with the home node.
Note: Pulse is ignored for pdbx and pedb.
An integer greater than or equal to 0. 600
MP_RESD -resd Whether or not the Partition Manager should connect to a job management system (LoadLeveler or the Resource Manager) to allocate nodes.
Notes:

MP_RESD only specifies whether or not to use a job management system.
When the Resource Manager is used, the actual system you are using is identified by the environment variable SP_NAME, of the control workstation on the SP system.
When running POE from a workstation that is external to the LoadLeveler cluster, the LoadL.so fileset must be installed on the external node (see Using and Administering LoadLeveler and IBM Parallel Environment for AIX: Installation for more information).
When running POE from a workstation that is external to the SP system, and using the Resource Manager, the ssp.clients fileset must be installed on the external node (see IBM Parallel Environment for AIX: Installation for more information).
yes no None
MP_RETRY -retry The period of time (in seconds) between processor node allocation retries if there are not enough processor nodes immediately available to run a program. This is only valid if you are using the SP system Resource Manager. An integer greater than or equal to 0. 0 (no retry)
MP_RETRYCOUNT -retrycount The number of times (at the interval set by MP_RETRY) that the Partition Manager should attempt to allocate processor nodes. An integer greater than or equal to 0. 0
MP_MSG_API -msg_api To indicate to POE which message passing API is being used by the parallel tasks. You need to set this environment variable if a parallel task is using LAPI alone or in conjunction with MPI. You do not need to set it if a parallel task is using MPI only. MPI LAPI or MPI,LAPI MPI
MP_RMPOOL -rmpool With regard to LoadLeveler, the name or number of the pool that should be used for non-specific node allocation. With regard to the Resource Manager, the number of the SP system pool that should be used for non-specific node allocation. This environment variable/command-line flag only applies to LoadLeveler or the SP system Resource Manager. An identifying pool number. None
MP_NODES -nodes To specify the number of physical nodes on which to run the parallel tasks. It may be used alone or in conjunction with MP_TASKS_PER_NODE and/or MP_PROCS, as described in Table 7. Any number from 1 to the maximum supported configuration. None
MP_TASKS_PER_NODE -tasks_per_node To specify the number of tasks to be run on each of the physical nodes. It may be used in conjunction with MP_NODES and/or MP_PROCS, as described in Table 7, but may not be used alone. Any number from 1 to the maximum supported configuration. None
MP_SAVEHOSTFILE -savehostfile The name of an output host list file to be generated by the Partition Manager. Any relative or full path name. None
MP_REMOTEDIR (no associated command line flag) The name of a script which echoes the name of the current directory to be used on the remote nodes. Any file specifier. None
MP_TIMEOUT (no associated command line flag) The length of time that POE waits before abandoning an attempt to connect to the remote nodes. Any number greater than 0. 150 seconds
MP_CHECKFILE (no associated command line flag) The base name of the checkpoint file. Any file specifier. None
MP_CHECKDIR (no associated command line flag) The directory where the checkpoint file will reside. Any path specifier. None

Table 9. POE Environment Variables/Command-Line Flags for Job Specification

The Environment Variable/Command-Line Flag(s): Set: Possible Values: Default:
MP_CMDFILE -cmdfile The name of a POE commands file used to load the nodes of your partition. If set, POE will read the commands file rather than STDIN. Any file specifier. None
MP_NEWJOB -newjob Whether or not the Partition Manager maintains your partition for multiple job steps. yes no no
MP_PGMMODEL -pgmmodel The programming model you are using. spmd mpmd spmd

The Environment Variable/Command-Line Flag(s):	Set:	Possible Values:	Default:
MP_CMDFILE -cmdfile	The name of a POE commands file used to load the nodes of your partition. If set, POE will read the commands file rather than STDIN.	Any file specifier.	None
MP_NEWJOB -newjob	Whether or not the Partition Manager maintains your partition for multiple job steps.	yes no	no
MP_PGMMODEL -pgmmodel	The programming model you are using.	spmd mpmd	spmd

Table 10. POE Environment Variables/Command-Line Flags for I/O Control

The Environment Variable/Command-Line Flag(s): Set: Possible Values: Default:
MP_LABELIO -labelio Whether or not output from the parallel tasks is labeled by task id. yes no no (yes for pdbx)
MP_STDINMODE -stdinmode The input mode. This determines how input is managed for the parallel tasks.

all
All tasks receive the same input data from STDIN.
none
No tasks receive input data from STDIN; STDIN will be used by the home node only.
a task id
STDIN is only sent to the task identified.
all
MP_HOLD_STDIN (no associated command line flag) Whether or not sending of STDIN from the home node to the remote nodes is deferred until the message passing partition has been established. yes no no
MP_STDOUTMODE -stdoutmode The output mode. This determines how STDOUT is handled by the parallel tasks. One of the following:

unordered
All tasks write output data to STDOUT asynchronously.
ordered
Output data from each parallel task is written to its own buffer. Later, all buffers are flushed, in task order, to STDOUT.
a task id
Only the task indicated writes output data to STDOUT.
unordered

The Environment Variable/Command-Line Flag(s):	Set:	Possible Values:	Default:
MP_LABELIO -labelio	Whether or not output from the parallel tasks is labeled by task id.	yes no	no (yes for pdbx)
MP_STDINMODE -stdinmode	The input mode. This determines how input is managed for the parallel tasks.	all All tasks receive the same input data from STDIN. none No tasks receive input data from STDIN; STDIN will be used by the home node only. a task id STDIN is only sent to the task identified.	all
MP_HOLD_STDIN (no associated command line flag)	Whether or not sending of STDIN from the home node to the remote nodes is deferred until the message passing partition has been established.	yes no	no
MP_STDOUTMODE -stdoutmode	The output mode. This determines how STDOUT is handled by the parallel tasks.	One of the following: unordered All tasks write output data to STDOUT asynchronously. ordered Output data from each parallel task is written to its own buffer. Later, all buffers are flushed, in task order, to STDOUT. a task id Only the task indicated writes output data to STDOUT.	unordered

Table 11. POE Environment Variables/Command-Line Flags for VT Trace Collection

The Environment Variable/Command-Line Flag(s): Set: Possible Values: Default:
MP_SAMPLEFREQ -samplefreq &check. -sfreq &check. The interval (in milliseconds) at which AIX kernel statistics are sampled when executing a program with tracing on. An integer greater than or equal to 0. 10
MP_TBUFFSIZE -tbuffsize &check. -tbsize &check. The size (in bytes) of the buffer used when generating trace files. nnnK nnnM 1M
MP_TBUFFWRAP -tbuffwrap &check. -tbwrap &check. A wraparound storage approach for trace records instead of the default three-tiered approach. With this approach, the system keeps overwriting the buffer instead of flushing it to a temp file. yes no no
MP_TMPDIR -tmpdir &check. The temporary directory to which output trace files are written.
Note: This environment variable/flag is also used by the pdbx and pedb commands to determine the directory to which individual startup files are written for each dbx task. This is where the pedb remote debug logfile is written.
Any path specifier. /tmp/username /tmp (for pedb and pdbx)
MP_TRACEDIR -tracedir -tdir The directory to which the final integrated trace file is built. Any path specifier. The current directory.
MP_TRACEFILE -tracefile -tfile The name of the output trace file created when executing a program with tracing on. Any file specifier. The name of the program with the suffix .trc added.
MP_TRACELEVEL -tracelevel -tlevel The level of VT tracing that should be generated during the execution of a program.
Note: pedb automatically sets the trace level to 0.
One of the following integers:

0
No trace records
1
Application Markers
2
AIX Kernel Statistic and Application Markers
3
Message Passing, Collective Communication, and Application Markers
9
All trace records
0
MP_TTEMPSIZE -ttempsize &check. -ttsize &check. The size (in bytes) of the temp file used when generating trace files. nnnM nnnG 10M

Note: &check. symbol denotes flags you can use when individually loading nodes.

Table 12. POE Environment Variables/Command-Line Flags for Diagnostic Information

The Environment Variable/Command-Line Flag(s): Set: Possible Values: Default:
MP_INFOLEVEL -infolevel &check. -ilevel &check. The level of message reporting. One of the following integers:

0
Error
1
Warning and error
2
Informational, warning, and error
3
Informational, warning, and error. Also reports high-level diagnostic messages for use by the IBM Support Center.
4, 5, 6
Informational, warning, and error. Also reports high- and low-level diagnostic messages for use by the IBM Support Center.
1
MP_PMDLOG -pmdlog Whether or not diagnostic messages should be logged to a file in /tmp on each of the remote nodes. Typically, this environment variable/command-line flag is only used under the direction of the IBM Support Center in resolving a PE-related problem. yes no no
MP_DEBUG_LOG (no associated command-line flag) The level of diagnostic messages written to $MP_TMPDIR/debug_log.pid.taskid. Typically, this environment variable/command-line flag is only used under the direction of the IBM Support Center in resolving a PE-related problem.
Note: MP_DEBUG_LOG is only valid for pedb.
0 - 4 0
MP_DEBUG_INITIAL_STOP (no associated command-line flag) The initial breakpoint in the application where pdbx or pedb will get control. One of the following:

"filename":line_number
function_name
The first executable source line in the main routine.
MP_PMDSUFFIX (no associated command-line flag) When using LoadLeveler, to determine a string to be appended to the normal partition manager daemon executable. The normal partition manager daemon executable specified is /etc/pmdv2. By setting MP_PMDSUFFIX, you can append a string to pmdv2. If MP_PMDSUFFIX is set to abc, for example, then the partition manager daemon that gets run on each node of the parallel task is /etc/pmdv2abc. When using the Resource Manager, to determine a string to be appended to the normal tcp service. The normal tcp service specified in /etc/services is named pmv2. By setting MP_PMDSUFFIX, you can append a string to pmv2. If MP_PMDSUFFIX is set to abc, for example, then the service requested in /etc/services is pmv2abc. Using the environment variable with LoadLeveler or the Resource Manager as described above permits testing of alternate versions of the Partition Manager daemon. Typically, this environment variable is only used under the direction of the IBM Support Center in resolving a PE-related problem. Any string. None

Note: &check. symbol denotes flags you can use when individually loading nodes.

Table 13. POE Environment Variables/Command-Line Flags for Message Passing Interface (MPI)

The Environment Variable/Command-Line Flag(s): Set: Possible Values: Default:
MP_MAX_TYPEDEPTH -max_typedepth To change the maximum depth of message derived data types. An integer greater than or equal to 1. 5
MP_EAGER_LIMIT -eager_limit To change the threshold value for message size, above which rendezvous protocol is used.
To ensure that at least 32 messages can be outstanding between any 2 tasks, MP_EAGER_LIMIT will be adjusted based on the number of tasks according to the following table (when MP_EAGER_LIMIT and MP_BUFFER_MEM have not been set by the user):
Number of Tasks MP_EAGER_LIMIT =============== ============== 1 - 16 4096 17 - 32 2048 33 - 64 1024 65 - 128 512
An integer less than or equal to 65536. 4KB
MP_BUFFER_MEM -buffer_mem To change the maximum size of memory used by the communication subsystem to buffer early arrivals. An integer less than or equal to 64MB. 2800000 bytes (IP) 64MB (US)
MP_INTRDELAY -intrdelay To tune the delay parameter without having to recompile existing applications. An integer greater than 0. 35 &usec. (TB2) 1 &usec. (TB3)
MP_USE_FLOW_CONTROL -use_flow_control To limit the maximum number of outstanding messages posted by a sender. yes no no
MP_CSS_INTERRUPT -css_interrupt Whether or not arriving packets generate interrupts. This may provide better performance for certain applications. yes no no
MP_TIMEOUT (no associated command-line flag) To change the length of time the communication subsystem will wait for a connection to be established during message passing initialization. An integer greater than 0. 150 seconds
MP_THREAD_STACKSIZE -thread_stacksize To specify the additional stacksize allocated for user programs executing on an MPI service thread. If you allocate insufficient space, the program may encounter a SIGSEGV exception. nnnnn nnnK nnM (where K=1024 bytes and M=1024*1024 bytes) None
MP_SINGLE_THREAD -single_thread To avoid mutex lock overheads in a program which is known to be single threaded. no yes no
MP_WAIT_MODE -wait_mode To specify how a thread or task behaves when it discovers it is blocked, waiting for a message to arrive. poll yield sleep poll for US sleep for IP
MP_POLLING_INTERVAL -polling_interval To change the polling interval, in microseconds. An integer between 1 and 2 billion. 400,000 for US 180,000 for IP

The Environment Variable/Command-Line Flag(s):	Set:	Possible Values:	Default:
MP_MAX_TYPEDEPTH -max_typedepth	To change the maximum depth of message derived data types.	An integer greater than or equal to 1.	5
MP_EAGER_LIMIT -eager_limit	To change the threshold value for message size, above which rendezvous protocol is used. To ensure that at least 32 messages can be outstanding between any 2 tasks, MP_EAGER_LIMIT will be adjusted based on the number of tasks according to the following table (when MP_EAGER_LIMIT and MP_BUFFER_MEM have not been set by the user): Number of Tasks MP_EAGER_LIMIT =============== ============== 1 - 16 4096 17 - 32 2048 33 - 64 1024 65 - 128 512	An integer less than or equal to 65536.	4KB
MP_BUFFER_MEM -buffer_mem	To change the maximum size of memory used by the communication subsystem to buffer early arrivals.	An integer less than or equal to 64MB.	2800000 bytes (IP) 64MB (US)
MP_INTRDELAY -intrdelay	To tune the delay parameter without having to recompile existing applications.	An integer greater than 0.	35 &usec. (TB2) 1 &usec. (TB3)
MP_USE_FLOW_CONTROL -use_flow_control	To limit the maximum number of outstanding messages posted by a sender.	yes no	no
MP_CSS_INTERRUPT -css_interrupt	Whether or not arriving packets generate interrupts. This may provide better performance for certain applications.	yes no	no
MP_TIMEOUT (no associated command-line flag)	To change the length of time the communication subsystem will wait for a connection to be established during message passing initialization.	An integer greater than 0.	150 seconds
MP_THREAD_STACKSIZE -thread_stacksize	To specify the additional stacksize allocated for user programs executing on an MPI service thread. If you allocate insufficient space, the program may encounter a SIGSEGV exception.	nnnnn nnnK nnM (where K=1024 bytes and M=1024*1024 bytes)	None
MP_SINGLE_THREAD -single_thread	To avoid mutex lock overheads in a program which is known to be single threaded.	no yes	no
MP_WAIT_MODE -wait_mode	To specify how a thread or task behaves when it discovers it is blocked, waiting for a message to arrive.	poll yield sleep	poll for US `sleep` for IP
MP_POLLING_INTERVAL -polling_interval	To change the polling interval, in microseconds.	An integer between 1 and 2 billion.	400,000 for US 180,000 for IP

Table 14. Other POE Environment Variables/Command-Line Flags

The Environment Variable/Command-Line Flag(s): Set: Possible Values: Default:
MP_DBXPROMPTMOD (no associated command-line flag) A modified dbx prompt. The dbx prompt \n(dbx) is used by the pdbx command as an indicator denoting that a dbx subcommand has completed. This environment variable modifies that prompt. Any value assigned to it will have a "." prepended and will then be inserted in the \n(dbx) prompt between the "x" and the ")". This environment variable is useful when the string \n(dbx) is present in the output of the program being debugged. Any string. None
MP_EUIDEVELOP -euidevelop &check. Whether or not the message passing interface performs more detailed checking during execution. This additional checking is intended for developing applications, and can significantly slow performance. You can also start and stop parameter checking with deb (for "debug") and min (for "minimum"). yes (for "develop") no (for "normal") deb (for "debug") min (for "minimum") no
MP_FENCE (no associated command-line flag) A "fence" character string for separating arguments you want parsed by POE from those you do not. Any string. None
MP_NOARGLIST (no associated command-line flag) Whether or not POE ignores the argument list. If set to yes, POE will not attempt to remove POE command-line flags before passing the argument list to the user's program. yes no no
MP_PMLIGHTS -pmlights The number of lights displayed (per row) on the Program Marker Array. An integer greater than or equal to 0. 0
MP_PRIORITY (no associated command-line flag) A dispatch priority class for execution. See IBM Parallel Environment for AIX: Installation for more information on dispatch priority classes. Any of the dispatch priority classes set up by the system administrator in the file /etc/poe.priority. None
MP_USRPORT -usrport The port id used by the Partition Manager to connect to the Program Marker Array. Any positive integer less than 32767. Standard TCP/IP practice suggests using ports greater than 5000 and less than 10000. 9999
MP_COREDIR -coredir To create a separate directory for each task's core file. Any valid directory name. coredir.taskid

The Environment Variable/Command-Line Flag(s):	Set:	Possible Values:	Default:
MP_DBXPROMPTMOD (no associated command-line flag)	A modified dbx prompt. The dbx prompt `\n(dbx)` is used by the pdbx command as an indicator denoting that a dbx subcommand has completed. This environment variable modifies that prompt. Any value assigned to it will have a "." prepended and will then be inserted in the \n(dbx) prompt between the "x" and the ")". This environment variable is useful when the string \n(dbx) is present in the output of the program being debugged.	Any string.	None
MP_EUIDEVELOP -euidevelop &check.	Whether or not the message passing interface performs more detailed checking during execution. This additional checking is intended for developing applications, and can significantly slow performance. You can also start and stop parameter checking with deb (for "debug") and min (for "minimum").	yes (for "develop") no (for "normal") deb (for "debug") min (for "minimum")	no
MP_FENCE (no associated command-line flag)	A "fence" character string for separating arguments you want parsed by POE from those you do not.	Any string.	None
MP_NOARGLIST (no associated command-line flag)	Whether or not POE ignores the argument list. If set to yes, POE will not attempt to remove POE command-line flags before passing the argument list to the user's program.	yes no	no
MP_PMLIGHTS -pmlights	The number of lights displayed (per row) on the Program Marker Array.	An integer greater than or equal to 0.	0
MP_PRIORITY (no associated command-line flag)	A dispatch priority class for execution. See IBM Parallel Environment for AIX: Installation for more information on dispatch priority classes.	Any of the dispatch priority classes set up by the system administrator in the file /etc/poe.priority.	None
MP_USRPORT -usrport	The port id used by the Partition Manager to connect to the Program Marker Array.	Any positive integer less than 32767. Standard TCP/IP practice suggests using ports greater than 5000 and less than 10000.	9999
MP_COREDIR -coredir	To create a separate directory for each task's core file.	Any valid directory name.	coredir.taskid

[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]