Guide and Reference

Distributing Data in an HPF Program

You must distribute your data before calling Parallel ESSL from your HPF program. This section shows how to use the HPF data distribution directives to distribute your data. An example of the data distribution techniques are also shown in a sample HPF program in Appendix B. "Sample Programs".

All the Parallel ESSL HPF subroutines, except the Banded Linear Algebraic Equations, Fourier Transforms, and the Random Number Generation subroutine, support block-cyclic distribution. The Banded Linear Algebraic Equations, Fourier Transforms, and the Random Number Generation subroutine only support block distribution.

This section shows examples of how you could distribute your data over one- or two-dimensional process grids:

• "Vectors"
• "Matrices"
• "Sequences (Fourier Transform)"

Notes:

1. The XL HPF default to map processes into a process grid is column-major order. For an example, see Table 13.

2. XL HPF supports only CYCLIC(1) in the DISTRIBUTE directive, unless it appears in the interface block of a local procedure. For more information, see "Using Extrinsic Procedures--The Parallel ESSL Subroutines".

3. All the examples in this section use CYCLIC (which has the same meaning as CYCLIC(1)) and ALIGN in the DISTRIBUTE directives. To find out where to code distribution directives in your HPF program, see "Sample HPF Programs".

   The examples show real data structures, but the techniques apply equally to complex data structures.

Vectors

Parallel ESSL supports block-cyclic distribution for vectors over one- or two-dimensional process grids, except for URNG. For URNG, vectors are distributed over a one-dimensional process grid using block distribution, where the length n of the vector x must be evenly divisible by the available processes np multiplied by the block size nb. In other words, n/((np)(nb)) must be an integer.

Distributed over One-Dimensional Process Grids

Block-Cyclically Distributing a Vector

This example shows how to distribute a vector of length 24 block-cyclically over a column-oriented, one-dimensional process grid with four processes.

      REAL(KIND(1.0D0)), DIMENSION(:,:) :: A
      REAL(KIND(1.0D0)), DIMENSION(24) :: X
!HPF$ PROCESSORS PROC(4)
!HPF$ ALIGN      X(:) WITH A(:,*)
!HPF$ DISTRIBUTE (CYCLIC,*) ONTO PROC :: A

This example shows how to distribute the same vector block-cyclically over a row-oriented, one-dimensional process grid with four processes.

      REAL(KIND(1.0D0)), DIMENSION(:,:) :: A
      REAL(KIND(1.0D0)), DIMENSION(24) :: X
!HPF$ PROCESSORS PROC(4)
!HPF$ ALIGN      X(:) WITH A(*,:)
!HPF$ DISTRIBUTE (*,CYCLIC) ONTO PROC :: A

Block Distributing a Vector

For URNG, this example shows how to distribute a vector of length 18 block-column over a row-oriented, one-dimensional process grid with six processes.

      REAL(KIND(1.0D0)), DIMENSION(18,18) :: A
      REAL(KIND(1.0D0)), DIMENSION(18) :: X
!HPF$ PROCESSORS PROC(6)
!HPF$ ALIGN      X(:) WITH A(1,:)
!HPF$ DISTRIBUTE (*,BLOCK) ONTO PROC :: A

Block-Cyclic Distribution over Two-Dimensional Process Grids

This example shows how to distribute a vector of length 18 block-cyclically over the third column of a two-dimensional process grid.

      REAL(KIND(1.0D0)), DIMENSION(:,:) :: A
      REAL(KIND(1.0D0)), DIMENSION(18) :: X
!HPF$ PROCESSORS PROC(2,3)
!HPF$ ALIGN      X(:) WITH A(:,3)
!HPF$ DISTRIBUTE (CYCLIC,CYCLIC) ONTO PROC :: A

This example shows how to distribute the same vector block-cyclically over the second row of a two-dimensional process grid:

      REAL(KIND(1.0D0)), DIMENSION(:,:) :: A
      REAL(KIND(1.0D0)), DIMENSION(18) :: X
!HPF$ PROCESSORS PROC(2,3)
!HPF$ ALIGN      X(:) WITH A(2,:)
!HPF$ DISTRIBUTE (CYCLIC,CYCLIC) ONTO PROC :: A

Matrices

Parallel ESSL, except for the Banded Linear Algebraic Equations, supports block-cyclic distribution for matrices over one- or two-dimensional process grids. The Banded Linear Algebraic Equations supports only block distribution for matrices over one-dimensional process grids.

Distributed over One-Dimensional Process Grids

Block-Cyclically Distributing a Matrix

This example shows how a 6 × 8 matrix A is distributed block-cyclically over a column-oriented, one-dimensional process grid with three processes.

      REAL(KIND(1.0D0)), DIMENSION(6,8) :: A
!HPF$ PROCESSORS PROC(3)
!HPF$ DISTRIBUTE (CYCLIC,*) ONTO PROC :: A

      REAL(KIND(1.0D0)), DIMENSION(6,8) :: A
!HPF$ PROCESSORS PROC(4)
!HPF$ DISTRIBUTE (*,CYCLIC) ONTO PROC :: A

Block Distributing a Matrix

This example shows how a 5 × 5 matrix A is distributed over a row-oriented, one-dimensional process grid with four processes, using block-column distribution.

      REAL(KIND(1.0D0)), DIMENSION(5,5) :: A
!HPF$ PROCESSORS PROC(4)
!HPF$ DISTRIBUTE (*,BLOCK) ONTO PROC :: A

This example shows how a 5 × 5 matrix A is distributed over a column-oriented, one-dimensional process grid with four processes, using block-row distribution.

      REAL(KIND(1.0D0)), DIMENSION(5,5) :: A
!HPF$ PROCESSORS PROC(4)
!HPF$ DISTRIBUTE (BLOCK,*) ONTO PROC :: A

This example shows how a 100 × 100 matrix A is distributed over a row-oriented, one-dimensional process grid with five processes, using block-column distribution.

      REAL(KIND(1.0D0)), DIMENSION(100,100) :: A
!HPF$ PROCESSORS PROC(5)
!HPF$ DISTRIBUTE (*,BLOCK(40)) ONTO PROC :: A

Block-Cyclic Distribution over Two-Dimensional Process Grids

This example shows how a 9 × 26 matrix A is distributed block-cyclically over a 2 × 3, two-dimensional process grid.

      REAL(KIND(1.0D0)), DIMENSION(9,26) :: A
!HPF$ PROCESSORS PROC(2,3)
!HPF$ DISTRIBUTE (CYCLIC,CYCLIC) ONTO PROC :: A

Sequences (Fourier Transform)

Parallel ESSL supports block data distribution for two- or three-dimensional sequences over a one-dimensional process grid. (For details on FFT-packed storage mode, see "Two-Dimensional Sequence" and "Three-Dimensional Sequences".)

Two-Dimensional Sequence

This example shows how an 8 × 12, two-dimensional sequence is distributed over a one-dimensional process grid with four processes, using block-column distribution. The sequence is distributed with a block size of 3.

      REAL(KIND(1.0D0)), DIMENSION(8, 12) :: Y
!HPF$ PROCESSORS PROC(4)
!HPF$ DISTRIBUTE (*, BLOCK) ONTO PROC :: Y

Three-Dimensional Sequences

This example shows how a three-dimensional sequence, with four planes that are each of size 2 × 2, is distributed over a one-dimensional process grid with two processes, using block plane distribution. The sequence is distributed with a block size of 2:

      REAL(KIND(1.0D0)), DIMENSION(2, 2, 4) :: X
!HPF$ PROCESSORS PROC(2)
!HPF$ DISTRIBUTE (*, *, BLOCK) ONTO PROC :: X