MPI Programming and Subroutine Reference

Appendix B. Profiling Message Passing

AIX Profiling

If you use prof, gprof or xprofiler and the appropriate compiler flag (-p or -pg), you can profile your program.

The message passing library is not enabled for prof or gprof, profiling counts. You can obtain profiling information by using the name-shifted MPI functions provided.

MPI Nameshift Profiling

To use nameshift profiling routines that are written to the C bindings with an MPI program written in C, or the FORTRAN bindings with an MPI program written in FORTRAN, do the following:

Create .o files for your profiling routines.
Use one of the following commands to list both the MPI program .o files and the profiling .o files as inputs:
- mpcc
- mpxlf
- mpcc_r
- mpxlf_r
- mpCC
- mpCC_r
See IBM Parallel Environment for AIX: Operation and Use, Volume 1 for more information on these commands.
Run the resulting executable normally.

To use nameshift profiling routines which are written to the C bindings with an MPI program written in FORTRAN, follow these steps:

If you are both the creator and user of the profiling library, follow all the steps (1 through 17).
If you are the creator of the profiling library, follow steps 1 through 6. You also need to provide the user with the file created in step 2.
If you are the user of the profiling library, follow steps 7 through 17. For step 14, use the file generated by the creator of the profiling library in step 2.

Based on the above, follow the appropriate steps:

Create a source file containing profiling versions of all the MPI routines you want to profile. As an example, create a source file called myprof.c containing the following code:

#include <stdio.h>
#include "mpi.h"
int MPI_Init(int *argc, char ***argv) {
  int rc;
 
  printf("hello from profiling layer MPI_Init...\n");
  rc = PMPI_Init(argc, argv);
  printf("goodbye from profiling layer MPI_Init...\n");
  return(rc);
}

Create an export file containing all of the symbols your profiling library will export. Begin this file with the name your profiling library will have and the name of the .o that will have the object code of your profiling routines. As an example, create a file called myprof.exp containing the following statements:
```
#!libmyprof.a(newmyprof.o)
MPI_Init
```
Create a file called mpicore.imp. This file will import all of the PMPI symbols that your profiling library needs. Begin this file with the statement #!libmpi.a(mpicore.o). The following is an example of mpicore.imp:
```
#!libmpi.a(mpicore.o)
PMPI_Init
```
Compile the source file containing your profiling MPI routines. For example:
```
cc -c myprof.c -I/usr/lpp/ppe.poe/include
```
The -I defines the location of mpi.h.
Create your profiling MPI library. Use the file created in step 2 as the export file and the file created in step 3 as the import file. Include any other libraries your profiling code needs, such as libc. For example:
```
ld -o newmyprof.o myprof.o -bM:SRE -H512 -T512 -bnoentry
-bI:mpicore.imp -bE:myprof.exp -lc
```
Archive the object module created in step 5 into a library. The library name should be the same as that listed in the first statement of the export file created in step 2. For example:
```
ar rv libmyprof.a newmyprof.o
```
Use the following command to extract mpifort.o from libmpi.a:
```
ar -xv /usr/lpp/ppe.poe/lib/libmpi.a mpifort.o
```
Use the following command to create a non-shared version of mpifort.o:
```
ld -o mpifort.tmp mpifort.o -r -bnso -bnoentry
```
Use the following command to extract mpicore.o from libmpi.a:
```
ar -xv /usr/lpp/ppe.poe/lib/libmpi.a mpicore.o
```

Use the following command to create an export list from the extracted mpicore.o:

/usr/bin/dump -nvp mpicore.o  | /usr/bin/grep "&hat.\[" | cut -f2-
| cut -c26- | grep -y "&hat.exp" | cut -c35- | sort | uniq > mpicore.exp

Delete all of the symbols selected for profiling in step 2 from mpicore.exp. Then create a new line at the top of the file. We'll call this the new line 1 To the new line 1, add #!libmpi.a(mpicore.o). Continuing with our example: MPI_Init would now be deleted from mpicore.exp. and #!libmpi.a(mpicore.o) would now comprise line 1 of mpicore.exp.
Create a file called vt.exp with the following statements:
```
#!libvtd.a(dynamic.o)
VT_instaddr_depth
```
Use the following command to create an export list from the extracted mpifort.o:
```
/usr/bin/dump -nvp mpifort.o  | /usr/bin/grep "&hat.\[" | cut -f2-
| cut -c26- | grep -y "&hat.exp" | cut -c35- | sort | uniq > mpifort.exp
```
insert #!libpmpi.a(newmpifort.o) as the first line of the new mpifort.exp file
Create a new version of mpifort.o from the non-shared version you created in step 8. It will import the symbols representing your profiling functions from your profiling library using the file created in step 2. It will import the remaining MPI symbols from mpicore.o using the file created in step 11. One additional symbol must be imported using the file created in step 12. The new mpifort.o will export symbols using the file created in step 13.
```
ld -o newmpifort.o mpifort.tmp -bI:
mpicore.exp -bI:myprof.exp -bI:vt.exp
-bE:mpifort.exp -bM:SRE -H512 -T512 -bnoentry
```
Use the following command to create a library containing a FORTRAN object which will reference your profiling library:
```
ar rv libpmpi.a newmpifort.o
```

Create a program that uses an MPI function you've profiled. An example would be a file called hwinit.f that contains the following statements:

 c -------------------------------------
       program  hwinit
       include 'mpif.h'
       integer forterr
 c
       call MPI_INIT(forterr)
 c
 c  Write comments to screen.
 c
       write(6,*)'Hello from task '
 c
       call MPI_FINALIZE(forterr)
 c
       stop
       end
 c

Compile your program linking in the library created in step 15. For example:
```
mpxlf -o hwinit hwinit.f -lpmpi -L.
```

Sample CPU MPI Time Program

The following is a sample MPI program that uses the name-shifted MPI interface to separate the amount of user and system CPU time used by MPI.

CPU MPI Time Example

#include "mpi.h"
#include <sys/types.h>
#include <time.h>
#include <sys/times.h>
 
#define ARRAY_SIZE 1000000
#define VALUE 123
 
struct tms mpitms;
double mpi_elapsed;
 
void main()
{
   int in[ARRAY_SIZE],out[ARRAY_SIZE],tasks,me,src,dest;
   int i;
   MPI_Status status[2];
   MPI_Request msgid [2];
 
   for (i=0;i<ARRAY_SIZE;i++)out[i]=VALUE;
 
   MPI_Init(&argc,&argv);
   MPI_Comm_size(MPI_COMM_WORLD,&tasks);
   MPI_Comm_rank(MPI_COMM_WORLD,&me);
 
   mpi_elapsed = MPI_Wtime();
 
   dest = (me==tasks-1) ? 0 : me+1;
   MPI_Isend(out,ARRAY_SIZE,MPI_INT,dest,5,MPI_COMM_WORLD,&msgid[0]);
   src = (me==0) ? tasks-1 : me-1;
   MPI_Irecv(in,ARRAY_SIZE,MPI_INT,src,5,MPI_COMM_WORLD,&msgid[1]);
   MPI_Waitall(2,msgid,status);
 
   for (i=0; i< ARRAY_SIZE; i++) {
   if(in[i] != VALUE )
      printf("ERROR on node %d, in = %d\n",me,in[i]);
          break;
          }
 
   MPI_Barrier(MPI_COMM_WORLD);
 
   mpi_elapsed = MPI_Wtime() - mpi_elapsed;
 
   printf("MPI CPU times: user %f, system %f, total %f sec\n",
                ((float)mpitms.tms_utime)/CLK_TCK,
                ((float)mpitms.tms_stime)/CLK_TCK,
                (float)(mpitms.tms_utime+mpitms.tms_stime)/CLK_TCK);
   printf("MPI Elapsed time: %f sec\n", mpi_elapsed);
 
   MPI_Finalize();
}
 
/***************************************/
/* Replacement functions for profiling */
/***************************************/
 
int MPI_Isend(void* buf, int count, MPI_Datatype datatype,
       int dest, int tag, MPI_Comm comm, MPI_Request *request)
{
        struct tms beforetms, aftertms;
        int rc;
        times(&beforetms);
 
    rc = PMPI_Isend(buf,count,datatype,dest,tag,comm,request);
 
        times(&aftertms);
        mpitms.tms_utime += (aftertms.tms_utime - beforetms.tms_utime);
        mpitms.tms_stime += (aftertms.tms_stime - beforetms.tms_stime);
        return (rc);
        }
 
int MPI_Waitall(int count, MPI_Request *array_of_requests,
              MPI_Status *array_of_statuses)
{
        struct tms beforetms, aftertms;
        int rc;
        times(&beforetms);
 
    rc = PMPI_Waitall(count,array_of_requests,array_of_statuses);
 
        times(&aftertms);
        mpitms.tms_utime += (aftertms.tms_utime - beforetms.tms_utime);
        mpitms.tms_stime += (aftertms.tms_stime - beforetms.tms_stime);
        return (rc);
        }
 
int MPI_Irecv(void* buf, int count, MPI_Datatype datatype,
       int source, int tag, MPI_Comm comm, MPI_Request *request)
{
        struct tms beforetms, aftertms;
        int rc;
        times(&beforetms);
 
    rc = PMPI_Irecv(buf,count,datatype,source,tag,comm,request);
 
        times(&aftertms);
        mpitms.tms_utime += (aftertms.tms_utime - beforetms.tms_utime);
        mpitms.tms_stime += (aftertms.tms_stime - beforetms.tms_stime);
        return (rc);
        }
 
int MPI_Barrier(MPI_Comm comm )
{
        struct tms beforetms, aftertms;
        int rc;
        times(&beforetms);
 
    rc = PMPI_Barrier(comm);
 
        times(&aftertms);
        mpitms.tms_utime += (aftertms.tms_utime - beforetms.tms_utime);
        mpitms.tms_stime += (aftertms.tms_stime - beforetms.tms_stime);
        return (rc);
        }

[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]