Derived Data Types with MPI

1. Introduction

The basic MPI communication mechanisms can be used to send or receive a sequence of identical elements that are contiguous in memory. It is often desirable to send data that is not homogene ous or that is not contiguous in memory. This would amortize the fixed overhead of sending and receiving a message over the transmittal of many elements. MPI provides two mechanisms to achi eve this:

  • The user can define derived datatype that specify more general data layouts
  • A sending process can explicitly pack noncontiguous data into a contiguous buffer and then send it; a reveiving process can explicitly unpack data received in a contiguous buffer and st ore in noncontiguous locations

This tutorial focuses on the construction and use of derived datatypes, why, how, and when to use them.


 

2. Why Use Derived Datatypes?

 

2.1 Basic MPI Datatypes

 

MPI Basic predefined Datatypes for C

MPI datatype C datatype
MPI_CHAR signed char
MPI_SHORT signed short int
MPI_INT signed int
MPI_LONG signed long int
MPI_UNSIGNED_CHAR unsigned char
MPI_UNSIGNED_SHORT unsigned short int
MPI_UNSIGNED_LONG unsigned long_int
MPI_UNSIGNED unsigned int
MPI_FLOAT float
MPI_DOUBLE double
MPI_LONG_DOUBLE long double
MPI_BYTE  
MPI_PACKED  

 

MPI Basic predefined Datatypes for FORTRAN

MPI datatype FORTRAN datatype
MPI_INTEGER INTEGER
MPI_REAL REAL
MPI_REAL8 REAL*8
MPI_DOUBLE_PRECISION DOUBLE PRECISION
MPI_COMPLEX COMPLEX
MPI_LOGICAL LOGICAL
MPI_CHARACTER CHARACTER
MPI_BYTE  
MPI_PACKED  

Given these datatypes and a count, you can handle messages of contiguous data of the same type.

 

2.2 Motivation

What if you want to specify:

  • non-contiguous data of a single type?
  • contiguous data of mixed types?
  • non-contiguous data of mixed types?

A few possible solutions are that you could:

  • make multiple MPI calls to send and receive each data element
  • use MPI_PACKED to send data that has been explicitly packed and/or MPI_UNPACKED to receive data that has been explicitly unpacked.
  • use MPI_BYTE to get around the datatype-matching rules. Like MPI_PACKED, MPI_BYTE can be used to match any byte of storage (on a byte-addressable machine), irrespective of the datatype of the variable that contains this byte.

Generally, however, these solutions are slow, clumsy, and wasteful of memory. Using MPI_BYTE or MPI_PACKED might also result in a program that isn't portable to a heterogeneous system of machines.

The idea of MPI derived datatypes is to provide a portable and efficient way of communicating non-contiguous or mixed types in a message. MPI derived datatypes provide a simpler, cleaner, more elegant and efficient way to handle this type of data.

 


3. What are Derived Datatypes?

Derived datatypes are datatypes that are built from the basic MPI datatypes. To better understand what is needed to construct such a datatype, you need to understand the general concept of an MPI datatype and something called a typemap.

Formally, the MPI Standard defines a general datatype as an object that specifies two things:

  • a sequence of basic datatypes
  • a sequence of integer (byte) displacements

An easy way to represent such an object is as a sequence of pairs of basic datatypes and displacements. MPI calls this sequence a typemap.

 


4. When and How Do I Use Derived Datatypes?

 

4.1 When to Use

When you want to create a datatype in C or FORTRAN, you do so by declaring the datatype before executing any statements. Your declarations are read by the compiler that sets up storage for your datatype. In contrast, MPI derived datatypes are created at run-time through calls to MPI library routines. Since MPI derived datatypes are often used to send or receive C or FORTRAN datatypes, in the typical scenario, you first declare your C or FORTRAN datatypes. Later, in the execution part of your program between calls to MPI_INIT and MPI_FINALIZE, you create and use your MPI derived datatypes.

 

4.2 How to Use

Before you can use a derived datatype, you must create it. Here are the steps you take:

  1. Construct the datatype.
  2. Allocate the datatype.
  3. Use the datatype.
  4. Deallocate the datatype.

You must construct and allocate a datatype before using it. On the other hand, once you have it constructed and allocated, you are not required to use or deallocate it.

 

4.2.1 Construct the datatype

MPI_Type_contiguous
The simplest constructor. Produces a new datatype by making count copies of an existing data type.
 
MPI_Type_vector
MPI_Type_hvector
Similar to contiguous, but allows for regular gaps (stride) in the displacements.
MPI_Type_hvector is identical to MPI _Type_vector except that stride is specified in bytes.
 
MPI_Type_indexed
MPI_Type_hindexed
An array of displacements of the input data type is provided as the map for the new data type.
MPI_Type_hindexed is identical to MPI_Type_indexed except that offsets are specified in bytes.
 
MPI_Type_struct
The most general of all derived datatypes. The new data type is formed according to
completely defined map of the component data types.

4.2.2 Allocate the datatype

A constructed datatype must be committed to the system before it can be used in a communication. The constructed datatype is committed with a call to MPI_TYPE_COMMIT. (There is no need to commit basic datatypes; they are pre-committed.) It can then be used in any number of communications. The syntax of MPI_TYPE_COMMIT is:

  • C
    int MPI_Type_commit (MPI_datatype *datatype)
    
  • FORTRAN
    MPI_TYPE_COMMIT (DATATYPE, MPIERROR)
      INTEGER DATATYPE, MPIERROR
    

 

4.2.3 Use the datatype

Derived datatypes can be used in all send and receive operations. You simply use the handle to the derived datatype as an argument in a send or receive operation instead of a basic datatype argument. Here is a sample C code segment:

MPI_Type_vector(count, blocklength, stride, oldtype, &newtype);
MPI_Type_commit (&newtype);
MPI_Send(buffer, 1, newtype, dest, tag, comm);

 

4.2.4 Deallocate the datatype

Finally, there is a complementary routine to MPI_TYPE_COMMIT, namely, MPI_TYPE_FREE, which marks a datatype for deallocation. The syntax of MPI_TYPE_FREE is:

  • C
    int MPI_Type_free (MPI_datatype *datatype)
    
  • FORTRAN
    MPI_TYPE_FREE (DATATYPE, MPIERROR)
      INTEGER DATATYPE, MPIERROR
    

 


5. DERIVED DATATYPES

This section presents the MPI functions for constructing derived datatypes.

 

5.1 CONTIGUOUS

    C       : MPI_Type_contiguous (count, oldtype, *newtype)
    Fortran : MPI_TYPE_CONTIGUOUS (count, oldtype, newtype,ierr)
    • IN count Number of blocks to be added
      IN oldtype Datatype of each element
      OUT newtype Handle (pointer) for new derived type
      OUT ierr reporting the success or failure

      MPI_TYPE_CONTIGOUS constructs a typemap consisting of the replication of a datatype into contiguous locations. newtype is the datatype obtained by concatenating count copies of oldtype.

    Example

    • C example
    • Fortran example
    • Sample program output:
      rank= 0 b= 1.0 2.0 3.0 4.0
      rank= 1 b= 5.0 6.0 7.0 8.0
      rank= 2 b= 9.0 10.0 11.0 12.0
      rank= 3 b= 13.0 14.0 15.0 16.0

    5.2 VECTOR & HVECTOR

        C       : MPI_Type_vector (count,blocklength,stride,oldtype,*newtype)
                  MPI_Type_hvector (count,blocklength,stride,oldtype,*newtype)
        Fortran : MPI_TYPE_VECTOR (count, blocklength, stride, oldtype, newtype, ierr)
                  MPI_TYPE_HECTOR (count, blocklength, stride, oldtype, newtype, ierr)
    
      • IN count Number of blocks to be added
        IN blocklen Number of elements in block
        IN stride Number of elements (NOT bytes) between start of each block
        IN oldtype Datatype of each element
        OUT newtype Handle (pointer) for new derived type

      The Vector constructor is similar to contiguous, but allows for regular gaps or overlaps (stride) in the displacements.

      Hvector: MPI_Type_hvector (in C) and MPI_TYPE_HVECTOR (in Fortran), respectively, are the same as those for MPI_TYPE_VECTOR given above, except that displacement stride is specified in bytes rather than by length.

      Example

      • C example
      • Fortran example
      • Sample program output:
        rank= 0 b= 1.0 5.0 9.0 13.0
        rank= 1 b= 2.0 6.0 10.0 14.0
        rank= 2 b= 3.0 7.0 11.0 15.0
        rank= 3 b= 4.0 8.0 12.0 16.0

      5.3 INDEXED & HINDEXED

          C       : MPI_Type_indexed (count, blocklens[], offsets[], old_type,*newtype)
                    MPI_Type_hindexed (count, blocklens[], offsets[], old_type, *newtype)
          Fortran : MPI_TYPE_INDEXED  (count, blocklens(), offsets(), old_type(),newtype, ierr)
                    MPI_TYPE_HINDEXED  (count, blocklens(), offsets(), old_type(),newtype, ierr)
      
        • IN count Number of blocks to be added
          IN blocklens Number of elements in block -- an array of length count
          IN offsets Displacements (an array of length count) for each block
          IN oldtype Datatype of each element
          OUT newtype Handle (pointer) for new derived type

        This constructor replicates a datatype, taking blocks at different offsets. It allows one to specify a noncontiguous data layout where displacements between successive blocks need not be equal.

        Hindexed: MPI_Type_hindexed (in c) and MPI_TYPE_HINDEXED (in FORTRAN), respectively, are the same as those for MPI_TYPE_INDEXED given above, except that offsets array is specified in bytes.

        Example

        • C example
        • Fortran example
        • Sample program output:
          
          
          rank= 0 b= 6.0 7.0 8.0 9.0 13.0 14.0
          rank= 1 b= 6.0 7.0 8.0 9.0 13.0 14.0
          rank= 2 b= 6.0 7.0 8.0 9.0 13.0 14.0
          rank= 3 b= 6.0 7.0 8.0 9.0 13.0 14.0

        5.4 STRUCT

            C       : MPI_Type_struct (count, blocklens[], offsets[], old_types[], *newtype)
            Fortran : MPI_TYPE_STRUCT (count,  blocklens(),  offsets(),  old_type(),  newtype,  ierr)
         
          • IN count Number of blocks to be added
            IN blocklens Number of elements in block -- an array of length count
            IN offsets Displacements (an array of length count) for each block
            IN oldtype Datatype of each element
            OUT newtype Handle(pointer) for new derived type

          To gather a mix of different datatypes scattered at many locations in space into one datatype that can be used for the communication.

          Example

          • C example
          • Fortran example
          • Sample program output:
            rank= 0 3.00 -3.00 3.00 0.25 3 1
            rank= 2 3.00 -3.00 3.00 0.25 3 1
            rank= 1 3.00 -3.00 3.00 0.25 3 1
            rank= 3 3.00 -3.00 3.00 0.25 3 1

           

          6. Conclusions

           

          • Derived datatypes provide a portable and elegant way of communicating non-contiguous or mixed types in a message.
          • Derived datatypes should provide an efficient method of sending data since the data can be moved from its location from one processor memory to a location on a different processor memory without any intermediate buffering.
          • Derived datatypes are datatypes that are built from the basic MPI datatypes.
          • Derived datatypes provide a template of the data that is to be sent. All the data in the datatype is identified by its offset from the base address. The base address is the address passed to the MPI routine using the derived datatype. This allows the same MPI datatype to be used for any number of variables of the same form.
          • MPI provides a number of different routines for creating derived datatypes, each aimed at certain types of data, i.e., contiguous data, non-contiguous data, and non-contiguous mixed data.
          • Every derived datatype must be committed before it can be used.
          • The MPI routine MPI_TYPE_EXTENT is useful for calculating displacements as it takes into account any alignment issues.

            extent(Typemap) = ub(Typemap) - lb(Typemap) + pad


            where ub is the upper bound (location of the last byte), lb is the lower bound (the location of the first byte) and pad, a concept used by MPI to require the address of a variable (in bytes) to be a multiple of its length (in bytes).

            Here is the syntax for the extent routine:

            • C
              int MPI_Type_extent(MPI_Datatype datatype, MPI_Aint *extent)
              
            • FORTRAN
              MPI_TYPE_EXTENT(DATATYPE, EXTENT, MPIERROR)
                  INTEGER DATATYPE, EXTENT, MPIERROR
              

            where

            • datatype is an input datatype handle
            • extent is an output integer for FORTRAN, or for C, a special integer type MPI_Aint, that can hold an arbitrary address

             

          • A received message does not need to fill the entire receive buffer. MPI defines two routines to help you handle this situation. The MPI_GET_COUNT routine returns the number of received elements of the datatype specified in the receive call. If you want to find out how many basic elements within this datatype were received, use the MPI_GET_ELEMENTS routine.

             

            • C
              int MPI_Get_count(MPI_Status *status,
                                MPI_Datatype datatype, int *count)
              int MPI_Get_elements(MPI_Status *status,
                                MPI_Datatype datatype, int *count)
              
            • FORTRAN
              MPI_GET_COUNT(STATUS, DATATYPE, COUNT, MPIERROR)
              MPI_GET_ELEMENTS(STATUS, DATATYPE, COUNT, MPIERROR)
                 INTEGER STATUS(MPI_STATUS_SIZE), DATATYPE, COUNT, MPIERROR
              

            where MPI_Status is an input variable specifying the status of the receive operation.


          7. References