Communicator for asynchronous communication of MPISparseBlockMat. More...

Public Member Functions
	MPIKroneckerGather (MPI_Comm comm=MPI_COMM_NULL)
	no communication

	MPIKroneckerGather (unsigned left_size, const std::map< int, thrust::host_vector< int > > &recvIdx, unsigned n, unsigned num_cols, unsigned right_size, MPI_Comm comm_1d)
	Construct from communication pattern.

template<template< typename > typename OtherVector>
	MPIKroneckerGather (const MPIKroneckerGather< OtherVector > &src)
	Construct from other execution policy.

MPI_Comm	communicator () const
	The internal MPI communicator used.

unsigned	buffer_size () const
	Number of pointers in receive buffer equals number of indices in `recvIdx`.

unsigned	chunk_size () const
	`nleft_sizeright_size`

bool	isCommunicating () const
	True if the gather/scatter operation involves actual MPI communication.

template<class ContainerType >
void	global_gather_init (const ContainerType &gatherFrom) const
	\( w' = P_{G,MPI} G_2 v\). Globally (across processes) asynchronously gather data into a buffer

template<class ContainerType >
void	global_gather_wait (const ContainerType &gatherFrom, Vector< const dg::get_value_type< ContainerType > * > &buffer_ptrs) const
	Wait for asynchronous communication to finish and gather received data into buffer and return pointers to it.

Friends
template<template< class > class OtherVector>
class	MPIKroneckerGather
	Enable copy from different Vector type.

Detailed Description

template<template< typename > typename Vector>
struct dg::MPIKroneckerGather< Vector >

Communicator for asynchronous communication of MPISparseBlockMat.

This is a version of MPIGather that is optimised for the Kronecker type communication pattern ("Hyperblocks") present in our EllSparseBlockMat It avoids data movement to the greatest possible extent and exposes send and receive buffers via pointers.

Imagine a communicator with Cartesian topology and further imagine that the grid topology is also Cartesian (vectors form a box) in Nd dimensions. A Sparseblockmat typically requires to gather slices of given index from other processes in a 1d communicator. This class provides pointers to these other indices. The pointers either reference data in an internal communication buffer (since it involves communciation to get the layers from neighboring processes) another buffer (if mpi communication requires to reorder input data) or the input vector itself (if the communication goes along the last dimension there is no need to reorder, in fact, here is the main gain we get from the pointer approach, we save on unnecessary data copies, which might be significant in cases where the communication to computation ratio is high). The size of the data each pointer references is the halo size, buffer_size()

The communication is done asynchronously i.e. the user can initiate the communication and signal when the results are needed at a later stage.

Note: If the number of neighboring processes in the given direction is 1, the buffer size is 0 and all members return immediately.; the pointers may alias each other (if the input contains less than 4 layers); the corresponding gather map is of general type and the communication can also be modeled in MPIGather

Template Parameters

Vector a thrust Vector e.g. thrust::host_vector or thrust::device_vector Determines the internal buffer type of the \( G_2\) gather operation

See also: dg::RowColDistMat

Constructor & Destructor Documentation

◆ MPIKroneckerGather() [1/3]

template<template< typename > typename Vector>

dg::MPIKroneckerGather< Vector >::MPIKroneckerGather ( MPI_Comm comm = MPI_COMM_NULL )

inline

no communication

Parameters

comm	optional MPI communicator: the purpose is to be able to store MPI communicator even if no communication is involved in order to construct MPI_Vector with it

◆ MPIKroneckerGather() [2/3]

template<template< typename > typename Vector>

dg::MPIKroneckerGather< Vector >::MPIKroneckerGather	(	unsigned	left_size,
		const std::map< int, thrust::host_vector< int > > &	recvIdx,
		unsigned	n,
		unsigned	num_cols,
		unsigned	right_size,
		MPI_Comm	comm_1d )

inline

Construct from communication pattern.

Parameters

left_size	(local) left size in `EllSparseBlockMat` (determines chunk size)
recvIdx	1d block index in units of chunk size. `recvIdx[PID]` contains the block indices on rank PID (in `comm_1d`) that the calling rank receives. The `global_gather_wait` function returns one pointer for each element of `recvIdx` pointing to a block of memory of size `chunk_size=nleft_sizeright_size`.
n	block size of `EllSparseBlockMat` (determines chunk size)
num_cols	local number of blocks columns in `EllSparseBlockMat`
right_size	(local) right size of `EllSparseBlockMat` (determines chunk size)
comm_1d	the one dimensional Cartesian communicator along which to exchange the hyperblocks

Note: not all participating pids need to have the same number of hyperblocks or have the same shape even though left_size, n, num_cols, and right_size must be equal between any two communicating pids

See also: dg::make_mpi_sparseblockmat

◆ MPIKroneckerGather() [3/3]

template<template< typename > typename Vector>

template<template< typename > typename OtherVector>

dg::MPIKroneckerGather< Vector >::MPIKroneckerGather ( const MPIKroneckerGather< OtherVector > & src )

inline

Construct from other execution policy.

This makes it possible to construct an object on the host and then copy everything on to a device

Template Parameters

OtherVector other container type

Parameters

src	source object

Member Function Documentation

◆ buffer_size()

template<template< typename > typename Vector>

unsigned dg::MPIKroneckerGather< Vector >::buffer_size ( ) const

inline

Number of pointers in receive buffer equals number of indices in recvIdx.

Returns: buffer size (may be different for each process)

Note: may return 0, which just means that the calling rank does not receive any data from any other rank including itself. The calling rank may still need to send data in global_gather_init

Attention: It is therfore not valid to check for zero buffer size if you want to find out whether a given rank needs to send MPI messages or not. The right way to do it is to call isCommunicating()

See also: isCommunicating()

◆ chunk_size()

template<template< typename > typename Vector>

unsigned dg::MPIKroneckerGather< Vector >::chunk_size ( ) const

inline

n*left_size*right_size

◆ communicator()

template<template< typename > typename Vector>

MPI_Comm dg::MPIKroneckerGather< Vector >::communicator ( ) const

inline

The internal MPI communicator used.

Returns: MPI Communicator

◆ global_gather_init()

template<template< typename > typename Vector>

template<class ContainerType >

void dg::MPIKroneckerGather< Vector >::global_gather_init ( const ContainerType & gatherFrom ) const

inline

\( w' = P_{G,MPI} G_2 v\). Globally (across processes) asynchronously gather data into a buffer

Template Parameters

ContainerType

Can be any shared vector container on host or device, e.g.

thrust::host_vector<double>
thrust::device_vector<double>
thrust::device_vector<thrust::complex<double>>

Parameters

gatherFrom source vector v; data is collected from this vector

Note: If !isCommunicating() then this call will not involve MPI communication but will still gather values according to the given index map

◆ global_gather_wait()

template<template< typename > typename Vector>

template<class ContainerType >

void dg::MPIKroneckerGather< Vector >::global_gather_wait	(	const ContainerType &	gatherFrom,
		Vector< const dg::get_value_type< ContainerType > * > &	buffer_ptrs ) const

inline

Wait for asynchronous communication to finish and gather received data into buffer and return pointers to it.

Call MPI_Waitall on internal MPI_Request variables and manage host memory in case of cuda-unaware MPI. After this call returns it is safe to use the buffer and the gatherFrom variable from the corresponding global_gather_init call

Parameters

gatherFrom	source vector v; data is collected from this vector
buffer_ptrs	(write only) pointers coresponding to `recvIdx` from the constructor

◆ isCommunicating()

template<template< typename > typename Vector>

bool dg::MPIKroneckerGather< Vector >::isCommunicating ( ) const

inline

True if the gather/scatter operation involves actual MPI communication.

This is more than just a test for zero message size. This is because even if a process has zero message size indicating that it technically does not need to send any data at all it might still need to participate in an MPI communication (sending an empty message to indicate that a certain point in execution has been reached). Only if none of the processes in the process group has anything to send will this function return false. This test can be used to avoid the gather operation alltogether in e.g. the construction of a MPI distributed matrix.

Note: this check involves MPI communication itself, because a process needs to check if itself or any other process in its group is communicating.

Returns: False, if the global gather can be done without MPI communication (i.e. the indices are all local to each calling process), or if the communicator is MPI_COMM_NULL. True else.

See also: buffer_size()

Friends And Related Symbol Documentation

◆ MPIKroneckerGather

template<template< typename > typename Vector>

template<template< class > class OtherVector>

friend class MPIKroneckerGather

friend

Enable copy from different Vector type.

The documentation for this struct was generated from the following file:

backend/mpi_gather_kron.h

Public Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

◆ MPIKroneckerGather() [1/3]

◆ MPIKroneckerGather() [2/3]

◆ MPIKroneckerGather() [3/3]

Member Function Documentation

◆ buffer_size()

◆ chunk_size()

◆ communicator()

◆ global_gather_init()

◆ global_gather_wait()

◆ isCommunicating()

Friends And Related Symbol Documentation

◆ MPIKroneckerGather