Communicator for asynchronous nearest neighbor communication. More...

Public Types
using	container_type = Vector

using	buffer_type = Buffer

using	pointer_type = get_value_type< Vector > *

using	const_pointer_type = get_value_type< Vector > const *

Public Member Functions
	NearestNeighborComm (MPI_Comm comm=MPI_COMM_NULL)
	no communication More...

	NearestNeighborComm (unsigned n, const unsigned vector_dimensions[3], MPI_Comm comm, unsigned direction)
	Construct. More...

template<class OtherIndex , class OtherBuffer , class OtherVector >
	NearestNeighborComm (const NearestNeighborComm< OtherIndex, OtherBuffer, OtherVector > &src)
	Construct from other Communicator. More...

unsigned	n () const
	halo size More...

const unsigned *	dims () const
	The dimensionality of the input vector. More...

unsigned	direction () const
	The direction of communication. More...

MPI_Comm	communicator () const
	The internal MPI communicator used. More...

Buffer	allocate_buffer () const
	Allocate a buffer object. More...

unsigned	buffer_size () const
	The size of the halo. More...

bool	isCommunicating () const
	True if the gather/scatter operation involves actual MPI communication. More...

int	map_index (int i) const
	Map a local matrix index to a buffer index. More...

void	global_gather_init (const_pointer_type input, buffer_type &buffer, MPI_Request rqst[4]) const
	Gather values from given Vector and initiate asynchronous MPI communication. More...

void	global_gather_wait (const_pointer_type input, const buffer_type &buffer, MPI_Request rqst[4]) const
	Wait for asynchronous communication to finish and gather received data into buffer. More...

Detailed Description

template<class Index, class Buffer, class Vector>
struct dg::NearestNeighborComm< Index, Buffer, Vector >

Communicator for asynchronous nearest neighbor communication.

Imagine a communicator with Cartesian topology and further imagine that the grid topology is also Cartesian (vectors form a box) in two or three dimensions. In each direction this box has a boundary layer (the halo) of a depth given by the user. Each boundary layer has two neighboring layers, one on the same process and one lying on the neighboring process. What this class does is to provide you with six pointers to each of these six layers (three on each side). The pointers either reference data in an internal communication buffer (since it involves communciation to get the layers from neighboring processes) another buffer (if mpi communication requires to reorder input data) or the input vector itself (if the communication goes along the last dimension there is no need to reorder, in fact, here is the main gain we get from the pointer approach, we save on unnecessary data copies, which might be significant in cases where the communication to computation ratio is high). The size of the data each pointer references is the halo size, buffer_size()

The communication is done asynchronously i.e. the user can initiate the communication and signal when the results are needed at a later stage.

Note: If the number of neighboring processes in the given direction is 1, the buffer size is 0 and all members return immediately.; the pointers may alias each other (if the input contains less than 4 layers); the corresponding gather map is of general type and the communication can also be modeled in GeneralComm, but not BijectiveComm or SurjectiveComm

Attention: Currently we cannot handle the case where the whole vector is the boundary layer (i.e. buffer_size()==input.size() and both neighboring layers are on different processes)

Template Parameters

Index	the type of index container (must be either thrust::host_vector<int> or thrust::device_vector<int>)
Buffer	the container for the pointers to the buffer arrays
Vector	the vector container type must have a resize() function and work in the thrust library functions ( i.e. must a thrust::host_vector or thrust::device_vector)

See also: dg::RowColDistMat

Member Typedef Documentation

◆ buffer_type

template<class Index , class Buffer , class Vector >

using dg::NearestNeighborComm< Index, Buffer, Vector >::buffer_type = Buffer

◆ const_pointer_type

template<class Index , class Buffer , class Vector >

using dg::NearestNeighborComm< Index, Buffer, Vector >::const_pointer_type = get_value_type<Vector> const *

◆ container_type

template<class Index , class Buffer , class Vector >

using dg::NearestNeighborComm< Index, Buffer, Vector >::container_type = Vector

◆ pointer_type

template<class Index , class Buffer , class Vector >

using dg::NearestNeighborComm< Index, Buffer, Vector >::pointer_type = get_value_type<Vector>*

Constructor & Destructor Documentation

◆ NearestNeighborComm() [1/3]

template<class Index , class Buffer , class Vector >

dg::NearestNeighborComm< Index, Buffer, Vector >::NearestNeighborComm ( MPI_Comm comm = MPI_COMM_NULL )

inline

no communication

Parameters

comm	optional MPI communicator: the purpose is to be able to store MPI communicator even if no communication is involved in order to construct MPI_Vector with it

◆ NearestNeighborComm() [2/3]

template<class Index , class Buffer , class Vector >

dg::NearestNeighborComm< Index, Buffer, Vector >::NearestNeighborComm	(	unsigned	n,
		const unsigned	vector_dimensions[3],
		MPI_Comm	comm,
		unsigned	direction
	)

inline

Construct.

Parameters

n	depth of the halo
vector_dimensions	{x, y, z} dimension (total number of points)
comm	the (cartesian) communicator
direction	0 is x, 1 is y, 2 is z

◆ NearestNeighborComm() [3/3]

template<class Index , class Buffer , class Vector >

template<class OtherIndex , class OtherBuffer , class OtherVector >

dg::NearestNeighborComm< Index, Buffer, Vector >::NearestNeighborComm ( const NearestNeighborComm< OtherIndex, OtherBuffer, OtherVector > & src )

inline

Construct from other Communicator.

Simply copies halo size, dimensions, communicator and direction and constructs a new object

Template Parameters

OtherIndex	other index type
OtherVector	other container type

Parameters

src	source object

Member Function Documentation

◆ allocate_buffer()

template<class Index , class Buffer , class Vector >

Buffer dg::NearestNeighborComm< Index, Buffer, Vector >::allocate_buffer ( ) const

inline

Allocate a buffer object.

The buffer object is only a colletion of pointers to the actual data

Returns: a buffer object on the stack

Note: if buffer_size()==0 the default constructor of Buffer is called

◆ buffer_size()

template<class Index , class Buffer , class Vector >

unsigned dg::NearestNeighborComm< Index, Buffer, Vector >::buffer_size ( ) const

The size of the halo.

Returns: the size of the halo (0 if no communication)

◆ communicator()

template<class Index , class Buffer , class Vector >

MPI_Comm dg::NearestNeighborComm< Index, Buffer, Vector >::communicator ( ) const

inline

The internal MPI communicator used.

can be e.g. used to assert that communicators of matrix and vector are the same

Returns: MPI Communicator

◆ dims()

template<class Index , class Buffer , class Vector >

const unsigned * dg::NearestNeighborComm< Index, Buffer, Vector >::dims ( ) const

inline

The dimensionality of the input vector.

Returns: dimensions ( 3)

◆ direction()

template<class Index , class Buffer , class Vector >

unsigned dg::NearestNeighborComm< Index, Buffer, Vector >::direction ( ) const

inline

The direction of communication.

Returns: direction

◆ global_gather_init()

template<class Index , class Buffer , class Vector >

void dg::NearestNeighborComm< Index, Buffer, Vector >::global_gather_init	(	const_pointer_type	input,
		buffer_type &	buffer,
		MPI_Request	rqst[4]
	)		const

inline

Gather values from given Vector and initiate asynchronous MPI communication.

Parameters

input	from which to gather data (it is unsafe to change values on return)
buffer	(write only) pointers to the received data after `global_gather_wait()` was called (must be allocated by `allocate_buffer()`)
rqst	four request variables that can be used to call MPI_Waitall

◆ global_gather_wait()

template<class Index , class Buffer , class Vector >

void dg::NearestNeighborComm< Index, Buffer, Vector >::global_gather_wait	(	const_pointer_type	input,
		const buffer_type &	buffer,
		MPI_Request	rqst[4]
	)		const

inline

Wait for asynchronous communication to finish and gather received data into buffer.

Calls MPI_Waitall on the rqst variables and may do additional cleanup. After this call returns it is safe to use data the buffer points to.

Parameters

input	from which to gather data (it is safe to change values on return since values to communicate are copied into `buffer`)
buffer	(write only) where received data resides on return (must be allocated by `allocate_buffer()`)
rqst	the same four request variables that were used in global_gather_init

◆ isCommunicating()

template<class Index , class Buffer , class Vector >

bool dg::NearestNeighborComm< Index, Buffer, Vector >::isCommunicating ( ) const

inline

True if the gather/scatter operation involves actual MPI communication.

This is more than just a test for zero message size. This is because even if a process has zero message size indicating that it technically does not need to send any data at all it might still need to participate in an MPI communication (sending an empty message to indicate that a certain point in execution has been reached). Only if NONE of the processes in the process group has anything to send will this function return false. This test can be used to avoid the gather operation alltogether in e.g. the construction of a MPI distributed matrix.

Note: this check may involve MPI communication itself, because a process needs to check if itself or any other process in its group is communicating.

Returns: False, if the global gather can be done without MPI communication (i.e. the indices are all local to each calling process), or if the communicator is MPI_COMM_NULL. True else.

See also: buffer_size()

◆ map_index()

template<class Index , class Buffer , class Vector >

int dg::NearestNeighborComm< Index, Buffer, Vector >::map_index ( int i ) const

inline

Map a local matrix index to a buffer index.

Parameters

i	matrix index

Returns: buffer index (0,1,...,5)

◆ n()

template<class Index , class Buffer , class Vector >

unsigned dg::NearestNeighborComm< Index, Buffer, Vector >::n ( ) const

inline

halo size

Returns: halo size

The documentation for this struct was generated from the following file:

backend/mpi_vector.h

Public Types

Public Member Functions

Detailed Description

Member Typedef Documentation

◆ buffer_type

◆ const_pointer_type

◆ container_type

◆ pointer_type

Constructor & Destructor Documentation

◆ NearestNeighborComm() [1/3]

◆ NearestNeighborComm() [2/3]

◆ NearestNeighborComm() [3/3]

Member Function Documentation

◆ allocate_buffer()

◆ buffer_size()

◆ communicator()

◆ dims()

◆ direction()

◆ global_gather_init()

◆ global_gather_wait()

◆ isCommunicating()

◆ map_index()

◆ n()