- Generated on Wed Sep 20 2023 15:18:50 for Discontinuous Galerkin Library by
1.9.3
|
Discontinuous Galerkin Library
#include "dg/algorithm.h"
|
Struct that performs collective scatter and gather operations across processes on distributed vectors using MPI. More...
Public Types | |
| using | value_type = get_value_type< LocalContainer > |
| reveal value type More... | |
| using | container_type = LocalContainer |
| reveal local container type More... | |
Public Member Functions | |
| LocalContainer | allocate_buffer () const |
Allocate a buffer object of size buffer_size() More... | |
| void | global_gather (const value_type *values, LocalContainer &buffer) const |
| \( w = G v\). Globally (across processes) gather data into a buffer More... | |
| LocalContainer | global_gather (const value_type *values) const |
| \( w = G v\). Globally (across processes) gather data into a buffer (memory allocating version) More... | |
| void | global_scatter_reduce (const LocalContainer &toScatter, value_type *values) const |
| \( v = G^\mathrm{T} w\). Globally (across processes) scatter data accross processes and reduce on multiple indices More... | |
| unsigned | buffer_size () const |
| The local size of the buffer vector w = local map size. More... | |
| unsigned | local_size () const |
The local size of the source vector v = local size of the dg::MPI_Vector More... | |
| bool | isCommunicating () const |
| True if the gather/scatter operation involves actual MPI communication. More... | |
| MPI_Comm | communicator () const |
| The internal MPI communicator used. More... | |
| virtual aCommunicator * | clone () const =0 |
| Generic copy method. More... | |
| virtual | ~aCommunicator () |
| vritual destructor More... | |
Protected Member Functions | |
| aCommunicator (unsigned local_size=0) | |
| only derived classes can construct More... | |
| aCommunicator (const aCommunicator &src) | |
| only derived classes can copy More... | |
| aCommunicator & | operator= (const aCommunicator &src) |
| only derived classes can assign More... | |
| void | set_local_size (unsigned new_size) |
| Set the local size of the source vector v. More... | |
Struct that performs collective scatter and gather operations across processes on distributed vectors using MPI.
In order to understand what this class does you should first really(!) understand what gather and scatter operations are, so grab pen and paper:
First, we note that gather and scatter are most often used in the context of memory buffers. The buffer needs to be filled wih values (gather) or these values need to be written back into the original place (scatter).
Gather: imagine a buffer vector w and a map (the "gather map") that gives to every element in this vector w an index into a source vector v where the value of this element should be taken from i.e. \( w[i] = v[\text{idx}[i]] \) Note that an index into the source vector v can appear several times or not at all. This is why the source vector v can have any size and even be smaller than w.
Scatter: imagine a buffer vector w and a map that gives to every element in the buffer w an index into a target vector v where this element should go to. Note again that an index into v can appear several times or never at all. If the index appears more than once, we perform a reduction operation (we sum up all elements) on these indices initializing the sum with 0. Note that \( v[\text{idx}[i]] = w[i] \) is an INCORRECT definition of this, because it does not show the reduction.
It is more accurate to represent the gather and scatter operation by a matrix. The gather matrix \( G\) is just a (permutation) matrix of 1's and 0's with exactly one "1" in each line. In a "coo" formatted sparse matrix format the values array would consist only of "1"s, the row array is just the index and the column array is the gather map.
\[ \vec w = G \vec v \]
where \( G \in \mathbb{R}^{N_w \times N_v}\) and \(N_w\) and \( N_v\) can have any sizes. We have \( G_{ij} = \delta_{\text{idx}_ij}\) The above definition uniquely identifies the scatter matrix as the transpose of the gather matrix of the same index map
\[ S\equiv G^\mathrm{T}\]
We have \( S_{ij} = \delta_{\text{idx}_j i}\) and \( \vec v = S \vec w\). A simple consistency test is given by \( (Gv)\cdot (Gv) = S(Gv)\cdot v\).
The scatter matrix can thus have zero, one or more "1"s in each line.
We distinguish between
bijective: If the gather map idx[i] is bijective, each element of the source vector v maps to exactly one location in the buffer vector w. In this case the scatter matrix S is the inverse of G and v and w have the same size.
surjective: If the gather map idx[i] is surjective, each element of the source vector v maps to at least one location in the buffer vector w. This means that the scatter matrix S can have more than one 1's in each line and w has at least the size of v.
general: In general the gather map idx[i] might or might not map an element of the source vector v. This means that the scatter matrix S can have one or more empty lines and w may be smaller than v.
\[ \vec{\text{idx}} = G \vec i\]
In case the index map is bijective the index map can be inverted and we have\[ \vec{\text{idx}^{-1}} = S \vec i\]
This class performs these operations for the case that v and w are distributed across processes. We always assume that the source vector v is distributed equally among processes, i.e. each process holds a chunk of v of equal size. On the other hand the local size of w may vary among processes depending on the gather/scatter map.
| LocalContainer | a container on a shared memory system (must be default constructible) |
| using dg::aCommunicator< LocalContainer >::container_type = LocalContainer |
reveal local container type
| using dg::aCommunicator< LocalContainer >::value_type = get_value_type<LocalContainer> |
reveal value type
|
inlinevirtual |
vritual destructor
|
inlineprotected |
only derived classes can construct
| local_size | the local size of the source vector v |
|
inlineprotected |
only derived classes can copy
| src | source |
|
inline |
Allocate a buffer object of size buffer_size()
buffer_size()==0 the default constructor of LocalContainer is called
|
inline |
The local size of the buffer vector w = local map size.
Consider that both the source vector v and the buffer w are distributed across processes. In Feltor the vector v is (usually) distributed equally among processes and the local size of v is the same for all processes. However, the buffer size might be different for each process.
isCommunicating()
|
pure virtual |
Generic copy method.
Implemented in dg::BijectiveComm< Index, Vector >, dg::SurjectiveComm< Index, Vector >, and dg::GeneralComm< Index, Vector >.
|
inline |
The internal MPI communicator used.
can be e.g. used to assert that communicators of matrix and vector are the same
|
inline |
\( w = G v\). Globally (across processes) gather data into a buffer (memory allocating version)
The transpose operation is global_scatter_reduce()
| values | source vector v; data is collected from this vector (must have local_size() elements) |
buffer_size() that holds the gathered data !isCommunicating() then this call will not involve MPI communication but will still gather values according to the given index map
|
inline |
\( w = G v\). Globally (across processes) gather data into a buffer
The transpose operation is global_scatter_reduce()
| values | source vector v; data is collected from this vector |
| buffer | on output holds the gathered data ( must be of size buffer_size() ) |
!isCommunicating() then this call will not involve MPI communication but will still gather values according to the given index map
|
inline |
\( v = G^\mathrm{T} w\). Globally (across processes) scatter data accross processes and reduce on multiple indices
This is the transpose operation of global_gather()
| toScatter | buffer vector w; (has to be of size given by buffer_size() ) |
| values | target vector v; on output contains values from other processes sent back to the origin (must have local_size() elements) |
!isCommunicating() then this call will not involve MPI communication but will still scatter and reduce values according to the given index map
|
inline |
True if the gather/scatter operation involves actual MPI communication.
This is more than just a test for zero message size. This is because even if a process has zero message size indicating that it technically does not need to send any data at all it might still need to participate in an MPI communication (sending an empty message to indicate that a certain point in execution has been reached). Only if NONE of the processes in the process group has anything to send will this function return false. This test can be used to avoid the gather operation alltogether in e.g. the construction of a MPI distributed matrix.
MPI_COMM_NULL. True else.
|
inline |
The local size of the source vector v = local size of the dg::MPI_Vector
Consider that both the source vector v and the buffer w are distributed across processes. In Feltor the vector v is (usually) distributed equally among processes and the local size of v is the same for all processes.
|
inlineprotected |
only derived classes can assign
| src | source |
|
inlineprotected |
Set the local size of the source vector v.
| new_size | the new local size for the source vector v |
1.9.3