Supports many-to-one and one-to-many asynchronous communication operations within a given group of processes by sending messages along the branches of a conceptual tree. More...

#include <tbox/AsyncCommGroup.h>

Inheritance diagram for SAMRAI::tbox::AsyncCommGroup:

[legend]

Classes
struct	ChildData

Public Member Functions
	AsyncCommGroup (const int nchild)
	Construct communication group. More...

virtual	~AsyncCommGroup (void)
	Destructor. More...

Communication methods
enum	BaseOp { undefined, gather, bcast, max_reduce, min_reduce, sum_reduce }
	Operations user would want to do. More...

enum	TaskOp { recv_start, recv_check, send_start, send_check, none }
	Tasks, executed in order, to complete a base operation. More...

bool	beginBcast (int *buffer, int size)
	Begin a broadcast communication. More...

bool	checkBcast ()
	Check the current broadcast communication and complete the broadcast if all MPI requests are fulfilled. More...

bool	beginGather (int *buffer, int size)
	Begin a gather communication. More...

bool	checkGather ()
	Check the current gather communication and complete the gather if all MPI requests are fulfilled. More...

bool	beginSumReduce (int *buffer, int size)
	Begin a sum reduce communication. More...

bool	checkSumReduce ()
	Check the current sum reduce communication and complete the sum reduce if all MPI requests are fulfilled. More...

bool	checkOperation ()
	Check the current communication and complete it if all MPI requests are fulfilled. More...

void	waitOperation ()
	Wait for the current operation to complete. More...

bool	isDone () const
	Whether the last communication operation has finished. More...

int	getNumberOfChildren () const

void	logCurrentState (std::ostream &co) const

virtual SAMRAI_MPI::request *	getRequestPointer () const
	Return the requests for use by this object. More...

void	checkMPIParams ()

	AsyncCommGroup (const AsyncCommGroup &r)
	Operation disallowed due to primitive internal memory management. More...

AsyncCommGroup &	operator= (const AsyncCommGroup &r)
	Operation disallowed by primitive internal memory management. More...

bool	beginReduce ()
	Begin a generic reduce communication. More...

bool	checkReduce ()
	Check the current gather communication. More...

void	reduceData (int output, int data) const
	Perform reduction on data that after it has been brought to the local process. More...

void	computeDependentData (const Array< int > &group)
	Compute the data that depends on the group definition. More...

void	resetStatus ()

Mappings between array indices, group positions and process ranks
const int	d_nchild
	Number of children per node. More...

int	d_idx
	Index of the local process in d_group_ranks. More...

int	d_root_idx
	Index of the root process in d_group_ranks. More...

int	d_group_size

int	d_parent_rank
	Rank of parent process in the group. More...

ChildData *	d_child_data
	Data on each child branch. More...

int	d_branch_size_totl
	Total of all branch sizes. More...

BaseOp	d_base_op
	Operation being performed. More...

TaskOp	d_next_task_op
	Next task in a current communication operation. More...

int *	d_external_buf
	External data input and output buffer. More...

int	d_external_size
	Size of d_external_buf. More...

Array< int >	d_internal_buf
	Internal buffer. More...

SAMRAI_MPI::request *	d_internal_requests
	Requests managed internally. More...

int	d_mpi_tag

SAMRAI_MPI::comm	d_mpi_communicator

bool	d_use_blocking_send_to_children

bool	d_use_blocking_send_to_parent

SAMRAI_MPI::status	d_mpi_status

int	d_mpi_err

Array< int >	d_group_ranks
	Array of process ranks in the group. More...

static Pointer< Timer >	t_reduce_data

static Pointer< Timer >	t_wait_all

int	toPosition (int index) const
	Convert the array index to the position. More...

int	toIndex (int position) const
	Convert the position to the array index. More...

int	toChildPosition (int parent_pos, int ic) const
	Compute the position of child child_id of a parent (whether or not that child really exists). More...

int	toOldest (int parent_pos) const
	Compute the oldest (lowest position) child position of a given position (whether or not that child really exists). More...

int	toYoungest (int parent_pos) const
	Compute the youngest (highest position) child position of a given position (whether or not that child really exists). More...

static void	freeTimers ()

Detailed Description

This class was created to perform certain group communications without using MPI global communications, which require creating new MPI communicators (can be expensive) and does not support asynchronous operations.

The supported communications are asynchronous in that you can start one and wait for it or check back on it occassionally until it completes. Asynchronous operations in conjunction with other groups can be done by using a AsyncCommStage to allocate the groups and to check for completed communications.

Supported operations are currently broadcast, gather and sum reduce. Only integer data is supported.

A tree is an acyclic graph in which a node at position pos has nchild children, and the following positions for its

parent: (pos-1)/nchild
first (oldest) child: pos*nchild+1
last (youngest) child: (pos+1)*nchild

For example, nchild=2 corresponds to a binary tree.

Communication is done by sending messages toward the root (for all-to-one operations) or leaves (for one-to-all operations). For the former, we receive data from the children and send to the parent. For the latter, we receive from the parent and send to the children. Thus every communication involves a receive and a send (except at the root and leaf nodes of the tree).

Using a tree generally gives better performance than having all processes in the the tree communicate directly with the root process. Using MPI communicators corresponding to the groups may faster than using this class, but the cost of creating MPI communicators MAY be expensive.

This class supports communication and uses MPI for message passing. If MPI is disabled, the job of this class disappears and the class is effectively empty.
The public interfaces still remain so the class can compile, but the implementations are trivial.

Member Enumeration Documentation

◆ BaseOp

enum SAMRAI::tbox::AsyncCommGroup::BaseOp

private

Enumerator
undefined
gather
bcast
max_reduce
min_reduce
sum_reduce

◆ TaskOp

enum SAMRAI::tbox::AsyncCommGroup::TaskOp

private

Enumerator
recv_start
recv_check
send_start
send_check
none

Constructor & Destructor Documentation

◆ AsyncCommGroup() [1/2]

SAMRAI::tbox::AsyncCommGroup::AsyncCommGroup ( const int nchild )

The number of children per node is flexible.

Parameters

nchild Number of children per node in the group, i.e., nchild=2 is a binary tree.

◆ ~AsyncCommGroup()

virtual SAMRAI::tbox::AsyncCommGroup::~AsyncCommGroup ( void )

virtual

◆ AsyncCommGroup() [2/2]

SAMRAI::tbox::AsyncCommGroup::AsyncCommGroup ( const AsyncCommGroup & r )

inlineprivate

Member Function Documentation

◆ setGroupAndRootIndex()

void SAMRAI::tbox::AsyncCommGroup::setGroupAndRootIndex	(	const Array< int > &	group,
		const int	root_index
	)

The root is specified by dereferencing group array with root_index.

◆ setGroupAndRootRank()

void SAMRAI::tbox::AsyncCommGroup::setGroupAndRootRank	(	const Array< int > &	group,
		const int	root_rank
	)

The rank of the root is root_rank, which must be one of the ranks given in the group.

◆ setMPITag()

void SAMRAI::tbox::AsyncCommGroup::setMPITag ( const int mpi_tag )

Attention: This class is NOT (and cannot be) responsible for ensuring that the MPI communicator and tag are sufficient to select the correct messages. Please specify appropriate values for the MPI communicator and tag. Very elusive bugs can occur if incorrect messages are received.

◆ setMPICommunicator()

void SAMRAI::tbox::AsyncCommGroup::setMPICommunicator ( SAMRAI_MPI::comm & mpi_communicator )

Attention: This class is NOT (and cannot be) responsible for ensuring that the MPI communicator and tag are sufficient to select the correct messages. Please specify appropriate values for the MPI communicator and tag. Very elusive bugs can occur if incorrect messages are received. To be safe, it is best to create a new communicator to avoid interference with other communications within SAMRAI.

◆ setUseBlockingSendToParent()

void SAMRAI::tbox::AsyncCommGroup::setUseBlockingSendToParent ( const bool flag )

The default is to use blocking send to parent. Because there is just one parent, short messages can be buffered by MPI to improve the performance of blocking sends. Blocking sends need not be checked for completion.

◆ setUseBlockingSendToChildren()

void SAMRAI::tbox::AsyncCommGroup::setUseBlockingSendToChildren ( const bool flag )

The default is to use nonblocking send to children. Nonblocking sends to children are generally appropriate as there are multiple children.

◆ beginBcast()

bool SAMRAI::tbox::AsyncCommGroup::beginBcast	(	int *	buffer,
		int	size
	)

Root process of broadcast may send less data (smaller size) than receivers of broadcast, in which case the missing data is considered irrelevant by the root.

If this method returns false, checkBcast() must be called until it returns true before any change in object state is allowed.

Returns: Whether operation is completed.

◆ checkBcast()

bool SAMRAI::tbox::AsyncCommGroup::checkBcast ( )

If no communication is in progress, this call does nothing.

Returns: Whether operation is completed.

◆ beginGather()

bool SAMRAI::tbox::AsyncCommGroup::beginGather	(	int *	buffer,
		int	size
	)

Sending processes of gather may send less data (smaller size) than receivers of broadcast, in which case the missing data is considered irrelevant by the sender.

If this method returns false, checkGather() must be called until it returns true before any change in object state is allowed.

On non-root processes, buffer should contain the data to be gathered. On the root process, it should have enough space for all the data from all the processes in the group.

Parameters

buffer	Data to gather.
size	Size of data contributed by each process.

Returns: Whether operation is completed.

◆ checkGather()

bool SAMRAI::tbox::AsyncCommGroup::checkGather ( )

Returns: Whether operation is completed.

◆ beginSumReduce()

bool SAMRAI::tbox::AsyncCommGroup::beginSumReduce	(	int *	buffer,
		int	size
	)

Assume all messages are the same size.

If this method returns false, checkSumReduce() must be called until it returns true before any change in object state is allowed.

Buffer should contain the data to be gathered.

Returns: Whether operation is completed.

◆ checkSumReduce()

bool SAMRAI::tbox::AsyncCommGroup::checkSumReduce ( )

Returns: Whether operation is completed.

◆ checkOperation()

bool SAMRAI::tbox::AsyncCommGroup::checkOperation ( )

◆ waitOperation()

void SAMRAI::tbox::AsyncCommGroup::waitOperation ( )

◆ isDone()

bool SAMRAI::tbox::AsyncCommGroup::isDone ( ) const

◆ getNumberOfChildren()

int SAMRAI::tbox::AsyncCommGroup::getNumberOfChildren ( ) const

◆ logCurrentState()

void SAMRAI::tbox::AsyncCommGroup::logCurrentState ( std::ostream & co ) const

◆ getRequestPointer()

virtual SAMRAI_MPI::request* SAMRAI::tbox::AsyncCommGroup::getRequestPointer ( ) const

privatevirtual

This object allocates the MPI_Request objects it requires. However, to operate within a communication stage (such as AsyncCommStage), this method may be overiden to provide externally managed requests.

Reimplemented in SAMRAI::tbox::AsyncCommStage::StagedGroup.

◆ checkMPIParams()

void SAMRAI::tbox::AsyncCommGroup::checkMPIParams ( )

private

◆ operator=()

AsyncCommGroup& SAMRAI::tbox::AsyncCommGroup::operator= ( const AsyncCommGroup & r )

inlineprivate

◆ beginReduce()

bool SAMRAI::tbox::AsyncCommGroup::beginReduce ( )

private

This method is the workhorse underneath the public reduce methods.

If this method returns false, checkOperation() must be called until it returns true before any change in object state is allowed.

Buffer should contain the data to be gathered.

Returns: Whether operation is completed.

◆ checkReduce()

bool SAMRAI::tbox::AsyncCommGroup::checkReduce ( )

private

This method is the workhorse underneath the public reduce methods.

Returns: Whether operation is completed.

◆ reduceData()

void SAMRAI::tbox::AsyncCommGroup::reduceData	(	int *	output,
		int *	data
	)		const

private

The exact reduce operation depends on the base operation.

◆ computeDependentData()

void SAMRAI::tbox::AsyncCommGroup::computeDependentData ( const Array< int > & group )

private

Extract and compute parts and characteristics of the tree relevant to the local process.

◆ resetStatus()

void SAMRAI::tbox::AsyncCommGroup::resetStatus ( )

private

◆ toPosition()

int SAMRAI::tbox::AsyncCommGroup::toPosition ( int index ) const

private

◆ toIndex()

int SAMRAI::tbox::AsyncCommGroup::toIndex ( int position ) const

private

◆ toChildPosition()

int SAMRAI::tbox::AsyncCommGroup::toChildPosition	(	int	parent_pos,
		int	ic
	)		const

private

Parameters

parent_pos	Position of the parent in the group.
ic	Index of the child. (Zero coresponds to the first child.)

◆ toOldest()

int SAMRAI::tbox::AsyncCommGroup::toOldest ( int parent_pos ) const

private

Same as toChildPosition( parent_pos, 0 );

◆ toYoungest()

int SAMRAI::tbox::AsyncCommGroup::toYoungest ( int parent_pos ) const

private

Same as toChildPosition( parent_pos, d_nchild-1 );

◆ freeTimers()

static void SAMRAI::tbox::AsyncCommGroup::freeTimers ( )

staticprivate

Free static timers.

To be called by shutdown registry to make sure memory for timers does not leak.

Member Data Documentation

◆ d_nchild

const int SAMRAI::tbox::AsyncCommGroup::d_nchild

private

◆ d_idx

int SAMRAI::tbox::AsyncCommGroup::d_idx

private

The group is defined by an array of process ranks. The index of a process refers to the index in the this array. The "position" of a process in the group represents the position in the group, where the nodes are numbered sequentially, starting at zero for the root.

We require that the root has position zero. If the index of the root is zero, then positions are identical to indices. If not, then the index of the root is swapped with zero to get positions. These two cases correspond respectively to following trivial and nontrivial maps.

*
*                 Trivial map           Nontrivial map
* Parameter     d_root_idx == 0         d_root_idx > 0
* ---------     ---------------         --------------
*
* index of root       0              d_root_idx
*
* index of          d_idx            d_idx
* local process
*
* index of            p              p == 0 ? d_root_idx :
* position p                         p == d_root_idx ? 0 :
*                                    p
*
* position of         i              i == 0 ? d_root_idx :
* index i                            i == d_root_idx ? 0 :
*                                    i
*
*

◆ d_root_idx

int SAMRAI::tbox::AsyncCommGroup::d_root_idx

private

◆ d_group_size

int SAMRAI::tbox::AsyncCommGroup::d_group_size

private

◆ d_parent_rank

int SAMRAI::tbox::AsyncCommGroup::d_parent_rank

private

If negative, there is no parent (this is the root). In send_start tasks, send only to children with valid ranks (not -1).

◆ d_child_data

ChildData* SAMRAI::tbox::AsyncCommGroup::d_child_data

private

◆ d_branch_size_totl

int SAMRAI::tbox::AsyncCommGroup::d_branch_size_totl

private

◆ d_base_op

BaseOp SAMRAI::tbox::AsyncCommGroup::d_base_op

private

◆ d_next_task_op

TaskOp SAMRAI::tbox::AsyncCommGroup::d_next_task_op

private

If d_next_task_op is none, there is no current communication operation (the last one is completed).

◆ d_external_buf

int* SAMRAI::tbox::AsyncCommGroup::d_external_buf

private

This provides the input and output for transfering data. The expected size of the buffer depends on the communication.

◆ d_external_size

int SAMRAI::tbox::AsyncCommGroup::d_external_size

private

◆ d_internal_buf

Array<int> SAMRAI::tbox::AsyncCommGroup::d_internal_buf

private

Used for gather and reduce operations but not for broadcast.

◆ d_internal_requests

SAMRAI_MPI::request* SAMRAI::tbox::AsyncCommGroup::d_internal_requests

mutableprivate

This is set to NULL if getRequestPointer() is overriden to provide the requests externally.

◆ d_mpi_tag

int SAMRAI::tbox::AsyncCommGroup::d_mpi_tag

private

◆ d_mpi_communicator

SAMRAI_MPI::comm SAMRAI::tbox::AsyncCommGroup::d_mpi_communicator

private

◆ d_use_blocking_send_to_children

bool SAMRAI::tbox::AsyncCommGroup::d_use_blocking_send_to_children

private

◆ d_use_blocking_send_to_parent

bool SAMRAI::tbox::AsyncCommGroup::d_use_blocking_send_to_parent

private

◆ d_mpi_status

SAMRAI_MPI::status SAMRAI::tbox::AsyncCommGroup::d_mpi_status

private

◆ d_mpi_err

int SAMRAI::tbox::AsyncCommGroup::d_mpi_err

private

◆ t_reduce_data

Pointer<Timer> SAMRAI::tbox::AsyncCommGroup::t_reduce_data

staticprivate

◆ t_wait_all

Pointer<Timer> SAMRAI::tbox::AsyncCommGroup::t_wait_all

staticprivate

◆ d_group_ranks

Array<int> SAMRAI::tbox::AsyncCommGroup::d_group_ranks

private

It is possible to code this class without storing all the ranks internally. However, it is easier to debug if the ranks are available.

The documentation for this class was generated from the following file:

/samrai/include/tbox/AsyncCommGroup.h

Classes

Public Member Functions

Communication methods

Mappings between array indices, group positions and process ranks

Detailed Description

Member Enumeration Documentation

◆ BaseOp

◆ TaskOp

Constructor & Destructor Documentation

◆ AsyncCommGroup() [1/2]

◆ ~AsyncCommGroup()

◆ AsyncCommGroup() [2/2]

Member Function Documentation

◆ setGroupAndRootIndex()

◆ setGroupAndRootRank()

◆ setMPITag()

◆ setMPICommunicator()

◆ setUseBlockingSendToParent()

◆ setUseBlockingSendToChildren()

◆ beginBcast()

◆ checkBcast()

◆ beginGather()

◆ checkGather()

◆ beginSumReduce()

◆ checkSumReduce()

◆ checkOperation()

◆ waitOperation()

◆ isDone()

◆ getNumberOfChildren()

◆ logCurrentState()

◆ getRequestPointer()

◆ checkMPIParams()

◆ operator=()

◆ beginReduce()

◆ checkReduce()

◆ reduceData()

◆ computeDependentData()

◆ resetStatus()

◆ toPosition()

◆ toIndex()

◆ toChildPosition()

◆ toOldest()

◆ toYoungest()

◆ freeTimers()

Member Data Documentation

◆ d_nchild

◆ d_idx

◆ d_root_idx

◆ d_group_size

◆ d_parent_rank

◆ d_child_data

◆ d_branch_size_totl

◆ d_base_op

◆ d_next_task_op

◆ d_external_buf

◆ d_external_size

◆ d_internal_buf

◆ d_internal_requests

◆ d_mpi_tag

◆ d_mpi_communicator

◆ d_use_blocking_send_to_children

◆ d_use_blocking_send_to_parent

◆ d_mpi_status

◆ d_mpi_err

◆ t_reduce_data

◆ t_wait_all

◆ d_group_ranks