Stage multiple asynchronous group communications so that the collective can advance asynchronously (as individual underlying MPI requests are completed). More...

#include <tbox/AsyncCommStage.h>

Classes
struct	StagedGroup
	Augmented AsyncCommGroup, to operate on a stage. More...

Public Member Functions
	AsyncCommStage ()
	Construct a stage that may begin allocating and managing groups. More...

virtual	~AsyncCommStage (void)
	Deallocate groups remaining in the stage and all internal data used to manage groups. More...

AsyncCommGroup *	allocateCommGroup (int nchild, RelaunchableJob *handle=NULL)
	Allocate a group in this stage. More...

RelaunchableJob *	advanceAny ()
	Advance to completion one group (any group) that is currently waiting for communication to complete. More...

int	advanceSome (Array< RelaunchableJob * > &completed)
	Advance to completion one or more groups (any groups) that are currently waiting for communication to complete. More...

bool	isDone () const
	Return whether the stage is done with all requested communications. More...

bool	numberOfOutstandingGroups () const
	Return the number of allocated groups that have pending communication. More...

bool	numberOfOutstandingRequests () const
	Return the number of pending SAMRAI_MPI::request objects. More...

Private Types
typedef StagedGroup *	StagedGroupPtr
	Trivial typedef used by SAMRAI template scripts to generate code to instantiate Array<StagedGroup*> (not used anywhere else). More...

Private Member Functions
void	destageGroup (StagedGroup *group)
	Clear references to a group allocated on this stage. More...

SAMRAI_MPI::request *	lookupRequestPointer (const int igroup) const
	Lookup and return the request pointer from the stage for the given group. More...

Static Private Member Functions
static void	freeTimers ()

Private Attributes
int	d_n_group
	Number of groups (including deallocated groups) that are still occupying space in the stage. More...

int	d_n_req
	Number of request slots, including unused ones that belonged to deallocated groups. More...

Array< StagedGroup * >	d_group
	Groups managed in this stage. More...

Array< SAMRAI_MPI::request >	d_req
	SAMRAI_MPI::request objects used by the groups. More...

Array< int >	d_req_to_group
	Map from request index to group index. More...

Array< int >	d_group_to_req
	Map from group index to (the group's first) request index. More...

Static Private Attributes
static Pointer< Timer >	t_wait_any

static Pointer< Timer >	t_wait_some

Friends
struct	StagedGroup

Detailed Description

Use this class when you

Have multiple group communications, each performed by a different AsyncCommGroup object.
Want the multiple communications to proceed independently and asynchronously.

This class allocates a set of AsyncCommGroup objects and manages the space for their SAMRAI_MPI::request's. The requests are staged such that a single MPI_Waitany or MPI_Waitsome call applies to all communication groups allocated by the stage. Thus the communications performed by the AsyncCommGroup objects can complete in the order allowed by the MPI messages. The exact order is NOT deterministic!

To advance the communication operation of any of the allocated AsyncCommGroup objects, use advanceAny() or advanceSome(). In general, advanceSome() has better performance than advanceAny() because it gets around the "starvation" problem. See the MPI documentation for a discussion of starvation.

This class supports communication and uses MPI for message passing. If MPI is disabled, the job of this class disappears and the class is effectively empty, except for allocating and deallocating AsyncCommGroup objects.
The public interfaces still remain so the class can compile, but the implementations are trivial.

Member Typedef Documentation

◆ StagedGroupPtr

typedef StagedGroup* SAMRAI::tbox::AsyncCommStage::StagedGroupPtr

private

Constructor & Destructor Documentation

◆ AsyncCommStage()

SAMRAI::tbox::AsyncCommStage::AsyncCommStage ( )

◆ ~AsyncCommStage()

virtual SAMRAI::tbox::AsyncCommStage::~AsyncCommStage ( void )

virtual

Member Function Documentation

◆ allocateCommGroup()

AsyncCommGroup* SAMRAI::tbox::AsyncCommStage::allocateCommGroup	(	int	nchild,
		RelaunchableJob *	handle = `NULL`
	)

You may specify a handle associated with the allocated group and pointing back to an object. These pointers are returned when their group communications are completed by the stage methods advanceAny() or advanceSome(). They help identify the user object waiting for the communication to complete.

To use the handle, create a user class inheritting from RelaunchableJob so that any handle can be dynamically casted back to the user class.

Usage outline:

Derive class UserClass from RelaunchableJob. Implement the pure virtual interfaces therein.
Associate a UserClass object with a AsyncCommGroup object by using the address of the UserClass object to allocate the group. See AsyncCommStage::allocateCommGroup().
Use the allocated group to initiate a group communication.
Repeat the above for any number of user objects.
Determine the specific user object associated with the communication completed by AsyncCommStage::advanceAny() and AsyncCommStage::advanceSome() by dynamically casting the output RelaunchableJob pointer to pointers to UserClass objects.

Usage example:

* class UserClass : RelaunchableJob { ... };
* UserClass user_object[10];
* AsyncCommStage stage;
* for ( i=0; i<10; ++i ) {
*    AsyncCommGroup *group =
*      stage.allocateCommGroup( 2, &user_object[i] );
*    useGroupToInitiateCommunication(group);
* }
* RelaunchableJob *handle = stage.advanceAny();
* UserClass *completed_user_object = dynamic_cast<UserClass*>(handle);
*

Allocating a group with a NULL handle works, but you lose the ability to map a specific communication back to a user object. See advanceAny() and advanceSome().

An allocated group should be deallocated using the delete operator when no longer needed. All allocated groups will be deleted when the stage goes out of scope. Since it is an error to delete a group that has pending message requests, it is also an error to delete a stage that has groups with pending message requests.

Parameters

nchild	Number of child branches per tree node (see AsyncCommGroup).
handle	Optional pointer back to a user object associated with the group. See class documentation.

◆ advanceAny()

RelaunchableJob* SAMRAI::tbox::AsyncCommStage::advanceAny ( )

This method uses MPI_Waitany, which may be prone to starvation. It is better to use advanceSome(), which uses MPI_Waitsome, which avoids starvation.

Returns: the RelaunchableJob pointer used to construct the group that is completed. NULL if no group in the stage had incomplete communication.

◆ advanceSome()

int SAMRAI::tbox::AsyncCommStage::advanceSome ( Array< RelaunchableJob * > & completed )

Parameters

completed Array space to put handleegies associated with the completed communications.

Returns: number of groups completed or 0 if none have outstanding communication. The strategy pointers corresponding to the completed groups are set in the completed array.

◆ isDone()

bool SAMRAI::tbox::AsyncCommStage::isDone ( ) const

◆ numberOfOutstandingGroups()

bool SAMRAI::tbox::AsyncCommStage::numberOfOutstandingGroups ( ) const

◆ numberOfOutstandingRequests()

bool SAMRAI::tbox::AsyncCommStage::numberOfOutstandingRequests ( ) const

◆ destageGroup()

void SAMRAI::tbox::AsyncCommStage::destageGroup ( StagedGroup * group )

private

To ONLY be be used just before freeing an allocated group. This clears the references so that the group can be deallocated without leaving behind broken references.

◆ lookupRequestPointer()

SAMRAI_MPI::request* SAMRAI::tbox::AsyncCommStage::lookupRequestPointer ( const int igroup ) const

private

The given group MUST have been allocated by the stage. The number of requests that the group may use is the same as the number of children the group has on each branch.

The pointer is NOT guaranteed to be the same for the life of the group, as a stage may rearange the array of SAMRAI_MPI::request objects. However, the pointer is valid until the next call to allocateCommGroup().

This is private because only friend class StagedGroup should use it.

◆ freeTimers()

static void SAMRAI::tbox::AsyncCommStage::freeTimers ( )

staticprivate

Free static timers.

To be called by shutdown registry to make sure memory for timers does not leak.

Friends And Related Function Documentation

◆ StagedGroup

friend struct StagedGroup

friend

Member Data Documentation

◆ d_n_group

int SAMRAI::tbox::AsyncCommStage::d_n_group

private

◆ d_n_req

int SAMRAI::tbox::AsyncCommStage::d_n_req

private

◆ d_group

Array<StagedGroup*> SAMRAI::tbox::AsyncCommStage::d_group

private

◆ d_req

Array<SAMRAI_MPI::request> SAMRAI::tbox::AsyncCommStage::d_req

mutableprivate

This is mutable because the const method getRequestPointer() needs to return a non-const SAMRAI_MPI::request pointer. The pointer must be non-const for use in MPI calls. getRequestPointer() should be const because no group should require a non-const stage just to get the request allocated for it.

◆ d_req_to_group

Array<int> SAMRAI::tbox::AsyncCommStage::d_req_to_group

private

◆ d_group_to_req

Array<int> SAMRAI::tbox::AsyncCommStage::d_group_to_req

private

◆ t_wait_any

Pointer<Timer> SAMRAI::tbox::AsyncCommStage::t_wait_any

staticprivate

◆ t_wait_some

Pointer<Timer> SAMRAI::tbox::AsyncCommStage::t_wait_some

staticprivate

The documentation for this class was generated from the following file:

/samrai/include/tbox/AsyncCommStage.h

Classes

Public Member Functions

Private Types

Private Member Functions

Static Private Member Functions

Private Attributes

Static Private Attributes

Friends

Detailed Description

Member Typedef Documentation

◆ StagedGroupPtr

Constructor & Destructor Documentation

◆ AsyncCommStage()

◆ ~AsyncCommStage()

Member Function Documentation

◆ allocateCommGroup()

◆ advanceAny()

◆ advanceSome()

◆ isDone()

◆ numberOfOutstandingGroups()

◆ numberOfOutstandingRequests()

◆ destageGroup()

◆ lookupRequestPointer()

◆ freeTimers()

Friends And Related Function Documentation

◆ StagedGroup

Member Data Documentation

◆ d_n_group

◆ d_n_req

◆ d_group

◆ d_req

◆ d_req_to_group

◆ d_group_to_req

◆ t_wait_any

◆ t_wait_some