peclet::core::halo::detail Namespace Reference

Classes
struct	NonDeduced
	Identity alias that blocks template argument deduction through it (C++20 std::type_identity). More...

Functions
bool	mpiReportsCudaAware ()
	Does the MPI library itself claim CUDA awareness (OpenMPI's MPIX_Query_cuda_support)? Some stacks under-report (measured: user-space OpenMPI 5.0.7 + UCX 1.20.1 with working cuda_ipc returns 0 — see core/docs/cuda-aware-mpi.md), so this is only used to AUTO-enable, never to veto an explicit env request.
bool	probeGpuAwareMpi ()
	Probe whether MPI actually accepts DEVICE pointers (GPU-aware MPI): checksummed loopback Sendrecv on MPI_COMM_SELF with device buffers, errors returned (not fatal) and the payload verified on device.
bool	gpuAwareMpi ()
	Whether to hand DEVICE pointers straight to MPI (GPU-aware MPI) instead of host-staging.
double	haloTimeout ()
	PECLET_CORE_HALO_TIMEOUT=<seconds>: turn every exchangeEnd into a deadline wait that names the pending messages and aborts (see GridHalo::waitAllDiagnosed).
int	nbxRoundTag (MPI_Comm comm, int baseTag)

Variables
constexpr long	kNbxRoundTags = 64
	Per-communicator round counter, kept as an MPI attribute so it lives and dies with the communicator (a handle-keyed map would confuse a freed handle with its reuse).
constexpr int	kNbxFamilies = 128
constexpr int	kNbxTagBase = 24576

Function Documentation

mpiReportsCudaAware()

bool peclet::core::halo::detail::mpiReportsCudaAware ( )

inline

Does the MPI library itself claim CUDA awareness (OpenMPI's MPIX_Query_cuda_support)? Some stacks under-report (measured: user-space OpenMPI 5.0.7 + UCX 1.20.1 with working cuda_ipc returns 0 — see core/docs/cuda-aware-mpi.md), so this is only used to AUTO-enable, never to veto an explicit env request.

Definition at line 39 of file grid_halo.hpp.

Referenced by gpuAwareMpi().

probeGpuAwareMpi()

bool peclet::core::halo::detail::probeGpuAwareMpi ( )

inline

Probe whether MPI actually accepts DEVICE pointers (GPU-aware MPI): checksummed loopback Sendrecv on MPI_COMM_SELF with device buffers, errors returned (not fatal) and the payload verified on device.

Necessary-but-not-sufficient (same-node cuda_ipc/cuda_copy can work while inter-node GPUDirect is broken). CAUTION: a stack with NO cuda transport SEGFAULTS inside UCX rather than returning an error (measured on the system OpenMPI here), so this probe is only run once the MPI has claimed CUDA awareness — it is the validation gate, not the discovery mechanism.

Definition at line 58 of file grid_halo.hpp.

References MPI_Comm_free(), MPI_Initialized(), MPI_INT, MPI_STATUS_IGNORE, and MPI_SUCCESS.

Referenced by gpuAwareMpi().

gpuAwareMpi()

bool peclet::core::halo::detail::gpuAwareMpi ( )

inline

Whether to hand DEVICE pointers straight to MPI (GPU-aware MPI) instead of host-staging.

Resolution (decided once per process): env PECLET_CORE_GPU_AWARE_MPI (legacy PECLET_CORE_CUDA_AWARE_MPI) set to 0 forces host staging, nonzero forces the device-pointer path (trusted, no probe — MPIX under-reports on some working stacks); UNSET auto-enables the device path when the MPI REPORTS CUDA awareness AND the loopback probe validates it, so a properly built GPU-aware MPI is used by default with no configuration. Stacks that work but under-report (see core/docs/cuda-aware-mpi.md) still need the env=1 opt-in — blind probing is not survivable there because an unsupported transport segfaults instead of erroring.

Definition at line 111 of file grid_halo.hpp.

References mpiReportsCudaAware(), and probeGpuAwareMpi().

Referenced by peclet::core::halo::GridHalo< T >::exchangeBegin(), peclet::core::halo::ParticleHalo< Dim >::forward(), and peclet::core::halo::ParticleHalo< Dim >::reverse().

haloTimeout()

double peclet::core::halo::detail::haloTimeout ( )

inline

PECLET_CORE_HALO_TIMEOUT=<seconds>: turn every exchangeEnd into a deadline wait that names the pending messages and aborts (see GridHalo::waitAllDiagnosed).

0 / unset = plain MPI_Waitall.

Definition at line 127 of file grid_halo.hpp.

Referenced by peclet::core::halo::GridHalo< T >::exchangeEnd().

nbxRoundTag()

int peclet::core::halo::detail::nbxRoundTag ( MPI_Comm  comm, int  baseTag  )

inline

Definition at line 49 of file nbx.hpp.

References kNbxFamilies, kNbxRoundTags, kNbxTagBase, MPI_Comm_create_keyval(), MPI_Comm_get_attr(), MPI_COMM_NULL_COPY_FN, MPI_COMM_NULL_DELETE_FN, MPI_Comm_set_attr(), and MPI_KEYVAL_INVALID.

Referenced by peclet::core::halo::NbxEngine::exchange().

Variable Documentation

kNbxRoundTags

constexpr long peclet::core::halo::detail::kNbxRoundTags = 64

constexpr

Per-communicator round counter, kept as an MPI attribute so it lives and dies with the communicator (a handle-keyed map would confuse a freed handle with its reuse).

Every rank of a communicator runs the same sequence of NBX rounds on it, so the counter agrees globally.

The round tag lives in a RESERVED range: [kNbxTagBase, kNbxTagBase + kNbxFamilies * kNbxRoundTags) = [24576, 32768), the top of the MPI-guaranteed tag space (MPI_TAG_UB >= 32767). A caller's baseTag selects one of kNbxFamilies 64-tag blocks (by baseTag % kNbxFamilies), the round rotates inside the block. Two rules follow, and they are what keeps rounds from aliasing:

1. direct point-to-point tags (MPI_Isend/Irecv outside the engine) stay BELOW kNbxTagBase — the suite's are 0..63 (AMR gathers 11/41/45, grid-halo field tags), 4096..20479 (flow's VoF block exchange), 7502/7503/7603/7604 (particle halo forwards);
2. distinct NBX call sites on one communicator use baseTags that differ modulo kNbxFamilies (0, 11, 7301, 7401, 7402, 7411, 7501 -> blocks 0, 11, 5, 105, 106, 115, 77). Before this range existed the tag was baseTag + round: the first rotated round of the particle topology (7501 + 1) matched forwardPositions' direct tag 7502, and the AMR family-0 rounds walked over the direct gather tags 11/41/45 — a hang of particle_halo_np8 on an oversubscribed 2-core runner and wrong ghost values in amr_distributed_*_np{4,8} (2026-09-05).

Definition at line 44 of file nbx.hpp.

Referenced by nbxRoundTag().

kNbxFamilies

constexpr int peclet::core::halo::detail::kNbxFamilies = 128

constexpr

Definition at line 45 of file nbx.hpp.

Referenced by nbxRoundTag().

kNbxTagBase

constexpr int peclet::core::halo::detail::kNbxTagBase = 24576

constexpr

Definition at line 46 of file nbx.hpp.

Referenced by nbxRoundTag().