block_exchange.hpp File Reference

flow — VoF Part III rung W0: the block gather/scatter (VOF_PLAN.md §10 design items 2 and 4). More...

#include <algorithm>
#include <array>
#include <cassert>
#include <cstdint>
#include <Kokkos_Core.hpp>
#include <map>
#include <stdexcept>
#include <type_traits>
#include <vector>
#include "mac_cutcell.hpp"
#include "vof/block_container.hpp"
#include "policy.hpp"

Go to the source code of this file.

Classes
struct	peclet::flow::vof::VofPiece
	One contiguous transfer: the global cells g of a block that rank r owns, and where they sit in the block's EXTENDED local index space. More...
struct	peclet::flow::vof::VofMoveJob
	One (piece, component) copy of VofBlockExchange::movePiece – patch cells [s, s + n) to block cells [q, q + n) – for the host's one-launch move (H-4e). More...
struct	peclet::flow::vof::VofMoveTable
	The job table of one move launch, by value. More...
class	peclet::flow::vof::VofBlockExchange
	The gather / scatter / table replication of the block container. More...
struct	peclet::flow::vof::VofBlockExchange::Patch
	This rank's own patch: the extended block the face velocity and the union colour live on. More...
struct	peclet::flow::vof::VofBlockExchange::GatherTable

Namespaces
namespace	peclet
namespace	peclet::flow
namespace	peclet::flow::vof

Functions
bool	peclet::flow::vof::vofMoveOverlap (const VofMoveJob &a, const VofMoveJob &b)
	Debug check of H-4(e)'s premise: two jobs writing the same block array overlap.
void	peclet::flow::vof::vofBuildPieces (const VofBox &box, I3 gs, const std::array< bool, 3 > &per, const std::vector< VofBox > &rankBox, std::vector< VofPiece > &out, int outside=0)
	Build the pieces of one block box against every rank's owned box, in canonical order.

Detailed Description

flow — VoF Part III rung W0: the block gather/scatter (VOF_PLAN.md §10 design items 2 and 4).

MPI-guarded; with size == 1 it is a strided copy and no MPI symbol is referenced, so the single-rank Python module builds this header unchanged.

The exchange, and why it is NOT an NBX handshake

The block table (id, box, master) is replicated on every rank and the flow decomposition is replicated too (BlockDecomposer::origins()/sizes() are global on every rank). So every rank can compute every message size as a pure function of those two: the overlap of a block's box with a rank's owned box. There is nothing dynamic to discover, and a sparse-neighbourhood handshake (core::halo::NbxEngine) would pay a full round of unexpected-message discovery to learn what both sides already know. Plain MPI_Isend/Irecv with precomputed counts is the right primitive here; the NBX engine stays for the genuinely dynamic case (rung W1's redistribution, where the assignment changes).

The plan

A block's box lives in UNWRAPPED global indices and may hang outside [0, gs). Per axis it is cut into contiguous GLOBAL runs (vofAxisRuns): one run on a non-periodic axis (for the colour the outside part has no owner and is the block's own clamp fill; the face-velocity gather takes it from the boundary rank's patch ghosts), one or two on a periodic axis that crosses the seam. The Cartesian product of the runs gives at most 8 global sub-boxes, each with a known block-local offset; intersecting each with each rank's owned box gives the pieces. For one (block, rank) pair the pieces are concatenated in a canonical order (sub-box index, then x-fastest inside the piece) — both sides walk the same loop, so both agree on the layout without exchanging it.

Arrival order cannot matter, which is what makes gate G1/G4 a bitwise gate: the runs partition the block's box, so every block-local cell is written by exactly one piece from exactly one owner; and the scatter combines with max, which is exact and commutative.

Staging

Packing is host-side (create_mirror_view_and_copy per call). W0 is a correctness rung and the block payloads are small (a 20³ extended block is 190 KB of face velocity); the device-resident packing kernel + the CUDA-aware path are the W1/W2 optimisation, exactly as core's grid halo grew its device-resident variant after the host-staged one was proven.

Definition in file block_exchange.hpp.