peclet-amr 0.2.0
Block-local-Morton AMR octree + collocated cut-cell Navier–Stokes solver (header-only C++20, Kokkos + MPI)
Loading...
Searching...
No Matches
peclet::amr::DistributedOctree< Dim, Bits > Class Template Reference

#include <distributed_octree.hpp>

Classes

struct  FaceGatherPlan
 Static topology of the face-neighbour gather: which out-slots are filled from a local leaf (directSlot/directLeaf) and which must be gathered from an owner (remoteCoords at remoteSlot). More...
 
struct  FaceInfo
 Classification of leaf i's face on (axis,dir): {state, global fine probe gc, owner, in-block neighbour localNb}. More...
 
struct  GatherHaloTopology
 Flattened, value-only topology for the face-neighbour gather, established ONCE: the per-matvec exchange then moves only double values (no coords, no locateGlobal) over a fixed graph — the octree analogue of GridHaloTopology::flatten(). More...
 

Public Types

using Octree = BlockOctree< Dim, Bits >
 
using M = typename Octree::M
 
using Code = typename Octree::Code
 
using Coord = typename Octree::Coord
 

Public Member Functions

 DistributedOctree ()=default
 
void init (IVec< Dim > globalRootSize, unsigned lmax, AmrGeometry< Dim > globalGeo, std::array< bool, Dim > periodic, MPI_Comm comm=MPI_COMM_WORLD)
 Decompose a globalRootSize grid of root cells over the communicator and build this rank's local block octree (uniform, all leaves at level lmax).
 
void initDecomposed (const decomp::BlockDecomposer< Dim > &dec, unsigned lmax, AmrGeometry< Dim > globalGeo, std::array< bool, Dim > periodic, MPI_Comm comm)
 As init, but on a GIVEN decomposition of the global root grid dec.globalSize() instead of the proportional ORB init builds: rank r of comm owns block r of dec, and its local octree is that block's root cells, uniform at level lmax.
 
Octree & local ()
 
const Octree & local () const
 
int rank () const
 
int size () const
 
MPI_Comm comm () const
 
unsigned lmax () const
 
const IVec< Dim > & blockOriginRoot () const
 
const IVec< Dim > & blockBrick () const
 
const IVec< Dim > & globalRootSize () const
 
const IVec< Dim > & blockFineOrigin () const
 
const IVec< Dim > & globalFineSize () const
 
const std::array< bool, Dim > & periodic () const
 
Index rootSpan () const
 
const decomp::BlockDecomposer< Dim > & decomposition () const
 The ORB over the global ROOT grid in the CURRENT root units — rank r owns block r.
 
const Vec< Dim > & h0 () const
 Phase 3: the FINEST spacing per axis (docs/amr_anisotropic.md §2).
 
double h0Scalar () const
 The single spacing of a CUBIC global grid.
 
const AmrGeometry< Dim > & globalGeometry () const
 
IVec< Dim > globalRootOf (Index i) const
 Global root-cell coordinate of local leaf i (lmax==0: the leaf is one root cell).
 
Index findGlobalRoot (const IVec< Dim > &g) const
 Local leaf covering global root-cell g, or -1 if not owned by this rank (lmax==0).
 
bool canLiftRoot () const
 True iff THIS rank's block nests one level deeper: the global root grid, this block's origin and this block's size are all even on every axis — flow's evenBlocks test, re-evaluated in the CURRENT root units.
 
void liftRoot ()
 Lift the root one level: the root CELL doubles on every axis, so the global root grid, this block's root origin and its root size all halve, rootSpan doubles and lmax grows by one — while every FINE quantity is untouched (blockFineOrigin, blockFineSize, globalFineSize, the leaf codes and the level bytes).
 
AmrGeometry< Dim > localGeometry () const
 World geometry of this rank's block (the global geometry shifted to its origin).
 
Code globalCode (Index i) const
 Global Morton code of local leaf i (origin in global fine coordinates).
 
Index balance ()
 Bring the whole distributed octree to a 2:1-balanced state.
 
Index rebalance (std::vector< std::vector< double > > &fields)
 Re-decompose by per-root-cell octree-leaf count and migrate leaves (with their field columns) to the new owners.
 
int rebalanceAlignment () const
 The alignment exponent the last rebalance chose (align = 2^a on every axis; 0 = the plain weighted ORB, and before any rebalance) and that partition's weight imbalance (max block weight over the mean; 0 before any rebalance) — the two numbers docs/amr_mg_core_boundary.md §11.4 asks to be logged.
 
double rebalanceImbalance () const
 
std::vector< double > faceNeighborGather (const std::vector< double > &field, double sentinel=kNoNeighbor) const
 For each local leaf and each of the 2*Dim faces, the neighbouring leaf's field value.
 
FaceGatherPlan buildFaceGatherPlan () const
 
std::vector< double > faceNeighborGather (const FaceGatherPlan &plan, const std::vector< double > &field, double sentinel=kNoNeighbor) const
 Face-neighbour gather using a precomputed FaceGatherPlan: only the field values move, the classification does not re-run.
 
GatherHaloTopology buildGatherHaloTopology (const FaceGatherPlan &plan) const
 Build the value-only gather topology from a FaceGatherPlan: classify each remote coord by owner (owner==rank folds into the local fills), then ONE NBX round in which each owner learns which of its local leaves to send to each requester (locateGlobal happens here, once — never per matvec).
 
FaceInfo faceAcross (Index i, int axis, int dir) const
 
std::vector< int > coverLevels (const std::vector< std::array< Coord, Dim > > &coords) const
 For each global fine coord (already wrapped into the domain), the level of the covering leaf on its owner, or -1 if none.
 
std::vector< double > coverValues (const std::vector< std::array< Coord, Dim > > &coords, const std::vector< double > &field, double sentinel=kNoNeighbor) const
 For each global fine coord, the covering leaf's field value on its owner (sentinel if none).
 

Static Public Attributes

static constexpr double kNoNeighbor = -1e300
 gather sentinel for "no neighbour"
 

Detailed Description

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
class peclet::amr::DistributedOctree< Dim, Bits >

Definition at line 51 of file distributed_octree.hpp.

Member Typedef Documentation

◆ Octree

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
using peclet::amr::DistributedOctree< Dim, Bits >::Octree = BlockOctree<Dim, Bits>

Definition at line 53 of file distributed_octree.hpp.

◆ M

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
using peclet::amr::DistributedOctree< Dim, Bits >::M = typename Octree::M

Definition at line 54 of file distributed_octree.hpp.

◆ Code

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
using peclet::amr::DistributedOctree< Dim, Bits >::Code = typename Octree::Code

Definition at line 55 of file distributed_octree.hpp.

◆ Coord

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
using peclet::amr::DistributedOctree< Dim, Bits >::Coord = typename Octree::Coord

Definition at line 56 of file distributed_octree.hpp.

Constructor & Destructor Documentation

◆ DistributedOctree()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
peclet::amr::DistributedOctree< Dim, Bits >::DistributedOctree ( )
default

Member Function Documentation

◆ init()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
void peclet::amr::DistributedOctree< Dim, Bits >::init ( IVec< Dim >  globalRootSize,
unsigned  lmax,
AmrGeometry< Dim >  globalGeo,
std::array< bool, Dim >  periodic,
MPI_Comm  comm = MPI_COMM_WORLD 
)
inline

◆ initDecomposed()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
void peclet::amr::DistributedOctree< Dim, Bits >::initDecomposed ( const decomp::BlockDecomposer< Dim > &  dec,
unsigned  lmax,
AmrGeometry< Dim >  globalGeo,
std::array< bool, Dim >  periodic,
MPI_Comm  comm 
)
inline

As init, but on a GIVEN decomposition of the global root grid dec.globalSize() instead of the proportional ORB init builds: rank r of comm owns block r of dec, and its local octree is that block's root cells, uniform at level lmax.

This is how a multigrid stage (docs/amr_mg_depth.md §6.5) builds the moved level on its target decomposition — a sibling merge's agglomerated(d) or a repartition's fresh ORB — on the stage's sub-communicator. Collective-free; comm is only queried for rank and size.

Exceptions
std::invalid_argumentif dec does not have one block per rank of comm.

Definition at line 95 of file distributed_octree.hpp.

References peclet::amr::DistributedOctree< Dim, Bits >::comm(), peclet::amr::BlockOctree< Dim, Bits >::init(), peclet::amr::DistributedOctree< Dim, Bits >::lmax(), peclet::amr::DistributedOctree< Dim, Bits >::periodic(), and peclet::amr::transferFieldGradients().

Referenced by peclet::amr::DistributedStage< Dim, Bits >::build().

◆ local() [1/2]

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
Octree & peclet::amr::DistributedOctree< Dim, Bits >::local ( )
inline

◆ local() [2/2]

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
const Octree & peclet::amr::DistributedOctree< Dim, Bits >::local ( ) const
inline

Definition at line 122 of file distributed_octree.hpp.

◆ rank()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
int peclet::amr::DistributedOctree< Dim, Bits >::rank ( ) const
inline

◆ size()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
int peclet::amr::DistributedOctree< Dim, Bits >::size ( ) const
inline

◆ comm()

◆ lmax()

◆ blockOriginRoot()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
const IVec< Dim > & peclet::amr::DistributedOctree< Dim, Bits >::blockOriginRoot ( ) const
inline

◆ blockBrick()

◆ globalRootSize()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
const IVec< Dim > & peclet::amr::DistributedOctree< Dim, Bits >::globalRootSize ( ) const
inline

◆ blockFineOrigin()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
const IVec< Dim > & peclet::amr::DistributedOctree< Dim, Bits >::blockFineOrigin ( ) const
inline

◆ globalFineSize()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
const IVec< Dim > & peclet::amr::DistributedOctree< Dim, Bits >::globalFineSize ( ) const
inline

◆ periodic()

◆ rootSpan()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
Index peclet::amr::DistributedOctree< Dim, Bits >::rootSpan ( ) const
inline

◆ decomposition()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
const decomp::BlockDecomposer< Dim > & peclet::amr::DistributedOctree< Dim, Bits >::decomposition ( ) const
inline

The ORB over the global ROOT grid in the CURRENT root units — rank r owns block r.

It follows every liftRoot (by BlockDecomposer::coarsened) and every rebalance, so it is always the decomposition of globalRootSize(): what core's coarse-level stage machinery (peclet::core::decomp::chooseStageTarget, RedistributeTopology) takes as the level's current decomposition. Replicated on every rank.

Definition at line 139 of file distributed_octree.hpp.

Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), and peclet::amr::ReplicatedTailStage< Dim, Bits >::build().

◆ h0()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
const Vec< Dim > & peclet::amr::DistributedOctree< Dim, Bits >::h0 ( ) const
inline

◆ h0Scalar()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
double peclet::amr::DistributedOctree< Dim, Bits >::h0Scalar ( ) const
inline

The single spacing of a CUBIC global grid.

Definition at line 143 of file distributed_octree.hpp.

◆ globalGeometry()

◆ globalRootOf()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
IVec< Dim > peclet::amr::DistributedOctree< Dim, Bits >::globalRootOf ( Index  i) const
inline

Global root-cell coordinate of local leaf i (lmax==0: the leaf is one root cell).

Definition at line 147 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::code(), and peclet::amr::transferFieldGradients().

Referenced by peclet::amr::DistributedOctree< Dim, Bits >::rebalance().

◆ findGlobalRoot()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
Index peclet::amr::DistributedOctree< Dim, Bits >::findGlobalRoot ( const IVec< Dim > &  g) const
inline

Local leaf covering global root-cell g, or -1 if not owned by this rank (lmax==0).

Used to build the nested fine↔coarse multigrid transfer maps.

Definition at line 157 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::find(), and peclet::amr::transferFieldGradients().

Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), and peclet::amr::ReplicatedTailStage< Dim, Bits >::build().

◆ canLiftRoot()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
bool peclet::amr::DistributedOctree< Dim, Bits >::canLiftRoot ( ) const
inline

True iff THIS rank's block nests one level deeper: the global root grid, this block's origin and this block's size are all even on every axis — flow's evenBlocks test, re-evaluated in the CURRENT root units.

The decision is collective (one rank's odd block stops the ladder for everyone), so a caller Allreduces this before lifting; see the lockstep depth in DistributedFlowMultigrid::buildImpl. This call itself is LOCAL (no communication).

Definition at line 175 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::canLiftRoot().

◆ liftRoot()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
void peclet::amr::DistributedOctree< Dim, Bits >::liftRoot ( )
inline

Lift the root one level: the root CELL doubles on every axis, so the global root grid, this block's root origin and its root size all halve, rootSpan doubles and lmax grows by one — while every FINE quantity is untouched (blockFineOrigin, blockFineSize, globalFineSize, the leaf codes and the level bytes).

The ORB follows by BlockDecomposer::coarsened, which halves every split and every block IN PLACE, so rank r's lifted block is exactly rank r's block halved: owner lookups, the leaf halo and the multigrid transfers stay where they were and a coarse cell's 2^Dim children are all local (§6.4). What changes is that every former root cell now has siblings, so local().coarsenIf merges one octet further — which is what a multigrid level below the root brick IS (§6.1).

Precondition: canLiftRoot() on EVERY rank. Lifting one rank alone would desynchronise the decomposition; coarsened() asserts the divisibility of every split and block in debug builds.

No communication of its own — but it must be done in LOCKSTEP: every rank lifts the same number of times, because every later collective on the lifted level (halo build, owner lookup, Allreduce) assumes the same decomposition everywhere. Does not throw; the preconditions are debug asserts. Like BlockOctree::liftRoot, it is applied to copies that become multigrid levels, never to the solver's own mesh.

Definition at line 202 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::liftRoot(), and peclet::amr::transferFieldGradients().

◆ localGeometry()

◆ globalCode()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
Code peclet::amr::DistributedOctree< Dim, Bits >::globalCode ( Index  i) const
inline

Global Morton code of local leaf i (origin in global fine coordinates).

Definition at line 227 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::bounds(), and peclet::amr::transferFieldGradients().

Referenced by peclet::amr::DistributedOctree< Dim, Bits >::rebalance().

◆ balance()

◆ rebalance()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
Index peclet::amr::DistributedOctree< Dim, Bits >::rebalance ( std::vector< std::vector< double > > &  fields)
inline

Re-decompose by per-root-cell octree-leaf count and migrate leaves (with their field columns) to the new owners.

The global mesh is unchanged — this is a pure redistribution of the same leaves, so it is exactly conservative and every cell's field value is preserved bit-for-bit; only ownership (and hence this rank's local octree + block geometry) changes. Each fields[c] is a column indexed by leaf slot (length numLeaves()); on return each holds the migrated/reordered column matching the new local leaf order.

The caller is responsible for the mesh being 2:1-balanced beforehand (it stays balanced, since the leaf set is untouched). Returns the number of leaves this rank migrated away. MPI-optional: a single rank keeps everything.

Definition at line 312 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::assign(), peclet::amr::DistributedOctree< Dim, Bits >::globalCode(), peclet::amr::DistributedOctree< Dim, Bits >::globalRootOf(), peclet::amr::BlockOctree< Dim, Bits >::level(), peclet::amr::BlockOctree< Dim, Bits >::numLeaves(), and peclet::amr::transferFieldGradients().

Referenced by NB_MODULE(), and peclet::amr::pybind::DistributedOctree::rebalance().

◆ rebalanceAlignment()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
int peclet::amr::DistributedOctree< Dim, Bits >::rebalanceAlignment ( ) const
inline

The alignment exponent the last rebalance chose (align = 2^a on every axis; 0 = the plain weighted ORB, and before any rebalance) and that partition's weight imbalance (max block weight over the mean; 0 before any rebalance) — the two numbers docs/amr_mg_core_boundary.md §11.4 asks to be logged.

Replicated.

Definition at line 437 of file distributed_octree.hpp.

◆ rebalanceImbalance()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
double peclet::amr::DistributedOctree< Dim, Bits >::rebalanceImbalance ( ) const
inline

Definition at line 438 of file distributed_octree.hpp.

◆ faceNeighborGather() [1/2]

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
std::vector< double > peclet::amr::DistributedOctree< Dim, Bits >::faceNeighborGather ( const std::vector< double > &  field,
double  sentinel = kNoNeighbor 
) const
inline

For each local leaf and each of the 2*Dim faces, the neighbouring leaf's field value.

Layout: out[i*(2*Dim) + 2*axis + (dir>0 ? 0 : 1)]. Domain boundaries (no neighbour) get sentinel. field is indexed by leaf slot.

Definition at line 445 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::numLeaves(), and peclet::amr::transferFieldGradients().

Referenced by peclet::amr::DistributedPoisson< Dim, Bits >::apply(), peclet::amr::pybind::DistributedOctree::face_neighbor_gather(), and peclet::amr::lohnerIndicatorDistributed().

◆ buildFaceGatherPlan()

◆ faceNeighborGather() [2/2]

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
std::vector< double > peclet::amr::DistributedOctree< Dim, Bits >::faceNeighborGather ( const FaceGatherPlan &  plan,
const std::vector< double > &  field,
double  sentinel = kNoNeighbor 
) const
inline

Face-neighbour gather using a precomputed FaceGatherPlan: only the field values move, the classification does not re-run.

Bit-identical to faceNeighborGather(field, sentinel).

Definition at line 565 of file distributed_octree.hpp.

References peclet::amr::DistributedOctree< Dim, Bits >::coverValues(), and peclet::amr::transferFieldGradients().

◆ buildGatherHaloTopology()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
GatherHaloTopology peclet::amr::DistributedOctree< Dim, Bits >::buildGatherHaloTopology ( const FaceGatherPlan &  plan) const
inline

Build the value-only gather topology from a FaceGatherPlan: classify each remote coord by owner (owner==rank folds into the local fills), then ONE NBX round in which each owner learns which of its local leaves to send to each requester (locateGlobal happens here, once — never per matvec).

Definition at line 602 of file distributed_octree.hpp.

References peclet::amr::DistributedOctree< Dim, Bits >::GatherHaloTopology::nFaces, and peclet::amr::transferFieldGradients().

Referenced by peclet::amr::LeafHalo< Dim, Bits >::finalize(), and peclet::amr::DistributedPoissonView< Dim, Bits >::init().

◆ faceAcross()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
FaceInfo peclet::amr::DistributedOctree< Dim, Bits >::faceAcross ( Index  i,
int  axis,
int  dir 
) const
inline

◆ coverLevels()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
std::vector< int > peclet::amr::DistributedOctree< Dim, Bits >::coverLevels ( const std::vector< std::array< Coord, Dim > > &  coords) const
inline

For each global fine coord (already wrapped into the domain), the level of the covering leaf on its owner, or -1 if none.

Owner-based request/reply.

Definition at line 686 of file distributed_octree.hpp.

References peclet::amr::BlockOctree< Dim, Bits >::level(), and peclet::amr::transferFieldGradients().

Referenced by peclet::amr::DistributedFvOperator< Dim, Bits >::init(), peclet::amr::LeafHalo< Dim, Bits >::resolveMisses(), and peclet::amr::transferGradients().

◆ coverValues()

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
std::vector< double > peclet::amr::DistributedOctree< Dim, Bits >::coverValues ( const std::vector< std::array< Coord, Dim > > &  coords,
const std::vector< double > &  field,
double  sentinel = kNoNeighbor 
) const
inline

For each global fine coord, the covering leaf's field value on its owner (sentinel if none).

Owner-based request/reply — the per-matvec ghost gather.

Definition at line 713 of file distributed_octree.hpp.

References peclet::amr::transferFieldGradients().

Referenced by peclet::amr::DistributedFvOperator< Dim, Bits >::apply(), peclet::amr::DistributedOctree< Dim, Bits >::faceNeighborGather(), peclet::amr::DistributedFvOperator< Dim, Bits >::jacobi(), and peclet::amr::transferGradients().

Member Data Documentation

◆ kNoNeighbor

template<int Dim, unsigned Bits = (Dim == 2 ? 32u : (Dim == 3 ? 21u : 16u))>
constexpr double peclet::amr::DistributedOctree< Dim, Bits >::kNoNeighbor = -1e300
staticconstexpr

gather sentinel for "no neighbour"

Definition at line 58 of file distributed_octree.hpp.


The documentation for this class was generated from the following file: