|
peclet-amr 0.2.0
Block-local-Morton AMR octree + collocated cut-cell Navier–Stokes solver (header-only C++20, Kokkos + MPI)
|
#include <distributed_octree.hpp>
Classes | |
| struct | FaceGatherPlan |
Static topology of the face-neighbour gather: which out-slots are filled from a local leaf (directSlot/directLeaf) and which must be gathered from an owner (remoteCoords at remoteSlot). More... | |
| struct | FaceInfo |
Classification of leaf i's face on (axis,dir): {state, global fine probe gc, owner, in-block neighbour localNb}. More... | |
| struct | GatherHaloTopology |
Flattened, value-only topology for the face-neighbour gather, established ONCE: the per-matvec exchange then moves only double values (no coords, no locateGlobal) over a fixed graph — the octree analogue of GridHaloTopology::flatten(). More... | |
Public Types | |
| using | Octree = BlockOctree< Dim, Bits > |
| using | M = typename Octree::M |
| using | Code = typename Octree::Code |
| using | Coord = typename Octree::Coord |
Public Member Functions | |
| DistributedOctree ()=default | |
| void | init (IVec< Dim > globalRootSize, unsigned lmax, AmrGeometry< Dim > globalGeo, std::array< bool, Dim > periodic, MPI_Comm comm=MPI_COMM_WORLD) |
Decompose a globalRootSize grid of root cells over the communicator and build this rank's local block octree (uniform, all leaves at level lmax). | |
| void | initDecomposed (const decomp::BlockDecomposer< Dim > &dec, unsigned lmax, AmrGeometry< Dim > globalGeo, std::array< bool, Dim > periodic, MPI_Comm comm) |
As init, but on a GIVEN decomposition of the global root grid dec.globalSize() instead of the proportional ORB init builds: rank r of comm owns block r of dec, and its local octree is that block's root cells, uniform at level lmax. | |
| Octree & | local () |
| const Octree & | local () const |
| int | rank () const |
| int | size () const |
| MPI_Comm | comm () const |
| unsigned | lmax () const |
| const IVec< Dim > & | blockOriginRoot () const |
| const IVec< Dim > & | blockBrick () const |
| const IVec< Dim > & | globalRootSize () const |
| const IVec< Dim > & | blockFineOrigin () const |
| const IVec< Dim > & | globalFineSize () const |
| const std::array< bool, Dim > & | periodic () const |
| Index | rootSpan () const |
| const decomp::BlockDecomposer< Dim > & | decomposition () const |
| The ORB over the global ROOT grid in the CURRENT root units — rank r owns block r. | |
| const Vec< Dim > & | h0 () const |
Phase 3: the FINEST spacing per axis (docs/amr_anisotropic.md §2). | |
| double | h0Scalar () const |
| The single spacing of a CUBIC global grid. | |
| const AmrGeometry< Dim > & | globalGeometry () const |
| IVec< Dim > | globalRootOf (Index i) const |
Global root-cell coordinate of local leaf i (lmax==0: the leaf is one root cell). | |
| Index | findGlobalRoot (const IVec< Dim > &g) const |
Local leaf covering global root-cell g, or -1 if not owned by this rank (lmax==0). | |
| bool | canLiftRoot () const |
True iff THIS rank's block nests one level deeper: the global root grid, this block's origin and this block's size are all even on every axis — flow's evenBlocks test, re-evaluated in the CURRENT root units. | |
| void | liftRoot () |
Lift the root one level: the root CELL doubles on every axis, so the global root grid, this block's root origin and its root size all halve, rootSpan doubles and lmax grows by one — while every FINE quantity is untouched (blockFineOrigin, blockFineSize, globalFineSize, the leaf codes and the level bytes). | |
| AmrGeometry< Dim > | localGeometry () const |
| World geometry of this rank's block (the global geometry shifted to its origin). | |
| Code | globalCode (Index i) const |
Global Morton code of local leaf i (origin in global fine coordinates). | |
| Index | balance () |
| Bring the whole distributed octree to a 2:1-balanced state. | |
| Index | rebalance (std::vector< std::vector< double > > &fields) |
| Re-decompose by per-root-cell octree-leaf count and migrate leaves (with their field columns) to the new owners. | |
| int | rebalanceAlignment () const |
The alignment exponent the last rebalance chose (align = 2^a on every axis; 0 = the plain weighted ORB, and before any rebalance) and that partition's weight imbalance (max block weight over the mean; 0 before any rebalance) — the two numbers docs/amr_mg_core_boundary.md §11.4 asks to be logged. | |
| double | rebalanceImbalance () const |
| std::vector< double > | faceNeighborGather (const std::vector< double > &field, double sentinel=kNoNeighbor) const |
| For each local leaf and each of the 2*Dim faces, the neighbouring leaf's field value. | |
| FaceGatherPlan | buildFaceGatherPlan () const |
| std::vector< double > | faceNeighborGather (const FaceGatherPlan &plan, const std::vector< double > &field, double sentinel=kNoNeighbor) const |
| Face-neighbour gather using a precomputed FaceGatherPlan: only the field values move, the classification does not re-run. | |
| GatherHaloTopology | buildGatherHaloTopology (const FaceGatherPlan &plan) const |
| Build the value-only gather topology from a FaceGatherPlan: classify each remote coord by owner (owner==rank folds into the local fills), then ONE NBX round in which each owner learns which of its local leaves to send to each requester (locateGlobal happens here, once — never per matvec). | |
| FaceInfo | faceAcross (Index i, int axis, int dir) const |
| std::vector< int > | coverLevels (const std::vector< std::array< Coord, Dim > > &coords) const |
| For each global fine coord (already wrapped into the domain), the level of the covering leaf on its owner, or -1 if none. | |
| std::vector< double > | coverValues (const std::vector< std::array< Coord, Dim > > &coords, const std::vector< double > &field, double sentinel=kNoNeighbor) const |
For each global fine coord, the covering leaf's field value on its owner (sentinel if none). | |
Static Public Attributes | |
| static constexpr double | kNoNeighbor = -1e300 |
| gather sentinel for "no neighbour" | |
Definition at line 51 of file distributed_octree.hpp.
| using peclet::amr::DistributedOctree< Dim, Bits >::Octree = BlockOctree<Dim, Bits> |
Definition at line 53 of file distributed_octree.hpp.
| using peclet::amr::DistributedOctree< Dim, Bits >::M = typename Octree::M |
Definition at line 54 of file distributed_octree.hpp.
| using peclet::amr::DistributedOctree< Dim, Bits >::Code = typename Octree::Code |
Definition at line 55 of file distributed_octree.hpp.
| using peclet::amr::DistributedOctree< Dim, Bits >::Coord = typename Octree::Coord |
Definition at line 56 of file distributed_octree.hpp.
|
default |
|
inline |
Decompose a globalRootSize grid of root cells over the communicator and build this rank's local block octree (uniform, all leaves at level lmax).
Definition at line 64 of file distributed_octree.hpp.
References peclet::amr::DistributedOctree< Dim, Bits >::comm(), peclet::amr::DistributedOctree< Dim, Bits >::globalRootSize(), peclet::amr::BlockOctree< Dim, Bits >::init(), peclet::amr::DistributedOctree< Dim, Bits >::lmax(), peclet::amr::DistributedOctree< Dim, Bits >::periodic(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::pybind::DistributedOctree::DistributedOctree().
|
inline |
As init, but on a GIVEN decomposition of the global root grid dec.globalSize() instead of the proportional ORB init builds: rank r of comm owns block r of dec, and its local octree is that block's root cells, uniform at level lmax.
This is how a multigrid stage (docs/amr_mg_depth.md §6.5) builds the moved level on its target decomposition — a sibling merge's agglomerated(d) or a repartition's fresh ORB — on the stage's sub-communicator. Collective-free; comm is only queried for rank and size.
| std::invalid_argument | if dec does not have one block per rank of comm. |
Definition at line 95 of file distributed_octree.hpp.
References peclet::amr::DistributedOctree< Dim, Bits >::comm(), peclet::amr::BlockOctree< Dim, Bits >::init(), peclet::amr::DistributedOctree< Dim, Bits >::lmax(), peclet::amr::DistributedOctree< Dim, Bits >::periodic(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedStage< Dim, Bits >::build().
|
inline |
Definition at line 121 of file distributed_octree.hpp.
Referenced by peclet::amr::pybind::DistributedOctree::adapt(), peclet::amr::DistributedStage< Dim, Bits >::build(), peclet::amr::ReplicatedTailStage< Dim, Bits >::build(), peclet::amr::pybind::DistributedOctree::centers(), peclet::amr::pybind::DistributedOctree::codes(), peclet::amr::distributedAdapt(), peclet::amr::pybind::DistributedOctree::face_neighbor_gather(), peclet::amr::DistributedPoissonView< Dim, Bits >::init(), peclet::amr::LeafHalo< Dim, Bits >::init(), peclet::amr::DistributedFvOperator< Dim, Bits >::init(), peclet::amr::AmrFlow< Bits >::initMpi(), peclet::amr::LeafHalo< Dim, Bits >::level(), peclet::amr::pybind::DistributedOctree::levels(), peclet::amr::pybind::DistributedOctree::lohner_indicator(), peclet::amr::lohnerIndicatorDistributed(), peclet::amr::LeafHalo< Dim, Bits >::lookup(), peclet::amr::pybind::DistributedOctree::num_leaves(), peclet::amr::DistributedPoisson< Dim, Bits >::numLeaves(), peclet::amr::pybind::DistributedOctree::rebalance(), peclet::amr::pybind::DistributedOctree::refine_to_sdf(), peclet::amr::pybind::DistributedOctree::refine_to_sphere(), peclet::amr::LeafHalo< Dim, Bits >::resolve(), peclet::amr::pybind::DistributedOctree::sizes(), peclet::amr::DistributedStage< Dim, Bits >::targetOctree(), peclet::amr::transferGradients(), and peclet::amr::pybind::DistributedOctree::write_vtu().
|
inline |
Definition at line 122 of file distributed_octree.hpp.
|
inline |
Definition at line 123 of file distributed_octree.hpp.
Referenced by NB_MODULE(), and peclet::amr::pybind::DistributedOctree::rank().
|
inline |
Definition at line 124 of file distributed_octree.hpp.
Referenced by NB_MODULE(), peclet::amr::pybind::DistributedOctree::size(), and peclet::amr::pybind::vec1().
|
inline |
Definition at line 125 of file distributed_octree.hpp.
Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), peclet::amr::DistributedPoissonView< Dim, Bits >::comm(), peclet::amr::LeafHalo< Dim, Bits >::comm(), peclet::amr::LeafHalo< Dim, Bits >::exchangeHost(), peclet::amr::DistributedGatherHalo< Dim, Bits >::init(), peclet::amr::DistributedOctree< Dim, Bits >::init(), peclet::amr::DistributedOctree< Dim, Bits >::initDecomposed(), peclet::amr::DistributedFvOperator< Dim, Bits >::residual(), peclet::amr::DistributedPoisson< Dim, Bits >::residual(), peclet::amr::DistributedPoissonView< Dim, Bits >::residual(), peclet::amr::DistributedPoisson< Dim, Bits >::residualNorm(), peclet::amr::DistributedPoissonView< Dim, Bits >::residualNorm(), and peclet::amr::LeafHalo< Dim, Bits >::resolveMisses().
|
inline |
Definition at line 126 of file distributed_octree.hpp.
Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), peclet::amr::ReplicatedTailStage< Dim, Bits >::build(), peclet::amr::DistributedOctree< Dim, Bits >::init(), peclet::amr::DistributedOctree< Dim, Bits >::initDecomposed(), and peclet::amr::pybind::DistributedOctree::lmax().
|
inline |
Definition at line 127 of file distributed_octree.hpp.
Referenced by peclet::amr::pybind::DistributedOctree::block_origin_root().
|
inline |
Definition at line 128 of file distributed_octree.hpp.
Referenced by peclet::amr::pybind::DistributedOctree::block_brick(), peclet::amr::DistributedStage< Dim, Bits >::build(), peclet::amr::ReplicatedTailStage< Dim, Bits >::build(), peclet::amr::LeafHalo< Dim, Bits >::init(), and peclet::amr::transferGradients().
|
inline |
Definition at line 129 of file distributed_octree.hpp.
Referenced by peclet::amr::ReplicatedTailStage< Dim, Bits >::build(), peclet::amr::pybind::DistributedOctree::global_root_size(), and peclet::amr::DistributedOctree< Dim, Bits >::init().
|
inline |
Definition at line 130 of file distributed_octree.hpp.
Referenced by peclet::amr::LeafHalo< Dim, Bits >::init(), peclet::amr::DistributedFvOperator< Dim, Bits >::init(), and peclet::amr::transferGradients().
|
inline |
Definition at line 131 of file distributed_octree.hpp.
Referenced by peclet::amr::pybind::DistributedOctree::cells(), peclet::amr::LeafHalo< Dim, Bits >::init(), peclet::amr::DistributedFvOperator< Dim, Bits >::init(), and peclet::amr::transferGradients().
|
inline |
Definition at line 132 of file distributed_octree.hpp.
Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), peclet::amr::ReplicatedTailStage< Dim, Bits >::build(), peclet::amr::LeafHalo< Dim, Bits >::init(), peclet::amr::DistributedOctree< Dim, Bits >::init(), and peclet::amr::DistributedOctree< Dim, Bits >::initDecomposed().
|
inline |
Definition at line 133 of file distributed_octree.hpp.
Referenced by peclet::amr::ReplicatedTailStage< Dim, Bits >::build(), peclet::amr::LeafHalo< Dim, Bits >::init(), and peclet::amr::transferGradients().
|
inline |
The ORB over the global ROOT grid in the CURRENT root units — rank r owns block r.
It follows every liftRoot (by BlockDecomposer::coarsened) and every rebalance, so it is always the decomposition of globalRootSize(): what core's coarse-level stage machinery (peclet::core::decomp::chooseStageTarget, RedistributeTopology) takes as the level's current decomposition. Replicated on every rank.
Definition at line 139 of file distributed_octree.hpp.
Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), and peclet::amr::ReplicatedTailStage< Dim, Bits >::build().
|
inline |
Phase 3: the FINEST spacing per axis (docs/amr_anisotropic.md §2).
Definition at line 141 of file distributed_octree.hpp.
Referenced by peclet::amr::pybind::DistributedOctree::extent(), peclet::amr::DistributedFvOperator< Dim, Bits >::init(), peclet::amr::AmrFlow< Bits >::initMpi(), and peclet::amr::pybind::DistributedOctree::spacing().
|
inline |
The single spacing of a CUBIC global grid.
Definition at line 143 of file distributed_octree.hpp.
|
inline |
Definition at line 144 of file distributed_octree.hpp.
Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), peclet::amr::DistributedFvOperator< Dim, Bits >::init(), peclet::amr::AmrFlow< Bits >::initMpi(), and peclet::amr::pybind::DistributedOctree::origin().
|
inline |
Global root-cell coordinate of local leaf i (lmax==0: the leaf is one root cell).
Definition at line 147 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::code(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedOctree< Dim, Bits >::rebalance().
|
inline |
Local leaf covering global root-cell g, or -1 if not owned by this rank (lmax==0).
Used to build the nested fine↔coarse multigrid transfer maps.
Definition at line 157 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::find(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedStage< Dim, Bits >::build(), and peclet::amr::ReplicatedTailStage< Dim, Bits >::build().
|
inline |
True iff THIS rank's block nests one level deeper: the global root grid, this block's origin and this block's size are all even on every axis — flow's evenBlocks test, re-evaluated in the CURRENT root units.
The decision is collective (one rank's odd block stops the ladder for everyone), so a caller Allreduces this before lifting; see the lockstep depth in DistributedFlowMultigrid::buildImpl. This call itself is LOCAL (no communication).
Definition at line 175 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::canLiftRoot().
|
inline |
Lift the root one level: the root CELL doubles on every axis, so the global root grid, this block's root origin and its root size all halve, rootSpan doubles and lmax grows by one — while every FINE quantity is untouched (blockFineOrigin, blockFineSize, globalFineSize, the leaf codes and the level bytes).
The ORB follows by BlockDecomposer::coarsened, which halves every split and every block IN PLACE, so rank r's lifted block is exactly rank r's block halved: owner lookups, the leaf halo and the multigrid transfers stay where they were and a coarse cell's 2^Dim children are all local (§6.4). What changes is that every former root cell now has siblings, so local().coarsenIf merges one octet further — which is what a multigrid level below the root brick IS (§6.1).
Precondition: canLiftRoot() on EVERY rank. Lifting one rank alone would desynchronise the decomposition; coarsened() asserts the divisibility of every split and block in debug builds.
No communication of its own — but it must be done in LOCKSTEP: every rank lifts the same number of times, because every later collective on the lifted level (halo build, owner lookup, Allreduce) assumes the same decomposition everywhere. Does not throw; the preconditions are debug asserts. Like BlockOctree::liftRoot, it is applied to copies that become multigrid levels, never to the solver's own mesh.
Definition at line 202 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::liftRoot(), and peclet::amr::transferFieldGradients().
|
inline |
World geometry of this rank's block (the global geometry shifted to its origin).
Definition at line 218 of file distributed_octree.hpp.
References peclet::amr::AmrGeometry< Dim >::origin, and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::pybind::DistributedOctree::centers(), peclet::amr::pybind::DistributedOctree::refine_to_sdf(), peclet::amr::pybind::DistributedOctree::refine_to_sphere(), peclet::amr::pybind::DistributedOctree::sizes(), and peclet::amr::pybind::DistributedOctree::write_vtu().
|
inline |
Global Morton code of local leaf i (origin in global fine coordinates).
Definition at line 227 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::bounds(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedOctree< Dim, Bits >::rebalance().
|
inline |
Bring the whole distributed octree to a 2:1-balanced state.
Returns the number of refinements this rank performed (local + cross-block).
Definition at line 239 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::balance2to1(), peclet::amr::BlockOctree< Dim, Bits >::code(), peclet::amr::BlockOctree< Dim, Bits >::level(), peclet::amr::BlockOctree< Dim, Bits >::numLeaves(), peclet::amr::BlockOctree< Dim, Bits >::refineIf(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::pybind::DistributedOctree::balance(), peclet::amr::distributedAdapt(), peclet::amr::pybind::DistributedOctree::refine_to_sdf(), and peclet::amr::pybind::DistributedOctree::refine_to_sphere().
|
inline |
Re-decompose by per-root-cell octree-leaf count and migrate leaves (with their field columns) to the new owners.
The global mesh is unchanged — this is a pure redistribution of the same leaves, so it is exactly conservative and every cell's field value is preserved bit-for-bit; only ownership (and hence this rank's local octree + block geometry) changes. Each fields[c] is a column indexed by leaf slot (length numLeaves()); on return each holds the migrated/reordered column matching the new local leaf order.
The caller is responsible for the mesh being 2:1-balanced beforehand (it stays balanced, since the leaf set is untouched). Returns the number of leaves this rank migrated away. MPI-optional: a single rank keeps everything.
Definition at line 312 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::assign(), peclet::amr::DistributedOctree< Dim, Bits >::globalCode(), peclet::amr::DistributedOctree< Dim, Bits >::globalRootOf(), peclet::amr::BlockOctree< Dim, Bits >::level(), peclet::amr::BlockOctree< Dim, Bits >::numLeaves(), and peclet::amr::transferFieldGradients().
Referenced by NB_MODULE(), and peclet::amr::pybind::DistributedOctree::rebalance().
|
inline |
The alignment exponent the last rebalance chose (align = 2^a on every axis; 0 = the plain weighted ORB, and before any rebalance) and that partition's weight imbalance (max block weight over the mean; 0 before any rebalance) — the two numbers docs/amr_mg_core_boundary.md §11.4 asks to be logged.
Replicated.
Definition at line 437 of file distributed_octree.hpp.
|
inline |
Definition at line 438 of file distributed_octree.hpp.
|
inline |
For each local leaf and each of the 2*Dim faces, the neighbouring leaf's field value.
Layout: out[i*(2*Dim) + 2*axis + (dir>0 ? 0 : 1)]. Domain boundaries (no neighbour) get sentinel. field is indexed by leaf slot.
Definition at line 445 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::numLeaves(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedPoisson< Dim, Bits >::apply(), peclet::amr::pybind::DistributedOctree::face_neighbor_gather(), and peclet::amr::lohnerIndicatorDistributed().
|
inline |
Definition at line 535 of file distributed_octree.hpp.
References peclet::amr::DistributedOctree< Dim, Bits >::FaceGatherPlan::nFaces, peclet::amr::BlockOctree< Dim, Bits >::numLeaves(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedPoisson< Dim, Bits >::init(), and peclet::amr::DistributedPoissonView< Dim, Bits >::init().
|
inline |
Face-neighbour gather using a precomputed FaceGatherPlan: only the field values move, the classification does not re-run.
Bit-identical to faceNeighborGather(field, sentinel).
Definition at line 565 of file distributed_octree.hpp.
References peclet::amr::DistributedOctree< Dim, Bits >::coverValues(), and peclet::amr::transferFieldGradients().
|
inline |
Build the value-only gather topology from a FaceGatherPlan: classify each remote coord by owner (owner==rank folds into the local fills), then ONE NBX round in which each owner learns which of its local leaves to send to each requester (locateGlobal happens here, once — never per matvec).
Definition at line 602 of file distributed_octree.hpp.
References peclet::amr::DistributedOctree< Dim, Bits >::GatherHaloTopology::nFaces, and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::LeafHalo< Dim, Bits >::finalize(), and peclet::amr::DistributedPoissonView< Dim, Bits >::init().
|
inline |
Definition at line 678 of file distributed_octree.hpp.
References peclet::amr::DistributedOctree< Dim, Bits >::FaceInfo::state, and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedFvOperator< Dim, Bits >::init().
|
inline |
For each global fine coord (already wrapped into the domain), the level of the covering leaf on its owner, or -1 if none.
Owner-based request/reply.
Definition at line 686 of file distributed_octree.hpp.
References peclet::amr::BlockOctree< Dim, Bits >::level(), and peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedFvOperator< Dim, Bits >::init(), peclet::amr::LeafHalo< Dim, Bits >::resolveMisses(), and peclet::amr::transferGradients().
|
inline |
For each global fine coord, the covering leaf's field value on its owner (sentinel if none).
Owner-based request/reply — the per-matvec ghost gather.
Definition at line 713 of file distributed_octree.hpp.
References peclet::amr::transferFieldGradients().
Referenced by peclet::amr::DistributedFvOperator< Dim, Bits >::apply(), peclet::amr::DistributedOctree< Dim, Bits >::faceNeighborGather(), peclet::amr::DistributedFvOperator< Dim, Bits >::jacobi(), and peclet::amr::transferGradients().
|
staticconstexpr |
gather sentinel for "no neighbour"
Definition at line 58 of file distributed_octree.hpp.