Architectural Levels of Parallelism

💻 Multi-Core

4-16 cores
Shared memory
Fast communication

Local development & testing

🖥️ HPC Clusters

100s-1000s nodes
Scheduled jobs
High-speed network

Large-scale simulations

☁️ Cloud

On-demand scaling
Pay-per-use
Flexible resources

Variable workloads

Tip

Strictly separate Orchestration, Initiation, individual Jobs and Aggregation in your implementations.
Switching form intra-node to inter-node might only affect the Orchestration layer.