swarmscale

Project Overview

Built during a ~10-day access window on ARF, TRUBA's new accelerated-computing cluster, as part of TÜBİTAK ULAKBİM's national HPC education program. swarmscale is a from-scratch CUDA C++17 + CUDA-aware-MPI mini-application that asks whether guaranteed per-agent collision avoidance (ORCA) is really unaffordable at swarm scale — and then, in a second act, whether the controller can be distilled into a neural network by imitation learning.

The full story — including the war stories and the honest negative results — is in the blog post; this page is the condensed technical reference. It grew out of my senior drone-swarm project, whose APF controller serves as swarmscale's parity oracle.

Stack

• Core: CUDA C++17, CUB radix sort, a 16-neighbor Seidel LP (ORCA) solved per agent in registers
• Multi-GPU: CUDA-aware OpenMPI over UCX — static 2-D domain decomposition, two-phase migration, one-cell ghost ring
• Cluster: SLURM batch jobs on 2 nodes × 4 NVIDIA H200 (NVLink intra-node, NDR InfiniBand inter-node), CUDA 12.4
• ML: PyTorch training stack (deterministic, byte-reproducible weight blobs), custom CUDA inference kernel, ONNX/TFLite edge export

Key Results

• Capacity: one H200 steps 2M agents in 3.0 ms and 8M in 12.5 ms; 8M agents on 8 GPUs step in 2.15 ms (p99 2.34 ms) — under a 100 Hz control deadline at the 99th percentile, ~3.7B agent-steps/s aggregate.
• Bitwise validation: the distributed simulation is bit-identical to the single-GPU run at 2, 4 and 8 ranks — made possible by a 64-bit cell/global-id sort key that fixes neighbor iteration order as a pure function of the agent set. The same determinism holds across GPU generations (H200 vs RTX 5080: zero differences in 27 field comparisons).
• The wall: multi-GPU stops paying between 50k and 250k agents per GPU — an order of magnitude stricter than literature-derived predictions.
• Systems findings: a UCX rendezvous pathology that serializes concurrent per-node GPU transfers (12.5× step-time impact, mitigated by threshold pinning); proof that wall-clock statistics are structurally blind to load imbalance (27× kernel-time spread behind a 1.0000 wall-time ratio); a measured null result for comm/compute overlap.
• Distillation: a 27,970-parameter permutation-invariant network reaches 0.0047 m/s median action error offline yet collides 94.7% of agents in closed loop (a measured 99× covariate-shift gap). Behind the teacher's own ORCA LP as a safety shield, it matches the teacher's safety to every printed digit up to 8M agents — but costs 5.6× more, and on a phone CPU the classical controller wins by 11.3×. After one DAgger round the shielded student out-throughputs the teacher at identical safety.

What I Learned

• Chaotic systems make "close enough" meaningless — the only trustworthy gates are per-step teacher-forced or bitwise, and designing for bitwise reproducibility from day one is what made every later claim checkable.
• The interconnect's protocol selection (one UCX environment variable) can dominate everything you did in CUDA.
• Imitation learning's covariate shift is not a footnote: offline metrics were near-perfect and the closed loop collapsed anyway. The architecture that survives is a learned proposer behind a classical certificate.
• Evidence discipline scales better than heroics: every published number traces to a job ID and a committed artifact, and 271 automated checks re-derive the report's figures — which is what made a post-hoc audit able to correct results instead of merely doubting them.