New top story on Hacker News: Show HN: Forkrun – NUMA-aware shell parallelizer (50×–400× faster than parallel)
Show HN: Forkrun – NUMA-aware shell parallelizer (50×–400× faster than parallel) 24 by jkool702 | 6 comments on Hacker News. forkrun is the culmination of a 10-year-long journey focused on "how to make shell parallelization fast". What started as a standard "fork jobs in a loop" has turned into a lock-free, CAS-retry-loop-free, SIMD-accelerated, self-tuning, NUMA aware shell-based stream parallelization engine that is (mostly) a drop-in replacement for xargs -P and GNU parallel. On my 14-core/28-thread i9-7940x, forkrun achieves: * 200,000+ batch dispatches/sec (vs ~500 for GNU Parallel) * ~95–99% CPU utilization across all 28 logical cores, even when the workload is non-existant (bash no-ops / `:`) (vs ~6% for GNU Parallel). These benchmarks are intentionally worst-case (near-zero work per task) because they measure the capability of the parallelization framework itself, not how much work an external tool can do. * Typically 50×–400× faster on real high-frequenc...