Original Summary

MDRAP v2.4.0 — We spent a sprint hunting down our 3 biggest performance bottlenecks Hey r/SideProjects ! 👋 Following up on my previous post about MDRAP , an open-source market data reliability and acceleration platform. After profiling the initial architecture, we found three areas that were consistently getting in the way of throughput: clock reads, the Python/native boundary, and persistence. So we spent the last sprint fixing those instead of just throwing more hardware at the problem. Here's what we learned. 1. The clock was costing more than expected The hot path was reading the system clock on every tick, which was adding roughly 15–20 ns per event . For a normal application, that's basically irrelevant. For a loop operating in the tens of millions of events per second, it starts to matter. We switched to the CPU's Invariant RDTSC counter ( CPUID.80000007H:EDX[8] ) and calibrated it against the invariant 2.61 GHz clock. That removed the OS clock-call overhead from the hot path. 2. ctypes was becoming expensive The next surprise was the Python/native boundary. Our original implementation used ctypes , and profiling showed significant overhead from marshaling Python objects across the FFI boundary — roughly 525 ns per scalar argument in our benchmark. So we replaced it with a dedicated CPython C-API extension: _fastpath_c.pyd using METH_FASTCALL . The result: 198K EPS → 423K EPS and compute latency dropped to around: 2.10 µs p50 That's roughly a 2.1x throughput improvement for the Python compute layer. 3. SQLite was the biggest architectural bottleneck This one was less surprising. Writing directly to SQLite from the tick loop meant that database locking and disk I/O could eventually become part of the critical path. Instead, we built an append-only memory-mapped Binary Journal ( .dbn / AOF). The tick loop writes into shared memory, while a background SHMDrainWorker handles the journal/persistence side asynchronously. That took our persistence benchmark from: 17.7K EPS → 349.4K EPS with p50 write latency going from: 795 µs → 2.7 µs That's about 19.7x higher throughput than the previous SQLite-based path in our benchmark. The current scorecard Native hot path: 20.54M EPS @ 48.7 ns/tick 10M-event sustained run: ~20.2M EPS / 0.56s Python compute: 423K EPS @ 2.10 µs p50 Durable persistence: 349K EPS @ 2.70 µs p50 1M-event multi-venue soak: 0 dropped events, flat memory RSS We also ran the changes through our release gate — 870+ tests passing across 14 stages . Obviously, these numbers are benchmark results from our particular hardware/configuration, not universal guarantees. We're sharing them mainly because the optimization process itself was interesting, and we'd love to see how other people would attack the same bottlenecks. MDRAP is MIT licensed and completely open source . 👉 GitHub: https://github.com/Aryan-20-04/mdrap If you've built anything involving high-throughput event processing, shared memory, Python

  • 情报分类：服务器与云资源
  • 分类依据：内容涉及服务器、云资源或网络线路
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/9/25 17:21:09