Ceph on NVMe Made No Sense to Us—So We Built a 40x Better Alternative

TLDR: Simplyblock delivers 4x performance with 25% of the resources compared to Ceph.

When it comes to software-defined storage (SDS), performance is more than just a technical metric—it’s the foundation for scalability, costs, and operational efficiency. Ceph released a blog post in February 2025 showing off its NVMe/TCP performance using a cluster size that makes your head spin—but not necessarily in a good way.

We designed simplyblock from the ground up to provide resource efficiency and performance using NVMe/TCP protocol, while keeping data protected and available. Hence, I felt compelled to run a few benchmarks myself to see how we compare.

My goal isn’t to bash Ceph. It’s a powerful tool and absolutely has its use cases. Though, performance and resource efficiency ain’t it. That’s why I believe it’s always good to know about your options and what the right tool for the job is.

Real-World Benchmarks: Ceph vs Simplyblock

The main purpose of Ceph’s blog post was to present their NVMe/TCP gateway. They wanted to hop on the NVMe over TCP train and show achievable performance. However, their design has a flaw. It’s still built upon their existing RBD devices and acts as its name implies—a gateway. That means that Ceph is not able to make real use of the benefits brought by NVMe/TCP, such as low latency, high throughput, and reduced protocol overhead. It’s merely scratching the surface of what’s possible.

Anyway, their prime example for a somewhat real-world read-write profile is random 70:30 (read/write). While this doesn’t apply to write-most use cases, it is a fair assumption for many data-driven ones.

Their test setup: up to 12 nodes with 288 OSDs (all NVMe devices). To retrieve performance numbers, they used fio, a well-known disk performance test toolkit, running with 32 jobs and 16KB blocksize. Unfortunately, there wasn’t any further explanation of their underlying hardware.

Ceph proudly announced about 1 million IOPS on the full extent of the benchmark. The graph looks more like 940k – 950k, but let’s run with 1 million for simplicity.

Simplyblock IOPS Performance at 16K Blocksize

A quick explainer on the difference in terminology. Simplyblock doesn’t use the term OSD, but we call it a storage node (or node for short). A machine (physical or virtual) running one or more storage nodes is called a storage host. Hence, Ceph Node = Simplyblock Storage Host, Ceph OSD = Simplyblock Storage Node. While the latter isn’t 100% similar, it is close enough.

My simplyblock test setup: up to 12 hosts, 32 Nodes with 2 NVMe each (64 NVMe devices in total). Details on the entire test setup are available from the full Simplyblock Performance Report.

While the 16K blocksize surprised me, I ran with it. For an explanation, I usually use 4K and 128K when I run performance tests, giving numbers for the far ends of the spectrum. In this case, 4K gives a number of the maximum achievable IOPS, while 128K typically hits network limits.

That said, simplyblock easily achieved over 4 million IOPS in the full extent of the cluster setup.

Let the Numbers Speak

Unfortunately, Ceph decided not to provide actual numbers but only a graph. Hence, I took numbers from the graph and rounded them up.

In every case, simplyblock not only exceeds Ceph’s performance, it does so using significantly fewer NVMe devices. At 12 hosts, simplyblock delivers over 4x the performance of Ceph with only 64 NVMe drives, compared to Ceph’s 288.

Bigger, Longer, Uncut: Simplyblock Can Do More

Let’s stay with the random read/write at 70/30 for a second. So far, we talked about 16K block sizes, but not all use cases use that. Writing large files is better written to larger 128K block sizes (like write-ahead logs), while many small files benefit from smaller block sizes, like 4K.

A 12-hosts simplyblock cluster generates over 9.6 million IOPS at 4K. That’s not a typo. 

Simplyblock delivers an astounding number of IOPS with less than 25% of the hardware (as we’ve seen at 16K). If you’re trying to do more with less (who isn’t?), this kind of performance density isn’t just impressive—it’s transformative.

At 128K, simplyblock still exceeds the 1 million IOPS. That’s the same amount of IOPS that Ceph provides at 16K block size.

To put all of this into a better perspective, IOPS isn’t the only significant number. Higher block sizes mean lower IOPS but higher throughput. As a reference point, simplyblock provides a throughput of over 210 Gbit/s at 4K and 745 Gbit/s at 128K with an 8-host cluster (32 storage nodes). No magic, no RDMA, no Fibre Channel, just plain NVMe over TCP on AWS.

You can find the full Simplyblock Performance Report.

This higher performance isn’t just about faster software. It also reflects superior NVMe device utilization. Ceph requires 288 NVMe drives and 12 nodes to achieve 1 million IOPS, while simplyblock already surpasses 1 million IOPS using just 4 hosts and 32 devices. That means each NVMe drive in a simplyblock cluster delivers over 44x more IOPS than in a Ceph setup. That’s where being NVMe/TCP-native solution makes the most significant difference.

Simplyblock’s Raw Performance

But simplyblock can do even more. So far, we looked at the 70/30 mix. To get the full achievable performance, though, 100% read and 100% write are better metrics. Typical use cases for these types of workloads are AI/ML training and append-only data stores.

The following numbers are generated by running 4 storage nodes (2 NVMe devices each) per storage host (hence the 0.25 host, which is one-fourth of a storage host). The storage cluster was scaled up to 8 storage hosts. I chose this setup to show simplyblock’s impressive scalability per storage node and host. Each storage node adds between 90% and 95% of additional performance.

Latency Difference That Really Matters

While IOPS and throughput are important, many use cases are latency-sensitive. At simplyblock we love to think of latency as a rigid boundary. That’s why the maximum number of IOPS isn’t the primary goal, but the number of IOPS at a particular latency target.
With network-attached storage, people think of access latency in terms of milliseconds. We don’t. Therefore, we set a very low latency target when we measure the maximum achievable IOPS on such targets. 300µs for reads, 500µs for writes. All writes are fully distributed and data-protected with an erasure coding schema 1+1.

Ceph admits that its latency increases under higher I/O demand. That’s partly due to TCP overhead, but also because of how Ceph handles data with librbd and its storage daemons (OSDs). Even with tuning, latency creeps up as throughput increases.

Simplyblock, however, is designed with latency stability in mind. Whether at IO depth 4 or 128, latency remains consistent, which means we provide

• more predictable application response times,
• lower upfront and ongoing operational costs,
• better behavior in multi-tenant environments with strict quality of service (QoS) rules

Scaling: Linear vs. Lagging

Ceph is known to scale linearly, but with caveats. A low linear increase still yields low overall results. Beyond a certain point, you hit diminishing returns unless you spend significant time tuning reactor cores, adjusting bdevs_per_cluster, and scaling librbd clients appropriately. That means more CPU usage, more complex configurations, and a bigger ops burden.

Simplyblock is built to scale linearly and efficiently out of the box. Each additional storage node adds over 90% additional performance without additional complexity. No arcane tuning parameters, no special clients or libraries. Just scale up or out.

This simplicity really hits home for cloud-native teams and infrastructure engineers.

• Simplyblock uses fewer hosts.
• Simplyblock gets better performance.
• Simplyblock reduces time tuning and increases time building more stuff.

Cost Efficiency: It’s Not Just About Performance

In the end, it all comes down to efficiency. Cost-efficiency and resource-efficiency. Simplyblock simply achieves more with less. Saving on costs, required physical rack space, and carbon footprint.With only 64 NVMe drives delivering industry-leading performance, simplyblock significantly reduces the cost of flash storage. Compare that to Ceph’s 288 drives for a fraction of the performance, and it’s clear which solution is more storage-efficient.

Lower Power, Cooling, and Maintenance = Lower OpEx

Your power and cooling bills go down with fewer machines and fewer drives running. That also means fewer failure points, less e-waste, and lower maintenance overhead. Importantly, this hardware efficiency translates to a smaller data center carbon footprint.

If sustainability is part of your IT strategy, simplyblock helps you do more with less—fewer watts consumed per IOPS delivered, and fewer resources wasted. This is a win not only for your budget but also for your environmental goals.

Higher ROI

With simplyblock, each dollar spent gives you more IOPS, throughput, and reliable performance.It delivers the same or more value on a 40x smaller hardware footprint, slashing upfront investment and dramatically increasing your long-term ROI. That kind of efficiency is a game-changer for anyone managing growth in a performance-sensitive, budget-conscious environment.

Ceph: A Respectable Legacy, But Not Built for Today

Let’s be clear—Ceph is not a bad system. It has strong community support, proven stability in large-scale deployments, and flexibility across block, object, and file storage.

But Ceph wasn’t built for today’s high-performance, low-latency, NVMe-based infrastructure. It’s a generalist in a world demanding specialists.

Tuning Ceph to achieve modern performance feels like modifying a minivan to win a Formula 1 race. You might get there, but it’s not efficient, and it’s not what the platform was designed for.

Simplyblock: The Next-Gen Performance Frontier

When comparing Simplyblock vs Ceph, it’s not just about which platform can hit higher IOPS or throughput. It’s about what those numbers really mean: better user experience, lower infrastructure costs, faster time to value, lower energy usage and CO2 impact, and last but not least, a radically higher return on investment.

Ceph has earned its place in the SDS pantheon. But if you’re looking to build for the future, minimize complexity, and maximize performance per dollar, simplyblock is the clear winner.

And hey, for low-latency, high-performance block workloads, simplyblock’s got you covered.

Want to see the data for yourself? Check out the full Simplyblock Performance Benchmark here. Or, better yet, try simplyblock on your own and watch your storage stack level up.

Because in this race, performance isn’t just a stat. It’s your edge.

Topics

Share blog post