Storage hardware heavyweight Micron has open-sourced a new database storage engine — software designed to accelerate the performance of various storage applications, like databases and object stores.
The move is an unusual one by a company normally tightly focussed on highly proprietary storage hardware – and represents a strategic move by the Idaho-based company up the stack from silicon to software.
It claims that the release massively outperforms commonly used storage engine RocksDB at six times the performance, 11 times lower latency and seven times greater write endurance running on SSDs.
Micron said it has designed the “heterogeneous storage engine” (HSE)* with massive-scale databases in mind, i.e. “billions of key counts, terabytes of data, and thousands of concurrent operations” and boasted that it “dramatically improves” throughput, latency, write-amplification, and read-amplification versus common alternatives for many workloads.
(The release has been built for Linux workloads “since that is the preferred operating system of data centers across the globe”).
With the release, the chip giant is particularly targeting developers using all-flash infrastructure who want to be able to customise or enhance code to give a performance boost for unique use cases; e.g. a trading company wanting to optimise high-speed risk analysis algorithms.
What is a Heterogeneous Storage Engine, Anyway?
The storage engine is software that has been designed to optimise the performance of memory hardware by orchestrating data placement across DRAM and multiple classes of SSDs or other solid-state storage.
Content from our partners
As Micron puts it, the HSE is designed to support SSDs and storage-class memory (SCM): “Legacy storage engines born in the era of hard disk drives (HDDs) failed to architecturally provide for the increased performance and
reduced latency of next-generation nonvolatile media.”
See also: Field Level Encryption: A Database Security Game-Changer?
The company is touting the software as ideal for powering NoSQL workloads that underpin a wide range of AI, Big Data, IoT or High Performance Computing (HPC) workloads. It has baked in integration with the increasingly popular NoSQL database MongoDB, although the HSE — available now on GitHub — can also be integrated with other NoSQL databases, and features a C API library that can be embedded in any application.
The release pits the HSE against RocksDB, a storage engine developed by Facebook — which Micron says it notably outperforms.
“Micron is uniquely positioned to build a software stack that accelerates applications running in today’s flash-based storage environments as well as storage class memory-based infrastructure of the future,” said Derek Dicker, GM of the Storage Business Unit at Micron. (The company is claiming improved throughput of particular storage applications by up to six times, an 11 fold reduction in latency and seven-fold improvement in SSD endurance.)
Like all such open source projects, visibility and adoption will be key to success. The firm already has the attention of Red Hat, whose Stefanie Chiras, GM of RHEL, described the release as having “enormous potential” for “lowering the latency between compute, memory and storage resources”.
Brad King, field CTO and co-founder of Scality, a data storage software specialist, added: “We look forward to working with Micron in the open-source community to further build out these innovations.”
He added: “As the demand for object-based storage continues to increase and it’s deployed for more and more workloads, it’s no surprise that our customers are increasingly interested in fast object storage.
“While our storage software can support ‘cheap and deep’ on the lowest-cost commodity hardware for the simplest workloads, it can also exploit the performance benefits of technologies like flash, storage class memory and SSDs for very demanding workloads.
“[HSE] enhances our ability to continue optimizing flash performance, latency and SSD endurance without trade-offs.”
*A footnote: Computer Business Review feels that nobody releasing software with aspirations of widespread adoption should put “heterogeneous” in the title; call us old-fashioned. Fresh suggestions for Micron — and comments on the open source release — welcomed at ed dot targett @ cbronline dot com
