Berlin-headquartered data Artisans, founded by the creators of data stream processing engine Apache Flink, this morning rolled out a new patent-pending tech that makes fast, serialisable ACID transactions possible directly on streaming data; an industry first.
The “streaming ledger” (no, no blockchain is involved) can be used to power applications that require real-time decisions on streaming data, for example in financial services. It was revealed today at the Flink Forward Berlin conference.
In the context of transaction processing, the acronym ACID refers to the four key properties of a transaction: atomicity, consistency, isolation, and durability.
This means that A) all changes to data are performed as if they are a single operation; C) data is in a consistent state when a transaction starts and ends; I) the intermediate state of a transaction is invisible to other transactions, so transactions that run concurrently appear to be serialised and D) after a transaction successfully completes, changes to data persist and are not undone, even in the event of a system failure.
See also: Open Source Platform Aims to Democratise “Machine Learning Zoo”
The ledger builds on Apache Flink and provides the ability to perform serialisable transactions from multiple streams across shared tables and multiple rows of each table; “the data streaming equivalent of multi-row transactions on a key/value store or even across multiple key/value stores.”
The global streaming analytics market is projected to reach $47.75 billion by 2025, growing at 34.98 percent from 2017 to 2025, according to MarketInsightsReports, and data Artisans’ tool allows users to implement stream processing for a completely new class of applications that previously relied on relational database management systems.
“Data-intensive, real time applications such as fraud detection, machine learning, and real time trade pricing can now be upgraded to the streaming era effortlessly”, the company said in a blog.
Content from our partners
data Artisans: “The Crown Discipline”
Stephan Ewen, co-founder and CTO at data Artisans said: “Guaranteeing serializable ACID transactions is the crown discipline of data management. It is a very hard problem – something that even some large established databases fail to provide. We are very proud to have come up with a way to solve this problem for real time data streams, and make it fast and easy to use”.
The company described the streaming ledger as “transcending a long-standing limitation of existing stream processing technology”. (Previous research has shown that nearly half FS IT decision makers would sacrifice consistency for time to market.)
data Artisans, which has offices in San Francisco, said it has released the programming API, including a serial runtime for local development, as a open-source project on GitHub.
“A Testament to the Power of Apache Flink”
data Artisans co-founder and CEO Kostas Tzoumas told Computer Business Review: “Until now, users of stream processing technology could only build applications that update a single row at a time with strong consistency (aka correctness) guarantees… users of stream processing can now build generalised applications with ACID guarantees without sacrificing the scalability, availability, and performance characteristics that data stream processing with Apache Flink offers.”
“We also see this as a testament to the power of Apache Flink and its unique capabilities to offer the building blocks for such an advanced technology.”
Want to read more? data Artisans has put out a whitepaper here.
