From Apple to Uber; British Gas to Netflix, the distributed, horizontally-scalable commit log Apache Kafka has been adopted by swathes of blue chips since its inception in 2011.
Its fault-tolerant architecture allows Kafka to deliver massive streams of message: hello Internet of Things (IoT); hello anything else that requires you to redundantly store huge amounts of data and use real-time stream processing on the data that goes through it.
With most companies worth their salt wanting real-time data pipelines in 2018, Apache Kafka’s KSQL streaming engine is a hugely popular choice for anyone wants to read, write, and process streaming data in real time, at scale, using a SQL-like syntax.
Big Data = Big Confusion? Try Big Pictures
Pulling insights out of streaming data in real-time can still be a huge challenge though; particularly for those who aren’t developers.
San Mateo, California-based Arcadia Data thinks it has the answer. The company today released the industry’s first native Apache Kafka visualization software.
Its “Arcadia Instant” can be installed on a standard desktop and allow analysts, as well as developers, to visualise real-time data streams through a drag-and-drop interface.
Priyank Patel, Arcadia Data’s co-founder and Chief Product Officer, told Computer Business Review: “Having the ability to create visualisations easily – to get real-time insight into streaming data – means businesses can react quickly and plan around which products are selling fastest; identify potential cybersecurity threats; and pre-empt disruptions to manufacturing or fleet operations.”
Data stream floods
Others will no doubt be hot on Arcadia’s heels: the potential is huge.
As technology analyst and research firm Gartner found last year: “Event stream processing is becoming a major part of business analytics [… to get] information from sensor data, web clickstreams, geolocation data, weather reports, market data, social media and other event streams.”
Priyank Patel said in a release from Arcadia: “Businesses have been limited in their adoption of streaming technologies by the coding-heavy environments that dominate industry offerings. These environments often take time to deliver results, diminishing the data agility that businesses seek today.”
He added: “Arcadia Data collaborated closely with Confluent to deeply integrate the streaming SQL architecture with Arcadia Instant. This leverages the native architecture to support a push-based query model. Such integration is not possible with non-native BI software that is limited by JDBC- and ODBC-based connectivity.”
An upcoming product release will include support for complex data types, allowing users to read almost any type of data stored in Apache Kafka right from the stream, including MAP and ARRAY data types that are common in JSON and Avro file formats, Arcadia said. With IHS estimating that 125 billion connected devices will be streaming data by 2030, analysts and business developers will have their hands full.
See also: The Changing Ways Enterprises Are Managing Their Data
