Apache Kafka — arguably one of the open source software world’s most significant projects from an enterprise perspective — has had a sweeping overhaul, with the release of version 2.5.
Apache Kafka 2.5 comes with seven notable new features and a 65 significant improvements. It also features 92 bug fixes.
Kafka is a publish-subscribe (“pub-sub”) messaging system that that lets users build distributed applications. It is widely used, including by AirBnB, Goldman Sachs, Netflix, and Twitter, to build real-time streaming data pipelines and applications that adapt to these data streams.)
Apache Kafka 2.5: What’s New?
New features rolled out this Month — resulting from the “Kafka Improvement Proposal” (KIP) process — include the following, among other new metrics and security features…
1) KAFKA-6049] – This new feature lets users add a co-group in the DSL (Domain Specific Language). This makes it easier for users to allow multiple streams to aggregate together to form a single larger object (e.g. a shopping website may have a cart stream, a wish list stream, etc.
2) [KAFKA-6144] – This new feature aims to improve availability when clusters are scaled up, by allowing Kafka to serve interactive queries from in-sync standbys. (Previously, interactive queries against state stores would fail when there is a rebalance in progress.)
3) [KAFKA-7251] – This new feature adds support for TLS 1.3 — an encryption protocol that improves security and speed.
4) [KAFKA-8843] – This new features adds support for TLS encryption between Kafka and configuration management software Apache Zookeeper.
As contributor Confluent notes: “When deploying a secure Kafka cluster, it’s critical to use TLS to encrypt communication in transit. Apache Kafka 2.4 already ships with ZooKeeper 3.5, which adds TLS support between the broker and ZooKeeper. However, configuration information has to be passed via system properties as -D command line options on the Java invocation of the broker or CLI tool (e.g., zookeeper-security-migration), which is not secure. KIP-515 introduces the necessary changes to enable the use of secure configuration values for using TLS with ZooKeeper.
“ZooKeeper 3.5.7 supports both mutual TLS authentication via its ssl.clientAuth=required configuration value and TLS encryption without client certificate authentication via ssl.clientAuth=none.”
5) [KAFKA-9352] – This fixes unbalanced assignment of topic-partition to tasks, i.e. to balance Kafka load across all so-called mirrormaker instances (workers), this feature helps to evenly assign all topic-partition to the tasks.
6) [KAFKA-9445] – This feature allows users to fetch a key from a single partition rather than iterating over all the stores on an instance
Other welcome changes include KIP-447, which simplifies the API for applications that read from and write to Kafka transactionally.
Previously, this use case typically required separate producer instances for each input partition, but now there is no special requirement.
Confluent has provided a tidy overview of some of the other key updates in this video. Enjoy!
See also: Vulnerabilities in the Core: Key Lessons from a Major Open Source Census
