Hortonworks has updated its Apache Spark in-memory platform in efforts to make it easier for enterprise customers to use its open source Hadoop data platform.
In a blog post, the California based company says it is integrating Spark with YARN so that Hadoop applications can run more efficiently with other engines, such as Hive, Storm and HBase.
"The current model of Spark-on-YARN leads to a less than ideal utilization of cluster resources, particularly when large datasets are involved," the blog post said.
It added: "Deeper integration of Spark with YARN will allow it to become a more efficient tenant along side other engines, such as Hive, Storm and HBase and others, simultaneously, all on a single data platform.
"This avoids the need to create and manage dedicated Spark clusters to support that subset of applications for which Spark is ideally suited and more effectively share resources within a single cluster."
The company is also developing ways to provide the engine with better governance, security and operations, while other improvements include better debugging facilities for apps that use Spark, and integration with a YARN feature called Application Timeline Server.
