View all newsletters
Receive our newsletter - data, insights and analysis delivered to you
  1. Technology
  2. Data
September 25, 2014

Hortonworks updates Spark to boost Hadoop

The updates include improved governance, security and operations.


Hortonworks has updated its Apache Spark in-memory platform in efforts to make it easier for enterprise customers to use its open source Hadoop data platform.

In a blog post, the California based company says it is integrating Spark with YARN so that Hadoop applications can run more efficiently with other engines, such as Hive, Storm and HBase.

"The current model of Spark-on-YARN leads to a less than ideal utilization of cluster resources, particularly when large datasets are involved," the blog post said.

It added: "Deeper integration of Spark with YARN will allow it to become a more efficient tenant along side other engines, such as Hive, Storm and HBase and others, simultaneously, all on a single data platform.

"This avoids the need to create and manage dedicated Spark clusters to support that subset of applications for which Spark is ideally suited and more effectively share resources within a single cluster."

The company is also developing ways to provide the engine with better governance, security and operations, while other improvements include better debugging facilities for apps that use Spark, and integration with a YARN feature called Application Timeline Server.

Content from our partners
European Technology Leadership: Deutsche Bank CTO Gordon Mackechnie
Print’s role in driving the environmental agenda
What finance leaders get wrong about digital transformation

Websites in our network
NEWSLETTER Sign up Tick the boxes of the newsletters you would like to receive. Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
I consent to New Statesman Media Group collecting my details provided via this form in accordance with the Privacy Policy