View all newsletters
Receive our newsletter - data, insights and analysis delivered to you
  1. Technology
  2. Data
September 25, 2014

Hortonworks updates Spark to boost Hadoop

The updates include improved governance, security and operations.

By Amy-Jo Crowley

Hortonworks has updated its Apache Spark in-memory platform in efforts to make it easier for enterprise customers to use its open source Hadoop data platform.

In a blog post, the California based company says it is integrating Spark with YARN so that Hadoop applications can run more efficiently with other engines, such as Hive, Storm and HBase.

"The current model of Spark-on-YARN leads to a less than ideal utilization of cluster resources, particularly when large datasets are involved," the blog post said.

It added: "Deeper integration of Spark with YARN will allow it to become a more efficient tenant along side other engines, such as Hive, Storm and HBase and others, simultaneously, all on a single data platform.

"This avoids the need to create and manage dedicated Spark clusters to support that subset of applications for which Spark is ideally suited and more effectively share resources within a single cluster."

The company is also developing ways to provide the engine with better governance, security and operations, while other improvements include better debugging facilities for apps that use Spark, and integration with a YARN feature called Application Timeline Server.

Content from our partners
Sherif Tawfik: The Middle East and Africa are ready to lead on the climate
What to look for in a modern ERP system
How tech leaders can keep energy costs down and meet efficiency goals

Topics in this article :
Websites in our network
Select and enter your corporate email address Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
  • CIO
  • CTO
  • CISO
  • CSO
  • CFO
  • CDO
  • CEO
  • Architect Founder
  • MD
  • Director
  • Manager
  • Other
Visit our privacy policy for more information about our services, how New Statesman Media Group may use, process and share your personal data, including information on your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.
THANK YOU