Sign up for our newsletter
Technology / Data

ODPi reins in Hadoop complexity with runtime specifications and standards

The ODPi has published its first runtime specification in a move to standardise Hadoop applications.

Formerly known as the Open Data Platform initiative, the group was set up in 2015 as a way to try and make sure that applications would work across multiple Apache Hadoop distributions.

The collaborative Linux Foundation project has the backing of big name companies such as Hortonworks, IBM, Pivotal and SAS but has been snubbed by big leading Hadoop players MapR and Cloudera.

Accompanied by a test suite to simplify and speed up the development of data-drive applications, the runtime specification aims to tackle the problem of too much diversity and complexity in the Hadoop ecosystem, something which the group says is impeding adoption.

White papers from our partners

The runtime specification is built on Apache Hadoop version 2.7 and features HDFS, YARN and MapReduce components.

The ODPi test framework and self-certification uses Apache Big Top for packaging, testing, and configuration. The reason for using Big Top is so that it is closely aligned with the Apache Software Foundation.

Additional technical features include the runtime compliance tests being directly linked to the ODPi’s runtime specification; this is to assist with compliance.

Rob Thomas, VP, product development, IBM Analytics said: "Big Data is the key to enterprises welcoming the cognitive era and there’s a need across the board for advancements in the Hadoop ecosystem to ensure companies can get the most out of their deployments in the most efficient ways possible.

"With the ODPi Runtime Specification, developers can write their application once and run it across a variety of distributions – ensuring more efficient applications that can generate the insights necessary for business change."

The features play to the ODPis objectives to standardise Hadoop to make it easier to use and to increase adoption of the technology.

It is unclear whether the initiatives standards will receive high levels of adoption but its backing by some big vendors will likely help.

Despite receiving the backing of the likes of IBM, SAS and Hortonworks, one of the leading companies in Hadoop, Cloudera, has no intention of joining.

Mike Olson, Chairman and Chief Strategy Officer at Cloudera, told CBR last year: "We had some early concern that this would be a major distraction in the market, but it’s certainly not playing out that way. Our position at the time of its launch was correct. In practice we haven’t seen ODP membership or non membership make much of a difference."

The ODPi has said that its Operations Specification will be available later this year. It will cover Apache Ambari, the Apache Software Foundation project that is designed for provisioning, managing, and monitoring Hadoop clusters.

John Mertic, senior manager of ODPi said: "ODPi complements the work done in the Apache projects by filling a gap in the big data community in bringing together all members of the Hadoop ecosystem."
This article is from the CBROnline archive: some formatting and images may not be present.