Sun is not revealing the price it will pay when it closes its deal with CFS, but for its money it will gain control of the Lustre file system that it hopes it can use to recover lost ground in the high performance computing market.

The move also helps Sun to continue to pitch itself as a supplier of open technology, as it said it will preserve Lustre’s open source status.

According to Sun, 20% of the world’s top 100 computer systems access NAS clusters powered by Lustre running on Linux and storing petabytes of data. The file system is used by the world’s most powerful computer – the IBM’s BlueGene/L installed at the US government’s Lawrence Livermore National Laboratory – and over half of the most powerful 30 machines on the planet.

In July CFS announced a plan to port Lustre to Sun’s ZFS file system, so creating a file system with greatly enhanced storage management. On the one hand Lustre maintains and recovers data integrity over the network, and on the other ZFS automatically verifies and maintains data integrity, CFS said then.

Sun would not give much information away yesterday, but did say that the integration will continue after the acquisition, and that Lustre will be ported to Sun’s Solaris OS. The technical team from CFS is tremendously excited at becoming a part of Sun’s storage business, according to the statement announcing the deal.

One of the customers for whom CFS and Sun have together built supercomputers is the Tokyo Institute of Technology, which is using a 1TB cluster of Sun’s Solaris-powered high-throughput Thumper NAS boxes linked by the Lustre cluster file system and feeding a cluster of Sun Fire X4600 servers.

Alongside IBM and its BlueGene mammoth machine, other server giants that have been using Lustre to create the storage for clustered supercomputers are Hewlett-Packard, and Sun itself.

Clustered or parallel file systems allow multiple servers or NAS filers to share access to the same data, so that multiple filers can collectively deliver the data throughput needed to feed supercomputers.

Other suppliers of clustered file systems include a number of start-ups such as Isilon Systems, Ibrix, Panasas and Exanet, and NAS giant Network Appliance.

These companies are mostly aiming at a different market, where commercial users want to use clustered NAS not only for throughput, but also to consolidate storage and reduce management overheads.

For the enterprise or commercial market, performance is not so critical, but simplicity of implementation and the availability of data management tools such as snapshots are much more important.

While Lustre delivers high performance, it does not play into this enterprise market, and is far more of a science project file-system than an enterprise tool. This is why HP bought PolyServe, which was a developer of enterprise-friendly clustered NAS, but still also builds Lustre-powered clusters for HPC customers.

Ovum analyst Carl Greiner said that Sun will want to move Lustre out of that very high-end niche and tap into the bigger market for enterprise-friendly clustered NAS. Sun will simplify Lustre, there’s no question about that, he said.

Our View

The plan to merge Lustre with ZFS is ambitious, and it is not going to be completed overnight.

The last major company to attempt to merge a cluster file system with a conventional file system was NetApp, which brought together its Spinnaker and WAFL file systems to create OnTap GX. The project took longer than most expected — three years – and by NetApp’s own admission it is not actually quite completed.

But Lustre as it stands is only a niche product, playing into a narrow market, so Sun will definitely want to convert Lustre into an enterprise-friendly product, with which it can recover some of the HPC ground that it has lost to Linux over the last few years.

With Sun owning Lustre, NetApp selling GX, and HP having acquired PolyServe, the pressure on EMC is mounting to make some sort of matching move into clustered file systems. Currently EMC resells software from Ibrix, and an extension of that arrangement into outright ownership of the start-up is a possibility.