IBM researchers have demonstrated breakthrough in storage performance by scanning 10 billion files in 43 minutes, exceeding the previous record of one billion files in three hours by a factor of 37.
This advancement unifies data environments on a single platform, instead of being distributed across several systems that must be separately managed, and it also reduces and simplifies data management tasks, allowing more information to be stored in the same technology.
The researchers tuned General Parallel File System (GPFS), a highly scalable, clustered parallel file system unveiled by IBM in 1998, to make this breakthrough possible.
They achieved this by using GPFS running on a cluster of 10 eight core systems and solid state storage. GPFS exploits the solid state storage appliances with only 6.8 terabytes of capacity for excellent random performance and high data transfer rates.
IBM expects this innovation to help organisations cope with the exploding growth of data, transactions and digitally-aware sensors and other devices that comprise Smarter Planet systems.
It is ideally suited for applications requiring high-speed access to large volumes of data such as data mining to determine customer buying behaviors across massive data sets, seismic data processing, risk management and financial analysis, weather modeling and scientific research.
IBM vice president of storage platforms Doug Balog said this demonstration of GPFS scalability will pave the way for new products that address the challenges of a rapidly growing, multi-zettabyte world.
"This has the potential to enable much larger data environments to be unified on a single platform and dramatically reduce and simplify data management tasks such as data placement, aging, backup and migration of individual files," Balog said.
