As anyone who has owned a home PC for the last decade will know, every two or three years the capacity of your hard drive starts to look woefully insufficient.
This is partly due to the sheer volume of files that we all accumulate over time, of course, but it also reflects the changing nature of those files. Where ten years ago we were all storing Word files and PDFs, today it is WAV or AIFF for audio and GIF, JPEG, MPG, AVI and a host of others for video, and most of them take up a lot more space than the average Word document.
Enter Web 2.0
This is result of a number of trends. Firstly, there has been a general improvement in the bandwidths available for internet access, with ever increasing broadband penetration, particularly in developed world markets. Second, a slew of new technologies such as RSS feeds, Rich Internet Applications (RIA) and mashups have emerged to make the internet a more customisable, interactive experience that can take advantage of the ubiquity of broadband access, at least in the developed world.
These new capabilities are collectively referred to as Web 2.0 technologies, a term coined in 2005 as a marketing phrase, but nonetheless pointing to a real change in the way people experience the internet. If you add into the mix the fact that digital cameras and webcams have dropped in price, while an increasing number of mobile phones now ship with a camera built in, we have all the elements needed to usher in a new phase in internet development: the age of User Generated Content (UGC).
Much has already been written about the phenomenon of social networking sites such as MySpace, Facebook and YouTube, all of which allow users to upload photos and, in some cases, videos to their page, then make them available to friends, colleagues, contacts and the general public around the world.
These sites, together with virtual worlds like Second Life and some of the graphically rich new computer games coming onto the market that let people play online, sites that offer streaming media and video on demand (VoD), represent the future of the internet. And one thing is clear: it needs a lot more bandwidth to access them satisfactorily, and a lot more storage for all those large files.
Content from our partners
Get ready for a yottabyte year
IT vendors are well aware of the market opportunity in such tendencies. For if the consumer at home is pondering what size hard drive should sit in his or her next PC, and even whether it might be time to invest in a little external storage device so as not to overburden the onboard HDD, imagine the kinds of storage requirements faced at the other end of the wire, in the websites and media shops creating and hosting all this rich content.
Industry statistics show demand for storage worldwide doubling every 18-24 months, such that one market participant (Dave Robertson, general manager of Hewlett-Packard’s StorageWorks division) predicts that, by 2013, we will see the first “yottabyte year,” in which vendors will collectively ship one yottabyte of storage hardware to their customers.
That’s a trillion terabytes, or 1,000,000,000,000,000,000,000,000 bytes. And here’s one more comparison to get an idea how much that represents: it is estimated that the entire Library of Congress would fit on 10 terabytes of storage.
It’s no coincidence that Google has been installing server and storage farms at data centres around the globe for the last few years, in anticipation of what it expects to be a veritable explosion of rich media content. Also symptomatic of the data explosion is the fact that content delivery networks (CDNs), which were very fashionable during the internet boom and then fell out of favour during the downturn, are again flavour of the month, despite even the credit crunch that’s raging in much of the developed world.
Jumping on the virtual world bandwagon
Meanwhile innumerable organisations from the real world, from media giant BBC, through car manufacturer Mazda to Dutch bank ABN Amro, have set up their own “islands” in Second Life, aware of the marketing opportunity of plugging into the rich vein of tech-savvy consumers that frequent it. Electric Artists and iVillage have held fashion shows in the virtual world. Germany’s Deutsche Bank, which has been operating a “branch of the future,” called Q110, in Berlin for the last four years, last year ran a campaign on Second Life featuring a virtual version of the branch, the idea being for visitors to collect a voucher there, print it off and take it to the Berlin branch to collect a gift and, of course, to get on its radar as a potential customer.
Aware that it too needs to be “down with the kids”, in July the British government’s Cabinet Office released a series of data sets and APIs as part of a programme called “Show Us a Better Way,” inviting the more techie elements of its citizenry to come up with mashups to publicise government services in a more entertaining and engaging manner for the younger generation. And of course, it all requires more storage…
Digital media storage goes mainstream
Until a few years ago, the storage requirements of the digital media industry were met by specialist companies, while the big names focused on the much larger, general enterprise customers. Now that broadcast television is going digital, telecoms operators want to deliver IPTV and VoD services over their networks and Web 2.0 is becoming the norm rather than the except on the internet, however, the mainstream players are moving to address what is already a burgeoning market.
HP, for instance, announced a storage system in May, that elicited epithets such as “vast”, “massive” and “immense” from journalists covering the news, and which the company itself calls Extreme.
The product in question is the HP StorageWorks 9100 Extreme Data Storage System (ExDS9100), but perhaps the best description of the box was as a “mega-NAS” device. It does indeed connect as a network-attached storage box, i.e. it attaches directly to the IP network rather than over the Fibre Channel protocol used by high-end storage area networks (SANs).
That also means that it stores data at the file rather than the block level, which is the preferred method for rich media content. Indeed, it is file-based data, most of it static (i.e. unchanging) after its initial creation, that is really ramping up in many enterprises on account of ever increasing amounts of video, audio and graphically rich content.
Analyst firm IDC reckons that just over half of the hard disks shipped in external storage systems this year are being deployed to support file- rather than block-based data applications, but by 2012, it expects that figure to increase to over 80%.
Servers and storage, cheek by jowl
What’s interesting about the ExDS9100 is that, rather than a gargantuan standalone NAS filer being accessed by applications sitting on servers elsewhere on an IP network, it is designed for them to sit inside the same chassis, thus reducing considerably the latency between storage back- and application front-end.
HP points out that it also enables companies: “To provision performance [i.e. processing muscle in the servers] and capacity [i.e. the storage] independently, which leads to greater flexibility in matching capabilities to unique workloads.”
Or as Robertson put it in launching the product: “As business requirements rapidly change and digital media files grow at exponential rates, many enterprises need to manage growth in a way that helps them profit from their storage infrastructure. That’s where the ExDS9100 really delivers.”
The product consists of a chassis from HP’s BladeSystem division, into which can fit up to 16 blade servers and up to ten high-density storage “blocks”, which are essentially compact, 1U-high drawers of hard drives running over the serial-attached SCSI (SAS) protocol, rather than the less performant serial-attached ATA (SATA) that’s much in vogue for backup appliances, where ultimate speed is less of a priority.
The entry-level version of the product ships with four servers and three storage blocks, giving it a storage capacity of 246TB, scaling up to 820TB when a full ten blocks are deployed.
The idea, therefore, is for the applications that will be requiring the files stored in the device to reside permanently on its blades, with the secret sauce that gets the right files delivered to the correct apps being a clustered file system that sits on a couple of the blades, controls where the files are located on the drives and serves them up to the applications.
Clustered file systems come of age
HP came by this technology through its acquisition, in February last year for an undisclosed sum, of specialist software developer PolyServe, a company with whom it had previously had an OEM relationship.
The Oregon-based ISV had been around since 1999, but its software had really begun to come into its own over the last few years as clustered file systems moved beyond niche scientific applications to the more mainstream commercial space as a way of overcoming the scalability constraints of conventional NAS filers.
Competitive edge
HP is by no means the only player in the market with a clustered file system for NAS scalability, but it points to its clout in both the general blade server and the disk storage markets as an important differentiating factor from the rest of the competition.
IDC, for instance, ranked HP as the number one vendor by revenue in disk storage systems worldwide in the fourth quarter of 2007. Robertson pointed out, in announcing the ExDS9100, that the leadership is in volume too, with HP being responsible for 45% of all the hard drives sold around the world, though that figure includes those shipping inside PCs, laptops and servers too, of course.
HP also intends to compete aggressively on price with this new uber-NAS box: Ian Duncan, director of marketing for the scalable NAS division at StorageWorks, has been quoted as saying that companies are typically spending between $4 and $5 per gigabyte to build out data centres with whitebox servers to support major internet operations, while HP has promised to bring the ExDS9100 to market at under $2 per gigabyte. The 246TB, entry-level configuration, for instance, is set to come in at under $500,000.
Single management interface
Another feature HP is highlighting with the ExDS9100 is ease of management, in that the entire system can be run by a single storage admin from one management interface. Thus, where today it can take several admins to manage one petabyte of data, HP wants to enable a single administrator to manage several petabytes, Roberson said.
The company cites in support of its argument another analyst, namely Mark Peters from the Enterprise Strategy Group. “Many companies are struggling with file-based growth – not only how to cope with the sheer growth, but also how to leverage their digital and static media to create additional revenue by delivering online services,” commented Peters.
“HP is aiming to address these dual market needs, which are much more than just petabyte scalability at an affordable price,” he went on. “Customers are looking for systems that combine scalability with simplified management, ease of use, and all-in-one application support. Put very simply, new business models require usability as much as storability.”
More features to come?
This is, of course, HP’s opening gambit in the high-end, multi-petabyte market of storage for Web 2.0 and digital media generally, and the $107bn-revenue company makes no bones about the fact that it intends to carve out a leading, if not dominant position there. In its first iteration, it can already boat some powerful advantages such as the single management interface and the co-location in one chassis of app servers and storage while the two can scale separately.
One suspects other features will be added into the product going forward. Some commentators have pointed out, for instance, that the first version to market does not offer embedded transcoding for video, which is already available from some of the more specialist niche vendors in digital media storage, for instance. Equally, the ExDS9100 won’t debut with quality of service for individual data streams, but one has to think that HP will add these kinds of features if enough customers start asking for them.
