Predictions are hard – it’s not always easy to say what will happen in the next week, let alone the next year or next decade, writes Ravi Mayuram, SVP of Engineering and CTO, Couchbase.
And looking ahead, it’s difficult to grasp just how few people knew what the next ten years would bring in 2010. After all, if you had bet $100 in 2010 that Donald Trump would be President of the U.S.; the U.K. would be preparing to exit the European Union; and that superheroes and Star Wars would dominate the box office, you would be looking forward to a hefty pay-out in the New Year.
In tech, the same is true – in 2009, many predicted that e-discovery would be one of the hottest applications of the decade. However, here at the beginning of a new decade, and with the benefit of hindsight, we can see that the 2010s was the decade when Big Data really took off. From data warehousing, in-memory processing and massively parallel processing in 2009, we have seen the use of Big Data explode in fascinating, and sometimes terrifying, ways. But what were the biggest Big Data innovations of the last ten years? We asked our brain trust of top Couchbase experts and came up with the following two innovations, and one painful lesson:
Pay-as-you-use computing has given Big Data a home in the cloud
In 2010, in-memory processing was the secret to crunching big data quickly and intelligently; using the extra speed granted by eliminating the storage pipeline together with massively parallel processing to ensure there was enough memory available to perform any calculation the business needed. Simply running enough computing power at once could be an expensive task for any business – especially since the end of Moore’s Law means that processing power will no longer become more affordable year-on-year. However, the cloud has already provided the answer. The elastic nature of the cloud means that organisations need to only pay for the processing power they actually use, instead of having to first make a significant capex investment. This means that true Big Data processing is now within reach of most enterprises. As a result, we are seeing an explosion of Big Data applications that wouldn’t have been imaginable for most businesses at the beginning of the century.