Technology, by its nature, is always changing. For CIOs and other technology leaders to make long-term plans, they must anticipate the pace and direction of technological progress. It is little wonder, then, that a core function of the IT analyst industry is to predict the future. But are these predictions effective and do CIOs really use them to steer their strategies?
Predictions from the likes of Gartner, Forrester and IDC set the direction for the technology industry. The media draws on them for stories and suppliers often use them to justify their product strategies. But predicting the future of technology is notoriously difficult and it is not hard to find examples of predictions that failed to materialise.
Back in 2011, IDC predicted that Microsoft’s Windows Phone would pass Apple’s iOS in market share by 2016. However, by the fourth quarter of 2016, Windows Phone OS shared less than 1% of the smartphone market. In early 2009, Gartner projected the “sharpest [PC] unit decline in history”, only to report the “strongest growth rate in seven years” later that year.
Similarly, Forrester predicted that advertisement expenditure would go flat in 2018, causing “a painful correction in the agency and adtech markets.” In fact, UK ad spend grew 6.3% in 2018, reaching £23.6bn – the ninth consecutive year of market growth and the highest annual since the recording of these figures started. In the US, meanwhile, ad spend increased 4.1% over the previous year.
[Keep up with Tech Monitor: Subscribe to our weekly newsletter]
How do analysts work out their predictions?
Misfires such as these might prompt technology leaders to question the value of analyst predictions. But the analysts insist that their forecasts are in high demand, and subject to rigorous internal review.
Renee Murphy, security and risk management analyst at Forrester, says that the company’s security and risk management predictions sell well and are sought-after reports by clients. Every year, she adds, Forrester does a post-mortem to analyse how accurate their predictions had been the year before.
“If you are an analyst making a prediction in your market, you’re pretty much left alone to your own devices to figure out what the prediction for the next quarter, year or next three to five years is going to be because you’re worried about how the software might impact your industry,” says Murphy. “Are they going in the right direction? What are the trends telling me?”
Murphy differentiates between two types of analysts: hedgehogs and foxes – a distinction made in Philip E Tetlock and Dan Gardner’s book Superforecasting: The Art and Science of Prediction. Hedgehog analysts are good at cherry-picking evidence so it leans in their favour. They will defend their position until they have overwhelming evidence that makes them change their mind. A fox analyst, on the other hand, bases their predictions on different strands and ideas and are inclined to predict in the long term.
“Neither one is wrong,” says Murphy. “Because I’m not good at the short term (I’m actually much better at the three to five years; it’s just the way I work), I want to take in as much information as I can and incrementally I’m willing to change my mind. But it also leads me to figure out that I don’t have a lot of confidence in what I’m saying until I hit critical mass of information and then it all clicks together and then I can predict three to five years. That’s the methodology.”
Murphy adds that the more people you put together as a group, the better at forecasting you will get: “you can’t do it by yourself.” But not just any people: people who understand less about the subject in question will make a better forecast.
“If Forrester wanted to forecast AI [artificial intelligence], they would invite someone like me, someone who doesn’t talk about AI,” says Murphy. “I talk about technology risk but I might have a viewpoint on what the prediction might be in the future. The more people you put together and the less they understand about it, the better that forecasting gets. It’s weird but it’s true.”
Gartner, meanwhile, points to the “rigorous” peer-review process to which its analysts work is subjected. Its analysts follow “clear and documented evaluation criteria in product, service and vendor reviews” and are governed by the company’s ethical principles. As a result, the company says, “Gartner is a trusted provider of expertise and insights to a global client base of more than 14,000 enterprises in 100 countries worldwide”. (IDC did not respond to a request to comment for this article.)
What do CIOs think of analyst predictions?
Analyst forecast reports did not score highly among source of advice for CIOs approached by Tech Monitor, however. Despite the wide coverage that predictions enjoy in the media and vendor press releases, IT leaders view them with caution.
Greg Morley, group CIO at United Living, says that he places little importance on analyst reports when designing his IT strategy and priorities. Although occasionally a prediction paper may lend weight to a business case for a particular technology project, Morley says that he finds peer advice and trade reports to be of significantly greater value.
“Strategic direction is primarily based on our own direction of travel and our growth objectives, aligned to our client and market expectations,” adds Morley. “Analyst predictions make for an interesting read; however, they tend towards the aspirational and are naturally weighted towards the ‘big spenders’ that have the scale, budget and maturity to match.”
Although Morley has not bought any reports in the past five years, he says that he would be keen to subscribe to an independent ‘counter-prediction’ review dissecting all predictions from the big analyst houses: “It would be interesting to see where analysts got it right and wrong.”
A private-sector CIO who did not want to be named also told Tech Monitor that analyst predictions are not the first source when looking for inspiration for IT strategy. This CIO, who has been a Gartner research subscriber for several years and has also participated in IDC CIO community events, says that predictions provide good context for the technology macro trends and are useful to keep in the background. However, they are “never anywhere near specific enough to where each business is starting from and where it’s trying to get to”.
Shaun Le Geyt, CIO at Parkinson’s UK, is vocal in his mistrust of analyst predictions and reports. He is especially sceptical of Gartner “in terms of being reliable”, he says.
Le Geyt’s mistrust is based on personal experience. A few years ago, after taking over as head of IT for a former employer, he had a call from a vendor who was offering a cloud migration and a back-up solution. During a chat with Le Geyt, the vendor spoke of a submission they had with Gartner. “In effect, what I understood was that what I was going to say was going to be fed back to Gartner and then would likely have appeared in one of the analyst’s reports in terms of shaping the market,” says Le Geyt. “I was really uncomfortable with it. In the end, I said [to the vendor]: you’ve got a service that, you know, doesn’t match with where the world is going in terms of cloud, SaaS, PaaS… It just doesn’t stand up.”
After an abrupt ending to the call, Le Geyt never heard from the vendor again and terminated the service as he did not need it. Instead, he migrated to Azure and had its back-up services there.
Another reason why Le Geyt feels “suspicious” of Gartner’s predictions and reports is that he is “bombarded” by them, he says. Eventually, he got in touch with the company to discuss how they could support him and his organisation.
“Basically, what it comes down to is you pay a lot of money for these reports that you’ve got access to, and you might get access to a person for ten, twelve thousand pounds – whatever it is that you subscribe to,” he says. “In terms of access to people, I can get that through my peer network. In terms of the reports they gave me, they just didn’t fit anywhere in terms of where we were headed. If anything, and even as a not-for-profit, we are ahead of the times and ahead of our thinking. To be really honest, I feel like Gartner is this massive selling machine selling data and information that I’m not sure that it’s up to date and it’s relevant.” (Gartner did not provide a specific response to the episode Le Geyt describes).
Instead of predictions or analyst reports, Le Geyt relies on his wide network of peers who, he says, “offer so much value”. He particularly appreciates advice from colleagues outside his sector, such as finance and utilities, because they offer him a different perspective.
“As a charity, we talk to corporates,” says Le Geyt. “There are things for them to learn from us and us to learn from them because fundamentally it’s all about people and not technology. And I think that’s what is missing from the analysts’ perspective. They’ll talk about the technology or the platforms or the systems, but they’re not necessarily looking at the individual needs or really understand the communities that you’re supporting.”
Forrester’s Murphy acknowledges that no CIO should rely on analyst predictions alone to direct their strategies. “If a CIO called me and said, ‘I’m going to quit listening to my people’, I would say you’re a CIO who isn’t going to have a job for very long,” says Murphy. “And I hear how you value [peer advice]. But if your peers are no better at that than you are, you’ve really limited your ability and you put your influence in the wrong place.”
