When choosing a mobile operator, all people want to know is which is the best one. But is this a question that’s even possible to answer?
Complicating the issue is that there are so many different areas that mobile operators compete on.
For example, O2 markets itself as the customer-centric network, while EE tends to trumpet its more advanced 4G roll-out as its key asset.
Since 2008, RootMetrics has driven hundreds of thousands of miles to try and find objective answers to these questions.
RootMetrics sends ‘scouters’ to drive up and down roads in a given area, conducting tests on all of the mobile operators and how their networks perform on given metrics such as speed.
Of course, RootMetrics’s methodology is not accepted by all parties.
Last summer, outspoken T-Mobile US CEO John Legere expressed his ire through a series of tweets after T-Mobile placed behind Verizon and AT&T in its RootScores.
A little road trip is not an accurate network study," he continued. "We trust crowd-sourced info w/ real cust results See: @Ookla. http://bit.ly/1MKs2bH"
"Limited, carrier-funded, months-old road trip results or millions of real-time, real customer results – you choose what to believe!" he wrote in a separate tweet.
Separately, in the UK both Vodafone and O2 have questioned RootMetrics methodology.
According to Scott Stonham, General Manager of Europe for RootMetrics, the company does use crowd-sourced results but does not consider them adequate to create truly objective measures of a network’s performance.
So how do the tests work? Stonham says that the tests are "designed to be as close as possible to consumer experience, not a simulation".
The key is that the tests are taken at the "same time, same place, doing the same thing."
The first step is RootMetrics defining which device to use, which tend to be what Stonham calls "the latest and greatest devices". They purchase the devices directly from a high street store as a consumer would.
Scott Stonham, RootMetrics
Unlike a consumer, however, RootMetrics then carries out a range of tests on the device to ensure it is putting the network in the best possible light.
The Rootmetrics scouters then drive or walk around the designated test area, which could be the entire UK, a Metropolitan area or sometimes an area as small as an airport.
The test cycle takes seven and a half minutes then repeats. An app is installed on each device, which as Stonham says "tells it to do consumer-type things".
This includes making phone calls, sending or receiving emails and sending texts. The kit the scouters they use contains a battery and a master device, which synchronises the phones to make sure they all do the same thing at the time
Effectively this removes the problem with crowd-sourced testing: while there may be a wealth of data available, there is no way of ensuring consistency across all of the results.
For example, factors such as the time the test is taken can have a huge effect, since there may be congestion at peak usage times.
As Stonham says, a traditional drive test company would drive up and down around a particular road time and time again to try and understand the performance in a particular street.
The key difference then is that RootMetrics simply establishes the general performance that a user would get in that area. He says that the firm takes ten times the minimum tests it would need in any given area.
So what makes a good network in RootMetrics’s book? Stonham says that RootMetrics’ emphasis on the reliability rather than just speed in the reports ("If you had a fast car and it never started you would get very frustrated") means that networks with less complexity tend to do better.
For data, this means that operators with more advanced LTE offerings tend to do better, as the technology is a simpler way of delivering data.
However, until voice over LTE technology is rolled out, LTE actually makes voice communications less reliable due to the additional complexity in moving back to 3G during voice calls.
He also says that operators with more spectrum tend to better in terms of speed.
While RootMetrics is confident in its testing methods, the examples of T-Mobile and O2 questioning its methodology show that we are some way off from RootScores being accepted as the industry standard.
For Stonham, who has worked on writing four standards in the mobile industry, this isn’t an issue.
"There is a lot of compromise that goes into coming up with a standard. While every business needs to make compromises, we believe that remaining independent allows us to focus more on consumer interest than industry interest.
"With standards you have lots of companies all around the table with different interests and you end up having to compromise."
He says that he wouldn’t want RootMetrics to be involved in standardisation because it would end up "compromising what we believe to be true".
