O2 executives have joined critics of RootMetrics mobile coverage reports, claiming that they do not represent customer experiences on mobile networks.
Asked about O2’s last place performance in a recent RootMetrics report, Derek McManus, O2 COO, said:
"We’re really focused on making sure we replicate customer experience. Despite asking Root on a number of occasions they won’t allow us in to see how they do the tests. The tests that they perform are not representative of what customers do, in our view.
"We’ve selected another independent provider to work with called P3. They are much more representative of what the customers actually do.
According to RootMetrics, for the report, the team collected 1.1 million samples and drove nearly 24,000 miles, conducted tests around the clock whilst driving, at stationary outdoor locations, and at nearly 1,500 indoor locations.
However, McManus said:
"Root doesn’t actually do testing that replicates what customers are experiencing. They don’t share any information with you, so they give a relative position score but not an actual quantitative score.
Content from our partners
"We have not focused on Root because we are committed to the journey we are on and we are working with P3 because we think that’s much more representative of the customer experience."
In a possible reference to EE, which generally performs best in RootMetrics tests, McManus said:
"There is only one operator that funds the Root tests. You can work from that what you want. But none of the other operators feel that it’s a representative test method."
Asked what customers do value from a network, McManus claimed that RootMetrics tests failed to examine several key parameters:
"What customers do is they tend to call mobile to mobile. Root doesn’t test that. People want to use social media and download video. [We look at] tests that are more relevant to that and more relevant to customer experience."
In addition, Paul Pattinson, Head of Service Management at O2, who highlighted that O2 has the largest base of iPhone customers, a device which RootMetrics does not use in its tests, said that customers valued reliability above all else.
"Customers want reliability and they want consistency of service fundamentally to enable them to use the applications that they live their lives or run their businesses with.
"There are very few applications that I know of that can actually use the higher speeds that are quoted; in fact, I don’t know of any, in terms of the app response times and other issues at the server end, forget what the network’s doing.
"When things go wrong, as they will with every network, fixed or mobile, they want an operator that listens to them and acts on what they are telling that operator, that doesn’t just say change a SIM card or turn your phone off and back on again."
Julie Dey, VP at RootMetrics, responded to the criticisms:
"We design and execute our own methodology independent from any network operator and are totally transparent about how we collect samples.
"Everything we do is designed to show how consumers experience mobile performance in real-world situations, which is why our testing covers everything from high and low network load situations (congestion), variations in speed from standing still to driving on the motorway, differences in coverage from poor to excellent, and indoor to outdoor signal situations.
"Device models are benchmarked to determine the best commercially available phone model from each network. No other company has the volume of data that we have to replicate mobile performance in everyday life, and for our last national report we collected 1.1 million samples. We stand behind our consumer-focused methodology and results.
"We make our high-level reports available for free to the public on our website. We also sell subscriptions to our in-depth data and analysis to whomever is interested in purchasing it, which can include carriers, tower operators, industry analysts, and others in the mobile space.
"Regardless of who buys our data, we remain completely and steadfastly independent, and our customers have zero influence over our testing, our analysis, and our results."
O2’s stance echoed comments from Three appearing to dismiss the most recent release of Rootscores. The operator commented to CBR:
"Three appreciates the role that independent tests play but we focus on direct feedback from customers who consistently tell us they are happy with the performance of the network.
"That performance will improve further with our continuing 4G rollout, the upcoming launch of VoLTE and the deployment of low frequency spectrum."
It is not just in the UK that RootMetrics tests have generated controversy. A recent report that put T-Mobile behind Verizon and AT&T in the USA attracted ire from outspoken T-Mobile CEO John Legere.
"That time of year again!" Legere wrote on Twitter. "Couple guys from @rootmetrics drive around, get paid by carriers to test networks on a single old-generation phone."
"A little road trip is not an accurate network study," he continued. "We trust crowd-sourced info w/ real cust results See: @Ookla. https://bit.ly/1MKs2bH"
"Limited, carrier-funded, months-old road trip results or millions of real-time, real customer results – you choose what to believe!" Legere added.
