As the SAP AG R/3 bandwaggon gathers momentum, more and more vendors are putting together specific SAP marketing groups and touting benchmark results for R/3 application modules such as Sales and Distribution, and Finance, as proof of their respective enterprise servers. With more than 2,000 SAP R/2 mainframe installations, client-server companies are fighting for the migration business, although even the largest R/3 site currently has fewer than 1,000 production users. Now Nashua, New Hampshire-based market researcher Illuminata Inc is throwing some long overdue cold water on the R/3 benchmarking trend in the report, SAP Benchmarking: Pay No Attention To The Man Behind The Curtain, which is destined to be the subject of more than few lines of trade copy this week. The report finds that, based on individual modules, R/3 benchmarks are not broadly representative of many real-world installations and that the high-end configuration num bers vendors crave say little about more modest configurations that suit typical customers. Illuminata believes the benchmarks have been run, in certain cases, on unsupported configurations, and estimates that they are no less expensive or time-consuming to run than TPC-style tests. Most damning is the lack of an independent auditing process, or consistent disclosure of parameters, it finds. In conclusion, it finds that the publication of R/3 benchmark results complement the sales agenda for both system providers and SAP itself. But as it now stands, the benchmark is narrow, opaque, proprietary and given to extremism. Unless and until SAP mandates consistent testing and responsible disclosure, the results do little to simplify and inform customers’ decisions.
