In response to challenges about the Transaction Processing Council’s credibility, brought about The Standish Group’s claims earlier this year that Oracle Corp had hyped up TPC-A results, the Council has now published a new set of guidelines for benchmark specifications. Called Clause 0, the guidelines have been designed to attempt to prohibit over-aggressive implementations of any part of the benchmark specification by test sponsors. This means that the Council now requires all companies to comply with certain restrictions to stamp out special benchmarks. In the past, some manufacturers have designed benchmarks to pretty up performance ratings by using special switches during testing, such as the discrete transction option. The new Clause 0 guidelines demand that all benchmarks include documented and publicly published results; and that the product being tested is used or can be bought by end-users in the market area that the benchmark represents. In another announcement the Council has also made it mandatory for suppliers to have independent audits done of their results. The Council will license its own set of auditors who will provide independent verification of results before they are published. Previously, the Council only recommended the use of auditors. Manufacturers will also have to bear the extra cost of using an auditor. In a letter to Council members, Kim Shanley, the Council’s chief executive, says that the organisation’s credibility had been undermined by being too lax in the enforcement of the spirit of benchmarking. Recognising this, the Council has now banned all benchmark specials and benchmark validation now has to go through a three tier submission process. First the benchmark has to comply with the new Clause 0 restrictions and then the results are validated by an external auditor. The results must then be registered with the Council, where they will be reviewed over a 60-day period by the Council member review board for final ratification.
