Although the whole area of representing the performance of a machine by a benchmark that results in a single rating of millions of instructions per second has long been discredited, the practice remains in widespread use due to the lack of any suitable alternative methods. To add to the confusion, there are some well known design flaws in the commonly used benchmarks: early versions of Dhrystone had a bug which improved performance by around 15%, and the program is said to favour the register maths design of RISC chips; and Linpack can falsely double the true floating point performance when tested on unsuitable application code. A common pitfall is to measure disk input output with a benchmark that uses files that will fit completely inside the test machine’s cache memory, which results in a very fast input-output rating when none is in fact taking place. Now, work by independent performance analysts Neal Nelson & Associates, Chicago, Illinois, has thrown doubt on the comparison of RISC (reduced instruction set) and CISC (complex instruction set) computers by MIPS ratings, in a series of tests on the Motorola 68020-based Sun 3-260 workstation and the SPARC-based Sun 4-280. According to Nelson, eight out of the eighteen tests carried out, after allowing for differences caused by different numbers of disk cache buffers, showed the older 4 MIPS Sun 3 to be running 50% faster than the 10 MIPS RISC-based model 4. Areas where this occurred included integer maths, sequential reads and writes of 512 byte records, and random reads from both memory cache and disk. The SPARC’s 32-bit integer maths performance is measured as one third slower than on the Motorola 68020,something that Nelson attributes to the fact that the benchmark uses programming techniques to place the work fields in main memory. Presumably, the Sparc would perform faster if both operands were in registers and the result was left in a register. In the test, all calculations operate on two main fields in main memory, and place the result back in main memory. It is significant that this memory-to-memory maths is very common in commercial applications, and yet Sun has been agressively promoting the Sparc chip to manufacturers who have a predominantly commercial customer base. More slowly on the Sun-4 Nelson offers no explanation for the fact that several basic disk input-output functions operate more slowly on the Sun 4, despite its larger disk and faster average access time. Areas where the Sparc is substantially faster than the 68020 include floating point performance, function calls, and string copy and compare, which measures character manipulation power. Sun may believe that RISC and Sparc are superior for single-user CAD/CAM and engineering applications, said Nelson, but there is no evidence to suggest that it is superior for multi-user business applications. At the time of writing, no-one from Sun was available for comment. Because there has never been a precise definition of what an average instruction is, contends Nelson, the whole concept of MIPS ratings is untenable. For example, on a mainframe both a very simple instruction and a very complex one might be represented as a single instruction at machine language or assembler language level, but the simple instruction might execute 10 times faster. So the machine could be described as both a 10 MIPS computer (if a program used only simple instructions) or a 1 MIPS computer (if using all complex instructions). Before the advent of RISC, the presumption was that manufacturers would calculate the MIPS rating using a mix of instructions that were basically the same across different CPUs. With RISC processors, however, register maths functions will execute much faster than on a CISC machine, while other memory manipulation instructions do not exist at all, and need a sequence of RISC instructions. According to Nelson, applications that have used registers extensively could benefit from a switch to RISC, while applications that cannot use register maths may actually slow down on a RISC machine.