Last month, figures from Chicago-based performance analysts Neal Nelson & Associates appeared to show that reduced instruction set processors, specifically the Sparc from Sun Microsystems, could sometimes run commercial applications more slowly than established complex instruction set chips like the Motorola 68020. Geoff Conrad looks behind the methodology and discovers a can of worms.
Everyone now recognises that comparing machines by MIPS ratings can be misleading, but the manner in which they are compared can be even more misleading. The comparisons by independent performance analysis Neal Nelson and Associates from Chicago, Illinois, illustrate this perfectly. Nelson compared the Motorola 68020-based Sun-3/260 workstation with the Sparc-based Sun-4/260 and, by carefully choosing his tests, made the 4 MIPS Sun-3 run 50% faster than the 10 MIPS Sun-4, and concluded that all RISC machines are unsuitable for multi-user business applications. An example of the Nelson technique is the test where all calculations operate on two main fields in the memory, and place the result back in main memory. It is significant, says Nelson, that this memory-to-memory math is very common in commercial applications, and yet Sun has been aggressively promoting the Sparc chip to manufacturers who have a predominantly commercial customer base. The result of this test was that the Sparc, Sun’s Scalable Processor Architecture Reduced Instruction Set Computer machine did 32-bit integer maths one third slower than the Motorola 68020 complex instruction set workstation. Fair comparison? This certainly appears to be an unfortunate result for the Sparc. But is it a fair comparison? The Sparc, like other RISC chips, does not do calculations this way, while the complex instruction set Motorola 68020 does. It is like racing a supersonic jet plane against a sports car on the ground, and then claiming that the car is the faster machine. When the Sparc does a calculation, the compiler has already placed the operands into its registers, and in a single cycle the chip performs the calculation and places the result in another register, and the compiler then writes it back to memory while the chip carries on calculating. And with its large complement of 120 registers, the processor has to wait for the register to be loaded from memory less than 1% of the time. Each load and store takes 10 cycles, and a typical instruction mix (allowing for branch misses as well as loads and stores, and a one-cycle execution) means the average number of cycles per instruction is around 1.4. The Nelson test forces two loads and a store for each calculation, giving 31 cycles per instruction instead of 1.4, making the machine 22 times slower than it would be operating normally. Presumably he used similar sorts of tests to produce the figures that result in several basic disk input-output functions operating more slowly on the Sun-4, despite its larger disk and faster access time. Nelson also gives some doubletalk about reduced and complex instruction set machines in general. He notes that in a complex instruction machine the handful of simple instructions that they also include execute 10 times faster than complex instructions, and from this draws the conclusion that it could be described as either all simple or all complex instructions. Only an independent performance analyst could make this statement, as everyone else uses a typical instruction mix. This still allows some massaging of the figures with the definition of typical, but nothing as blatantly unreal as Nelson’s figures. He does note that before the advent of RISC most people used an instruction mix. He then implies that RISC MIPS are false because they measure register-to-register instructions that execute much faster than complex instructions, a fact he seems to regard as cheating. But that is precisely why RISC machines were designed to operate on registers, ideally in a single cycle, although there are some people who still insist that the speed demonstrated by RISCs is almost all down to the much grea
ter number of registers they typically feature, and has nothing to do with the use of single-cycle instructions. While complex mathematical functions may take several RISC instructions to execute, and the occasional unavoidable loads and stores take a large numberof cycles (10 on the Sun-4 Sparc), he maintains, incorrectly, that this is not taken into account in their MIPS rating. The Sparc performs register-to-register operations in one cycle, but a typical instruction mix brings the average number of cycles per instructions up to 1.42. With a cycle time of 62nS, this translates into 11.36 MIPS, making the Sun-4/280’s rating of 10 MIPS on the conservative side. As a final non-sequitur, Nelson claims that the only applications that could benefit switching from a complex to a reduced instruction set machine are those that explicitly use registers extensively, while those that do not may actually slow down on a RISC machine. Nonsense This is nonsense, as the RISC’s use of registers is totally transparent to the user and program – they are allocated by the compiler and are not available explicitly to the application. Every program compiled to run on a RISC machine must use the registers because most of the RISC instructions will only act on operands placed in the registers. There are those who will argue that Sun’s implementation of RISC is hardly the best one around, and Hewlett-Packard Co is highly delighted with the benefits its own implementation of RISC has conferred on its business as well as its technical computers. The bold gamble to throw away its entire hardware past in favour of the untried Spectrum Precision Architecture was a real bet-your-company decision, but all the signs are that it has paid off handsomely for the company in reduced manufacturing costs and increased efficiency, with no sacrifice of performance, and that Hewlett is now much better placed than it would have been had it developed a traditional 32 bit processor for the HP3000 line and bought in all its Unix processors rather than just the low-end ones. So, although Nelson’s figures have been used by competitors of Sun in both the RISC and non-RISC fields, it may be that performance evaluations of RISC and non-RISC architectures can become just as confusing and as doubtful as the MIPS game.
Content from our partners
