SPEC, the Standard Performance Evaluation Corp, which measures computer systems performance, is now offering a benchmark suite for comparing the performance of different Java virtual machine client platforms, including network computers. SPEC president Kaivalya Dixit, who works in IBM Corp’s network computer division says the benchmark applications will run on Sun and Microsoft Corp Java implementations but aren’t really designed to evaluate embedded system performance. Sun Microsystems Inc has been pushing for SPEC to come up with a Java benchmark ever since it very embarrassingly got caught with its hand in the cookie jar last year (CI No 3,190). Until now the industry has been using Pendragon Software Inc’s CaffeineMarks to compare Java performance, however it was Java-owner Sun itself which was last year caught so blatantly tweaking its software that it was forced to retract results which purported to show its Java-on-Solaris performance some 50% ahead of Java running on Windows NT. SPEC’s SPECjvm98 suite includes eight different tests, five of them real applications, although performance benchmarks aren’t known for bearing much resemblance to real-world application use. Seven tests are used for computing performance metrics. One test validates some of the features of Java, such as testing for loop bounds. SPEC says the suite measures the time it takes to load the program, verify the class files, compile on the fly if a just-in-time (JIT) compiler is used, and execute the test. From the software perspective, these tests measure the efficiency of JVM, JIT compiler and operating system implementations on a given hardware platform. From the hardware perspective, the benchmark measures CPU (integer and floating-point), cache, memory, and other platform-specific hardware performance. The suite includes Sun’s javac JDK Java compiler and jack parser-generator. The benchmarks are run as applets from a web server, although SPECjvm98 doesn’t measure Abstract Windows Toolkit or graphics performance. SPEC says it was too difficult to figure out if all systems do the same amount of work. SPECjvm98 is available now for $100. SPEC used a 133MHz IBM PowerPC 604 client with 96Mb RAM, running AIX as the reference platform. Its performance is 1.0. Machines are evaluated according to memory size as in Java work the amount of memory available for garbage collection is crucial. SPEC has three categories of results – up to 48Mb, up 256Mb and beyond. Dixit said he’d wanted 32Mb, 64Mb, 128Mb and 256Mb categories but they were decided while he was on holiday. The benchmarks can be used to evaluate NC network computer performance and IBM plans to put its NCs through their paces soon. An IBM RS/6000 43P-140 with 64Mb RAM and JDK 1.1.6 with a JIT performs 10.1 on the benchmark. A Compaq Computer Corp Professional Workstation 6000 with a 300MHz Pentium II and 128Mb RAM running Sun’s JDK 1.1.6 and Symantec’s JIT on Windows NT 4 performed 10.7, while Sun’s 360MHz Ultra 60 Model 1360 with 256Mb RAM, Solaris 2,6 and a JDK 1.2 beta – without any HotSpot compiler technology – performed 18.0. There’s a one month moratorium on new results to allow vendors to test (and no doubt tweak) their systems. Pendragon president Ivan Philips said his company would continue to expand its CaffeineMark range of free Java benchmark services. The company said it offered solutions for embedded system evaluation and used synthetic programs rather than SPEC’s application-based code.
Content from our partners
