Part two of a three-part series from Software Futures, a sister publication
In a recent survey of 50 large-enterprise IT managers, market research firm Forrester Research found that two thirds of those surveyed believed Unix systems scale well – while nearly the same proportion stated that Windows NT does not. Lab tests reinforce the Forrester report.
By Lloyd Blythen
In March the independent consultancy Bloor Research published ‘The Realities of Scalability’, detailing how it put two databases – Microsoft’s SQL Server and IBM’s DB2 – on the rack along with IBM’s AIX version of Unix and Microsoft’s Windows NT Server 4.0. Although careful to point out that the results shouldn’t be compared as if they were benchmarks, Bloor noted significant variations in scalability between the systems – and not just when each database was hosted on its native platform. The company set out to learn How well does an operating system/DBMS combination perform and scale when a realistic workload is run on a moderate-sized SMP server, with a moderate level of user/manager sophistication in tuning? (SMP – Symmetric Multi-Processing – is the processor scalability mechanism found in AIX and NT.) This, surely, is more useful than the contrived billion-transaction test; even Microsoft’s group VP (platforms and applications) Paul Maritz acknowledged that No-one needs a billion transactions a day right now. Bloor Research examined software scalability using established metrics: take one system – not a cluster of 25 – and see how well it accommodates more users or exploits the availability of more processors. The company summarized its findings by saying, DB2 running on the AIX platform is appreciably more scalable than DB2 running on Windows NT which is, in turn, more scalable than Microsoft SQL Server running on NT. When either database was subjected to read-only tests the NT systems scaled better than DB2 on AIX; they showed near perfect scaling with an almost-linear performance improvement as extra processors were added to the box. But that’s cold comfort for Microsoft because, in terms of absolute performance, both NT systems were well behind DB2 on AIX. They might, given their superior scalability, conceivably have caught up with the latter at some (high) CPU count, but this is pointless speculation given the limit on NT’s ability to utilize a large number of CPUs. The one bright note for Microsoft came in medium-contention read/write tests, in which the DB2/AIX combo showed virtually no improvement with more than four processors and did not match the transaction rate of DB2 on NT. However, the difference in absolute performance was small, and this time the trend was reversed: DB2 scaled better on AIX than on NT.
Many users
If Microsoft’s ‘billion transactions per day’ figure is to have any relevance, it must be as a reflection of how many users the company thinks its server software can cope with. Almost invariably among real-world applications the number of users changes more often than the number of CPUs that serve them. Bloor’s results for user scalability, therefore, provide a worthwhile indication of whether a Microsoft server is inherently capable of handling larger loads – alone or as one member of a cluster – than a roughly-equivalent Unix one. These results are more conclusive than Bloor’s findings on CPU scalability and suggest that Microsoft had better keep its head down. On a four- CPU box, DB2 for AIX outstripped everything else by a wide margin. But suppose we ignore absolute performance and consider scalability instead, as Microsoft would apparently have us do: in that case what counts is the change in performance as more users are added, and the trend is startling. Indeed the sheer lack of any trend in the DB2/AIX figures is most startling of all: from 100-odd users on up to nearly 400, DB2 on AIX appears to champ on its pipe-stem and work on with no drop in performance. On NT, DB2 actually managed better than 100% user-scalability for a while, speeding up as more users c
ame online and barely hitting its straps below 250. In contrast with AIX, however, NT allowed DB2’s performance to droop above this level. When Microsoft’s SQL Server was run up on its native platform, NT, the results were not spectacular. With four CPUs the system sagged slowly but steadily as more users were added. It might appear that Bloor’s test engineers gave up on SQL Server/NT at the 250-user mark. In fact they retuned the system to use only three of the four available CPUs, backed off to 200 users, then carried on to complete the test at the same maximum load – 384 users – that was applied to the two DB2 systems. Predictably, having only three CPUs available gave this configuration the worst performance figures of all, as indicated by the line at the lower right of the graph. Perhaps strangely, it appears not to have affected the scalability of the all-Microsoft system, which was just as poor as on four CPUs; adding more users caused performance to decline at roughly the same rate.
Content from our partners
Fatal seizure
Why did Bloor switch from four CPUs to three when testing the Microsoft system for user scalability? Leaving aside its poorer single-box results, and whether clustering or enhanced processor scalability is the way to improve on them, the answer to this question is key to Microsoft’s readiness for the large-enterprise market. From the Bloor report: To use [all four CPUs], when there are over 350 users, invites the possibility of a fatal seizure of Microsoft SQL Server for Windows NT. The only way that was found to get around this was to leave one processor free (a ‘3 + 1’ configuration). Ouch.
