In the SOA space generally and business process management (BPM) market specifically, there is little in the way of consistent, comparable benchmarks to help users make more informed purchasing decisions. On the other hand, speed is only one of the things requried of an ESB (as some of the commenters below have been pointing out).
So is it valuable, or misleading, to see transaction benchmark figures? One vendor, open source BPM player Intalio, has just published some benchmarks for BPEL transactions, albeit their own rather than an industry standard benchmark (because there is no such thing).
Intalio announced a new benchmark in which it says it achieved record performance in the execution of business process execution language (BPEL) processes.
Intalio BPMS version 4.3 executed over 3.5 million persistent processes in less than 24 hours, running on a dual-CPU machine equipped with Intel Xeon processors and 2GB of RAM. Persistence was offered by a single instance of the MySQL database engine deployed on a separate server. When persistence was de-activated, over 17 million transient processes were executed within the same 24-hour period, according to the company.
It also demonstrated in a separate benchmark that roundtrip calls made to external web services through an embedded enterprise service bus (ESB) could be completed within less than 14 milliseconds, making it in effect a real-time BPM system.
“These results make us confident that real-time processes can be deployed on top of a service-oriented architecture (SOA) today without significant impact in terms of performance,” said Ismael Ghalimi, CEO of Intalio.
I’m on a parallel news deadline right now so will not state categorically that other vendors have not put out similar benchmarks — I’ll have a scout around tomorrow and see what I can dig up. Back in August 2005 Cape Clear’s CEO Annrai O’Toole told me that his company at that time held the record for the highest number of concurrent BPEL transactions, 40,000, in an ESB implementation for its customer Stratus. The number of transactions that can be handled in 24 hours, and the number of concurrent transactions, are two different metrics of course.
In the mean time, if you are a vendor (or indeed any other reader) and you have a benchmark for BPEL transactions, feel free to add it to the comments section below and I will update this blog as I receive new benchmark data.
UPDATE: I just found this document by the School of Engineering & Applied Science at Washington University. While it offers some benchmark data only for a small-footprint BPEL workflow engine for mobile devices called Silver, their take on what to look for in a BPEL benchmark and how to try and compare like with like is quite informative.
UPDATE 2: Intalio’s Ismael Ghalimi confirmed that the benchmark is their own, rather than an industry standard. He told me: “The benchmark has been internally developed. We will polish it and publish it with detailed documentation so that other vendors can use it as well.” It will be interesting to see whether any other vendors choose to aim their own ESBs at Intalio’s benchmark – there may be concern that it is a benchmark that plays into their own technology’s areas of strength.
[Incidentally if you navigated over here because you are more interested in open source than ESBs or SOA, then be sure to check out my colleague Matthew Aslett’s dedicated Open Source blog here.]
