New Internal IBM Report Says "Another Flawed Study"After publicly retracting the results of J2EE versus .NET benchmark tests it conducted back in 2002, The Middleware Company (TMC) bravely ventured recently to revisit this minefield. From IBM's point of view, TMC has managed to blow itself up all over again! Read on...*** Who takes these "studies" seriously anymore?They are flawed in so many aspects. Most of these studies are financed by the companies themselves and they are always objective right? The applications used in the benchmark are typically non-representative for most businesses. A benchmark that says that such-and-such infrastructure is better to run the benchmark is just saying that. This is not necessarily valid for my application, running my workload on my hardware. In the comparisons are never working from the same base. Actually they cannot be, because the tested architectures are significantly different. Also, if company A does the test comparing A and B, they will make sure that A is completely optimized and streamlined and B is just a plain out-of-the-box installation. The bickering after the "study" will completely invalidate the whole report. Lies, lies and statistics As mentioned in an earlier article here, I just find it interesting how IBM and Microsoft fight over these silly reports. And the article certainly brings home some points that are pitfalls for Performance Testers. The following comment sums it up nicely: "A real performance test should compare one competitor to another with as much commonality as possible. "