In a report released Wednesday, Microsoft spelled out how it tests browsers in-house, and again stressed that it doesn't buy the idea that benchmarks — such as those that score JavaScript performance — accurately compare the players.
“These benchmarks necessarily characterize only a narrow set of the browser functions in a very constrained way,” Microsoft's report said. “End users, however, do not operate in a controlled environment.”
Microsoft's tests pitted IE8 Release Candidate 1 (RC1), which launched in late January, against Google Inc.'s Chrome 1.0 and Mozilla Corp.'s Firefox 3.0.5, a version from mid-December. The company timed how long it took each browser to completely render the 25 most-popular destinations on the Web, as ranked by the Web metrics firm comScore Inc., which included google.com, facebook.com, amazon.com, and others.
IE8 was fastest in rendering 12 of the 25 sites, said Microsoft, while Chrome took second by beating the others on nine sites. Firefox, meanwhile, was a distant third, coming in first on just four of the 25 domains.
Microsoft did not test other browsers, such as Apple Inc.'s Safari or Opera Software ASA's Opera, said James Pratt, a senior product manager on the IE development team, because it wanted to focus on rivals that “had a good share on the Windows platform.”
Both Opera and Safari for Windows have shares of less than 1%, according to the most recent data from Net Applications Inc., with the former, on all platforms, accounting for 0.7% and the latter just 0.3%.
Nor did Microsoft put IE8 in the ring with later versions of Chrome and Firefox. Chrome, for instance, is currently at 2.0.169.1 as a developer-only build, while Firefox just rolled out 3.1 Beta 2. Both browsers boast better performance, specifically faster JavaScript rendering. “IE8 RC1 is a release candidate, and was very close to being done,” explained Pratt when asked why newer versions of Chrome and Firefox had not been used. “But Google and Mozilla were still actively working on [those newer browsers], and they weren't super stable.”
JavaScript benchmarks have become a point of dispute between Microsoft and its rivals. While Mozilla, Google, Apple and Opera have all updated their JavaScript engines in the last eight months, and have then trumpeted scores in JavaScript test suites like SunSpider, Microsoft executives have dismissed the bragging as so much noise.
Dean Hachamovitch, IE's general manager, has called claims of competitors a “drag race” that Microsoft isn't interested in joining, while Pratt has downplayed comparisons of any kind. “We're at the point, with what people do in the browser, that users can't really tell the difference between browser [performance],” he said in a January interview.
Pratt said that the just-released report backed that up. “As you can see from the scores, the differences between the browsers are actually very small,” he said.
When Computerworld last tested the major browsers' JavaScript performance, immediately after the release of the public beta of Safari 4, IE8 ranked last.
Although Google did not respond to a request for comment on Microsoft's benchmarks, Mozilla's Mike Shaver, who heads all development at the company, applauded any attempt to boost IE's performance. “I don't think anyone here has had a chance to really look at their methodology yet or tried to reproduce their results, but to whatever extent Microsoft is working to improve the performance of IE it's a good thing for the Web,” said Shaver in an e-mail late Thursday.
Microsoft's browser performance report can be downloaded from the company's Web site.
