Symantec cripples Windows XP with defective update

Symantec last week crippled a large number of Windows XP machines when it shipped customers a defective update to its antivirus software, the company acknowledged Friday.

“After a full evaluation and root cause analysis … we have determined that the issue was limited to machines running a combination of Windows XP, the latest version of the SONAR technology, the July 11th rev11 SONAR signature set, and certain third-party software,” said Orla Cox, of the company’s security response team in a July 14 blog post.

SONAR, for “Symantec Online Network for Advanced Response,” is an anti-malware technology that spots suspicious, and possibly malicious, files by monitoring software behavior.

Symantec did not identify the “certain third party software” that contributed to the problem, which caused Windows XP PCs to show the notorious “Blue Screen of Death” (BSOD) error display, then reboot, only to endlessly repeat the cycle.

The closest the company came to pointing fingers was to note that the blue screens were triggered by software that “implements a file system driver using kernel stack-based file objects, typical of encryption drivers.”

The SONAR update caused new file operations that created the conflict that led to the system crashes, Symantec said.

Users of Symantec Endpoint Protection (SEP), antivirus software, run primarily by enterprises, began reporting blue-screening XP systems early Thursday, July 12. Symantec later confirmed that other titles in its portfolio, including the consumer-grade Norton 2010, 2011 and 2012, as well as Norton 360, were also affected.

The flawed update was served to customers for about eight hours, from 6:25 p.m. PT on July 11 to 2:15 a.m. PT July 12, when Symantec yanked the update. It replaced the defective update about a half hour later.

Some users reported substantial numbers of affected Windows XP machines. Someone identified as Mark Daeth said more than 1,000 systems at his workplace had blue-screened.

“We have pushed out R12 to as many PCs as we can, but over 30% of our PC environment still will not boot,” said Daeth on Thursday, referring to the revised SONAR update.

Daeth is the IT manager at Charlotte-based AAA Carolinas, the American Automobile Association group responsible for North Carolina and South Carolina members.

Not surprisingly, customers were irate, with one calling the gaffe “a total farce.”

“The support is a joke, the quality control is a joke and the software is not much better,” charged Andrew Parkes in a comment appended to the Symantec blog. “Yes, I know these things happen, but any half decent quality control/testing process would surely of highlighted the issue?”

Symantec is the second antivirus vendor to cripple or damage Windows systems with a flawed update in the last two months.

In mid-May, German security company Avira released a buggy signature update to its behavioral-based monitoring system that blocked virtually every legitimate Windows executable file — those with the “.exe” extension — and prevented applications from launching.

Nearly every major antivirus software supplier — including McAfee, Microsoft, Symantec and Trend Micro — have shipped defective definitions. In some cases, those mistakes have wreaked as much or more havoc as the latest blunder by Symantec.

According to security vendor Opswat, which reports on usage share every quarter, Symantec accounted for 15.1% of the all operating antivirus products in North American as of June — second behind Microsoft’s 22% — and 10.3% globally, for the fourth spot after Avast, Microsoft and ESET.

In light of the blue screens, Symantec said it was revamping its quality assurance testing process “to improve compatibility testing” and would hold off any future SONAR signature updates until the new procedures were in place.

A document on Symantec’s website describes the issue and offers a workaround.