Mixing green flavor in high performance computing

The supercomputing world is one of giant government labs, big machines and speeds measured in hundreds of trillions of calculations per second. But a new company called SiCortex is trying to bring the benefits of high-performance computing to smaller enterprises and research groups, particularly those worried about the rising cost of electricity.

Competing against the likes of Dell, HP and IBM, SiCortex officials hope their approach combining high-performance computing with energy-efficient design is disruptive enough to shake up what they believe has become a stagnant market.

“This is a systems company, and it's been a long time since anybody actually built and deployed a systems company,” says chief engineer and co-founder Matt Reilly.

SiCortex has 84 employees and is headed up by CEO Chris Stone, a former Novell executive responsible for engineering and product management from 2002 to 2004.

Also among its top executives is co-founder and chief architect Jud Leonard, who previously co-founded Agile Systems and TLW. Both Leonard and Reilly worked at Digital Equipment Corporation, while Reilly is also a veteran of Compaq and Intel.

SiCortex shipped its first beta machine in July 2007 and went into production early in 2008.

In an attempt to convince customers that its systems really are unusually efficient, SiCortex developed a benchmark called the Green Computing Performance Index, which measures performance per kilowatt and gives SiCortex a score about 70% better than the IBM Blue Gene/P, one of the most advanced supercomputers in the world.

“We chose a processor whose design supported power savings in ways that were important to us,” Leonard says. “We watched where power was being spent and worked very hard to control it. It's a matter of tackling it up and down the line. That's how you get order-of-magnitude savings, instead of 15% savings.”

SiCortex sells three machines: a desktop computer with 72 processors, a mid-range system with 1,458 processors, and the biggest of all, a 5,832-processor system that costs more than $1 million and delivers speeds of eight teraflops, which means it can perform 8 trillion calculations per second. The fastest supercomputer in the world, an IBM machine based at Los Alamos National Laboratory in New Mexico, performs more than 1,000 trillion calculations per second.

But SiCortex isn't aiming to build the fastest supercomputer in the world, or even the most reliable. The markets the company targets — small enterprises, collaborative groups, university departments and divisions within national labs — won't spend enough to get five nines of availability, says SiCortex CTO John Goodhue.

For its processors, SiCortex purchased intellectual property from several sources including the company MIPS Technologies, and modified the design to suit its own needs. The Linux-based machines use a network of Leonard's design.

“In an Ethernet or Infiniband environment, you have processing nodes and you have separate switches. That's called an indirect network,” Leonard says. “Ours is a direct network in which each component of the system includes a small portion of the switch fabric and you wire them together so you don't need a separate component to do the fabric switching.”

An individual processor can be very fast, but isn't worth much in a cluster if the entire system has slow communication, Reilly says.

“If you're going to scale up to hundreds of thousands of processors you have to get the communication bottleneck out of the way or at least you have to mitigate it,” Reilly says. “So we start with a relatively modest processor and connect it to a really fabulous communications network.”

SiCortex has sold about 60 computers, which are named for the number of processors (for instance, SC072, SC1458 and SC5832). Inside the roughly 6-foot-tall 5832 are as many as 36 motherboards, each of which has 162 processors. Nine of the motherboards fit into the 1458, which costs several hundred thousand dollars depending on the configuration, while the 72-processor desktop uses a smaller version of the SiCortex motherboard.

The desktop by itself costs about $25,000, but organizations that buy the larger systems get a couple free.

Purdue University, among SiCortex's earliest customers, purchased the 5832 and has outfitted it with 4,536 processors so far. Purdue CIO Gerry McCartney is impressed by the vendor's ability to provide massive computational power at a low cost, but notes that it can't be used in all high-performance computing environments.

Rather than provide ultra-fast processors, SiCortex takes inexpensive, slow processors and gives you a ton of them stitched together with very fast interconnect technology, McCartney says.

“Here's the difficulty: Which is better, a motor coach that can hold 50 people, 10 Chevettes that can hold 50 people, or 50 Ferraris that can move 50 people? [SiCortex] is the Chevette model,” he says.

SiCortex's model is ideal for many applications that require parallel processing, McCartney says. A 5832 might be useful for designers of animated films, who need to render millions of frames, he speculates. At Purdue, researchers in computational fluid dynamics are using the SiCortex for research on jet engines.

But applications that require huge amounts of sequential calculations would not be ideal for a SiCortex system, McCartney says.

“It's a niche machine,” he says. “It's not for all types of high-performance computing. It's an awful lot of very slow processors. [That's good] if you can chop your jobs up into bite-sized pieces. If you can only use one processor at a time, this thing's going to be horrible. You've really got to be able to chop your job up so you can use a couple hundred processors simultaneously.”

If the SiCortex model does fit your computing needs, it brings great benefits in terms of power and cooling. If not for SiCortex, McCartney says he would need to buy additional cooling equipment and transformer capacity in order to substantially increase his computing power.

“To put in a couple million dollar machine, I have to spend at least another two or three million on infrastructure that doesn't do anything for me compute-wise,” he says. “In fact, it just increases my electricity bill.”

In addition to Purdue, SiCortex's customer base includes NASA, the Department of Energy's Argonne National Lab, an unnamed intelligence agency within the Department of Defense, General Electric and various other universities including Yale and Columbia.

Sales have included about 10 SC5832s, and about 20 mid-range machines, including the SC1458 and a 648-processor model that was discontinued. About half of the sales were desktop machines, on which customers can develop applications for use on the larger SiCortex systems.

While the systems are mostly in research settings today, the men behind SiCortex don't see any reason to limit its reach. Like almost every IT vendor, SiCortex is examining the current fascination with cloud computing, and says its own architecture can help with the database acceleration, searching and caching necessary to make the Web go faster. The SC5832 holds up to 8TB of RAM and every processor can get to that large amount of memory pretty quickly, Goodhue says.

For now SiCortex officials say they are having success selling to organizations that are facing strict power limits and must find some way to get more out of their computing infrastructure.

Your typical IT manager may not have to worry about the power bill, but as Reilly says, “there are people out there for whom the energy-per-answer is the key.”