Facebook networking chief: No more secret ASIC commands

Facebook’s head of network operations has great expectations for software defined networking (SDN), though he may not be relying on commercial hardware vendors to bring SDN to the social networking giant’s own infrastructure. 

“SDN is the way things are going to be. It is not a fad. This will be the way the networks will be built going forward,” said Najam Ahmad, director of technical operations at Facebook, who oversees Facebook’s production and corporate network. Prior to joining Facebook, Ahmad worked as general manager of global networking services at Microsoft.

Ahmad was part of a panel about SDN at the Interop New York conference, held this week. He spoke with the IDG News Service afterward.

SDN has generated a lot of discussion at this year’s Interop. Many wondered if it is just this year’s buzz phrase, or as others believe, the future of networking.

For Ahmad, SDN solves an important problem – making Facebook’s network as flexible as the rest of its IT stack. “We want to deploy, manage, monitor and fix the network using software,” Ahmad said.

Today’s switches, routers and other network equipment stymies Facebook from making the most efficient use of its network and the Internet, he said. The company needs to reduce its network latency as much as possible, simply in order to remain responsive to its billion users scattered around the globe.

Admittedly, Facebook doesn’t manage its network with typical network operation centre (NOC)-styled operations, in which network administrators monitor screens for alerts and then fix problems as they arise.

Instead, the company aggressively automates network management as much as possible, by writing scripts that can anticipate and mitigate issues before they come up, as well as to maximise network performance.

Facebook has been limited, however, by its networking equipment. “With traditional networking, you buy a box. You get command line interface, and protocols, but that is all you get,” Ahmad said.

Ahmad has felt frustrated, for instance, when finding a certain issue in some networking gear could only be addressed by the vendor dialing directly into the equipment itself and issuing secret commands to an ASIC (application-specific integrated circuit), commands not available to customers through the standard command line interface (CLI).

“We can’t manage our environment like that,” Ahmad said. “We want to write against the hardware directly.”

Today, the only way you can get changes in products is to convince the vendor to make the changes or appeal to a standards body, Ahmad said.

Being a rather large company, Facebook does have some influence with hardware manufacturers in terms of what features to add, or changes to make. But this process is too slow and cumbersome for the company’s liking.

“You work with them for six months, get an update, go test it and then a year later have something,” Ahmad said. “It is hampering our ability to actually build stuff and manage stuff at our scale.”

SDN allows the networking equipment to be “disaggregated,” Ahmad said, in much the same way the computer itself was as it moved from monolithic mainframes to personal computers, which are built from standardised components. Today, a computer is a collection of discrete components – the hardware, the operating system (OS) and the applications. Developers then can write programs, or build components to address a certain need.

“We want the same structure in network,” Ahmad said. “We want to do things in software. When you do things in a closed vertically integrated model, you can’t do software development. You only have CLI.”

For Facebook, the value in SDN is that it makes more of the underlying architecture of network equipment available so that it gives programmers more control of the hardware. This is one of the reasons Facebook started the [Open Compute Project], which seeks to create a set of specifications for data centre hardware that will be open to programmers.

“We know what problems we have today, we don’t know what we will run into tomorrow, when we do run into something, we want to fix it very quickly,” Ahmad said.

For instance, Facebook would like to do more with the Border Gateway Protocol (BGP), the protocol used to make core routing decisions on the Internet. BGP estimates the best path that packets should take to get to their destination, though Ahmad would like to improve process with working with BGP by having more control over routers.

“BGP doesn’t do a very good job at path selection,” Ahmad said, noting that once a path is selected, there is no feedback mechanism to shift to another path should the transmission experience congestion en route.

Despite the company’s optimism about SDN, Facebook has thus far not purchased much commercial-off-the-shelf SDN software.

Part of the reason for this is that Facebook does not use virtualisation, upon which many of today’s SDN offerings rely on that technology. VMware, for instance, has weaved OpenFlow into its own virtualisation offerings, having purchased OpenFlow creator Nicira last year.

“SDN is more a philosophy than a product,” Ahmad said. “OpenFlow is a product to make SDN happen, but there are other ways to do SDN as well.”

Nonetheless, the company is bullish on SDN, the philosophy. Some SDN software could even be released as open source from Facebook itself, in much the same way it released its own performance improvements to the PHP language. The company is hoping that, as with Linux and other open source projects, programmers can add more functionality to networking than any single vendor could.

“Networking needs to come out of the dark ages, or the closed era, and come into the open. Gone are the days of the tightly integrated stuff. We need to leverage the large developer base out there to get networking moving a lot faster than it is today,” Ahmad said.