Fiber Channel, CEE, and FCoE – what they’re really all about

Over the last ten years, Fiber Channel has become the standard technology worldwide for Storage Area Networks (SANs) and external storage systems in the mid- and high-end ranges. Especially in terms of external storage systems, Fiber Channel has slowly but surely replaced parallel SCSI as the standard interface.Today, Fiber Channel is, and will remain, the de facto standard on the data storage market for many years.

Fiber Channel

An important factor for Fiber Channel's success is, and has been, the unique advantages of new storage applications specially developed by Fiber Channel. The most widely used Fiber Channel-based solutions can be found in the areas of storage consolidation and connectivity.

On the one hand, Fiber Channel has also developed high-performance backup solutions that heavily reduce backup times, as well as special backup applications that make “LAN Free” and “Serverless” backup possible. On the other, they have developed solutions for a broad group of users that deal with redundant data storage using mirroring or replication. Redundant data storage is often combined with error- and catastrophe-tolerant solutions that use “Fail Over” over great distances, but also within data centers and also here for environments in which such “features” had not been financially feasible before.

Fiber Channel has developed data mobility for a wide range of users. Another of Fiber Channel's decisive advantages is improved possibilities for managing storage and storage networks. Cost-efficient management, despite the annual multiplication of the storage to be managed, faster and more efficient allocation of storage resources, and simplified change management.

Storage virtualization is difficult to realize without the networking and flexible allocation of resources within SAN environments. Even the virtualization level itself is often realized in fabric-based SAN .

To describe Fiber Channel as merely “another network protocol” would not do justice to the actual demands of the data storage market. In comparison to the classic and most successful network technology, the Ethernet, there are several concrete differences.

Let's start with the fundamental development requirements:

In essence, Fiber Channel must replace the cable between server and storage, supplying in effect a very fast and reliable connection between these system components. If this connection fails, even for just an extremely short time, or if individual data blocks are lost, changed, or transferred in the wrong order, the consequences for the server can be fatal: usually a server failure or at least the failure of important applications. On the operating system side, there are also no mechanisms for catching such fatal errors. The cable connections that Fiber Channel replaces can thus be considered extremely critical.

Ethernet and its protocols

Ethernet was developed about 30 years ago out of a need for different computers to exchange data directly and use common resources (e.g. printers). In other words, to connect autonomous computer systems. Although it is inconvenient when this connection fails, the networked computers do not also fail as a result. For this reason, and also to keep the infrastructure cheap and simple, a “Collision Domain” was created for use with Ethernet, i.e. an environment of network nodes with a clear addresses (MAC addresses) that are linked together in a chain (Daisy Chain) connected to the same medium (cable). All nodes receive all data packets and use the MAC address in the block header to verify that the packet was sent to nodes on the same network. If one node wants to send packets, the network node waits for a short, specified amount of time during which no data can be sent on the cable (“Carrier Sense [CS]”) before sending the data packets. If two or more network nodes attempt to send data over the cable simultaneously (“Multiple Access [MA]”), both are corrupted, i.e. a “collision” occurs. All network nodes are equipped with “Collision Detection (CD)”, so that they know which data blocks were not transferred. There is no form of acknowledgement. It is obvious that this rudimentary CSMA/CD protocol will quickly outgrow its usefulness. However, Ethernet was already very widely distributed, relatively affordable, and simple to use. The solution for more complex demands and high scalability was provided by “Top Level” protocols. TCP and IP (TCP/IP) are the two most commonly used today. Unfortunately, this has caused a dramatic rise in both the protocol overhead and latency times. While modern Ethernet-based LANs really don't have much in common with the initial Ethernet configurations, Ethernet is still based on the same protocol standard, with the same disadvantages.

•No guarantee for the order of the data blocks transferred (“In-order delivery”)

•No “Lossless transport” of data blocks

•No “Acknowledged link level”

•No “Buffer to buffer management”

•No Name server (SNS) for recognizing and allocating resources

•No possibility of defining zones or distributing resources among zones

•No “Remote state change notifications (RSCNs)”

Unfortunately, these are precisely the characteristics that are essential for connecting storage systems with servers. Because, as explained earlier, the extremely critical SCSI cable is replaced by a network.

It is also possible to map these characteristics in higher protocol levels, in what are referred to as Top Level Protocols, but this has a significant negative impact on both the overhead and the latency times.

It is important that Ethernet not be equated with TCP/IP. Ethernet comprises the physical and transport levels (layers 1 and 2), while TCP and IP comprise higher levels. The TCP/IP protocol can thus also be transferred over other transport layers, such as Fiber Channel.

Fiber Channel and CEE

Then why make a comparison between two network protocols with such different characteristics? Both serve a useful a purpose in their own fields of application and have been developed to meet certain requirements. In computer centers, both protocols are the dominant network environments for their respective fields of application: Ethernet for computer-computer linking and Fiber Channel for computer-storage linking.

However, the ability to use both technologies together, not just in parallel, would help simplify the network infrastructure, specifically with regard to cabling. Moreover, just one interface on the computer would suffice to connect both network and storage capabilities.

That Ethernet infrastructure, which is so commonly used today, can be used without modification as network infrastructure for storage connections in layers 1 and 2 has already been discussed. However, an improved protocol, better suited to meet today's growing requirements for both environments –that is, computer-computer and computer-storage networks – could be used instead.

This improved Ethernet protocol is often called “Converged Enhanced Ethernet (CEE). CEE is, however, not an initiative from an individual manufacturer (like “Datacenter Ethernet (DCE),” for example) but rather a recent step in the progress of technology designed to meet new industry standards.

CEE is not merely a “new Ethernet,” but rather a protocol that meets the needs of both storage networks (SANs) and classic networks. For this reason, CEE cannot function in the Ethernet infrastructure that is currently in use in computer centers.

To convert “Standard Ethernet” to “Converged Enhanced Ethernet” (CEE), the previously mentioned requirements of the storage environment must be absolutely met. At 10 GB/sec, even the first generation of CEE is an extremely fast network. More than 16 manufacturers are working with the IEEE and IETF institutions to develop standards.

Due to its high requirements, CEE is a complex and therefore – at least in the foreseeable future – not inexpensive technology that is principally of interest to large computer centers. For smaller and mid-sized facilities, the classic Ethernet infrastructure for computer-computer linking and Fiber Channel for computer-storage linking will likely remain the dominant technologies.

One topic that warrants a brief comment here is the physical layer. As demonstrated by the 10 GB/sec transfer rate, CEE places different demands on the physical layer 1 than compared with the classic gigabyte Ethernet. This is another reason that implementing CEE requires a new Ethernet infrastructure.

CEE's most important improvement as compared with classic Ethernet is the elimination of the risk of packet loss. To do this, Fiber Channel uses a buffer management system based on “Buffer-to-buffer credits” with corresponding confirmation by the “R-RDY” frame. Such input buffers are not used in classic Ethernet. Instead, “Media access control” (MAC) frames, which enable step-by-step control of incoming frames, are used. The “IEE 0802.3X flow control standard” used for CEE is based on such a “Pause” frame, which can hold the link's transmission port for a certain period of time. A second “Pause” frame with the value “0” can reactivate the data transfer before the pause time expires.

In conjunction with an appropriate input buffer, lossless transport of frames is possible, although it may not be quite as efficient as with Fiber Channel. As with Fiber Channel, IEE 0802.3X must be implemented bi-directionally to ensure performance comparable to Fiber Channel.

The IEEE initiative “Priority-based flow control” (FCF) deals with the prioritization of frames (PFC). This standard IEEE 802.1Qbb allows application-specific bandwidth reservations in CEE. PFC can ensure that time-critical protocols and applications can work at a higher priority, and that sufficient bandwidth for these protocols and applications is reserved.

Ethernet Congestion Management (ECM) uses another technology to achieve reliable flow control between endpoints and to maximize the available bandwidth. The IEEE 802.1Qau Group is concerned with the recognition and deactivation of overload statuses in network nodes by using “mirroring” of frames at the starting point (instead of “frame dropping”), if these frames cannot be immediately resent in case of overload. The sender slows down the transfer of frames until no more “reflected” frames are received. In this way, overloads can be dealt with without losing frames.

In what ways are CEE and standard Ethernet similar?

CEE makes available more advanced and improved physical and transport layers for standard Ethernet frames. Frames can be transferred as usual, including Top Level protocols. However, layers 1 and 2 are significantly altered, which means the existing Ethernet infrastructure can no longer be used.

Let's take a closer look at the transfer of Fiber Channel protocol using this “new Ethernet,” i.e. “Fiber Channel over Ethernet (FCoE)”. In this case, ANSI T11 is responsible for standardization.

FCoE

Actually, it should be called “FCoCEE.” “FcoE” gives the impression that the Fiber Channel protocol can be used on any Ethernet. As previously described, this is not the case!

In principle, FCoE is a simple “encapsulation” of all Fiber Channel frames in one Ethernet frame. One difficulty here is that the maximum size of an Ethernet frame is 1518 bytes, while a Fiber Channel frame can be as long as 2112 bytes. Splicing the FC frames would result in unnecessary time and cause delays. For this reason, “jumbo frames” are used. Jumbo frames can be up to 9 KB – large enough for an entire Fiber Channel frame. The size of these “payloads”, in other words, the “freight capacity” of the jumbo frames, is unrestricted, so that potentially larger Fiber Channel frames can be later “encapsulated”.

In FCoE, the Fiber Channel frame is packaged in a completely unaltered state in the payload of an Ethernet jumbo frame. What is still missing are the “fabric services” that are included perforce in Fiber Channel environments. The most important ones are the Fabric Controller and the Simple Name Server (not to be confused with the Name Server familiar to TCP/IP environments). This means that FCoE environments do not consist of just a single CEE network, but rather that the corresponding fabric services from the infrastructure must also be kept readily available. If fabric services are not available in a given CEE environment, then FCoE is impossible.