What is an ethernet fabric?

In a classic Ethernet network, the connections between switches, or Inter-Switch Links are not allowed to form a loop or frames aren’t delivered. Spanning Tree Protocol (STP) prevents loops by creating a tree topology with only one active path between any two switches. This means that ISL bandwidth is limited to a single logical connection, since multiple connections between switches are prohibited. Enhancements to Ethernet tried to overcome this limitation. Link Aggregation Groups (LAGs) were defined so that multiple links between switches were treated as a single connection without forming loops. But, a LAG must be manually configured on each port in the LAG and is not very flexible.

The tree topology requires traffic to move up and down the tree, or "north-south,” to get to an adjacent rack. When most of the access traffic is between servers in a rack, this is not a problem. But server clusters, such as those required for clustered applications and server virtualization, have traffic between servers in multiple racks, travelling "east-west,” so the tree topology increases latency with multiple hops and restricts bandwidth with single links between switches.

STP automatically recovers when a link is lost. However, it halts all traffic through the network and must reconfigure the single path between all switches in the network before allowing traffic to flow again. Halting all traffic for tens of seconds up to minutes on all links limits scalability and constrains traffic to applications that can tolerate data path blocking to achieve link resiliency. In the past, traffic relied on TCP to handle this interruption in service, but today, with almost all data center applications running in a 24 x 7 high availability mode and storage traffic growing on the Ethernet network, loss of connectivity in the data path for even a few seconds is unacceptable.

Classic Ethernet switch architecture presents other limitations. Each switch has its own control and management planes. Each switch has to discover and process the protocol of eachas it arrives on an ingress port. As more switches are added, protocol processing time increases adding latency. Each switch and each port in the switch has to be configured individually, since there is no sharing of common configuration and policy information between switches. Complexity increases, configuration mistakes increase, and operations and management resources do not scale well.

In an Ethernet fabric, the control path replaces STP with link state routing, while the data path provides equal-cost multipath forwarding at Layer 2 so data always takes the shortest path using multiple ISL connections without loops. Combined with the fabric’s control plane, scaling bandwidth is made simple. For example, it becomes possible to automate the formation of a new trunk when a new switch connects to any other switch in the fabric.

If a trunk link fails or is removed, traffic is rebalanced on the existing links non-disruptively.

Finally, if an ISL is added or removed anywhere in the fabric, traffic on other ISLs continues to flow instead of halting as with STP.

With this architecture in place, a group of switches can be defined as part of a "logical chassis”, similar to port cards in a chassis switch. This simplifies management, monitoring, and operations since policy and security configuration parameters can be easily shared across all switches in the logical chassis. In addition, because information about connections to physical and virtual servers and storage is now known to all switches in the fabric, the fabric can ensure all network policies and security settings continue to be applied to any given virtual machine no matter whether it moves or where it resides.

In modern, virtualized data centers, IT groups need to better scale virtual server environments, provide application mobility, and reduce infrastructure complexity and management overhead.

Scaling virtual Server Environments

When organisations attempt to scale virtual server environments, the network often presents challenges due to Spanning Tree Protocol, the growing number of GbE connections per server, low utilization, and link failure recovery.

Enabling virtualization capabilities, such as Virtual Machine (VM) mobility, requires VMs to migrate within a single Layer 2 network, since non-disruptive migration of VMs across Virtual LANs (VLANs) using Layer 3 protocols is not supported by virtualization hypervisors.

In traditional Layer 2 Ethernet networks, to create a highly available network, organisations designate paths through the network as active or standby using STP. While this provides an alternate path, only one path can be used at a time, which means that network bandwidth is not well utilized. Since one of the goals of server virtualization is to increase utilization of the physical server, increased utilization of network bandwidth should also be expected.

To increase network utilization, Multiple Spanning Tree Protocol (MSTP) and similar protocols allow for separate spanning trees per VLAN. While this improves bandwidth utilization, the STP limit of one active path between switches remains. And, because traffic paths are manually configured with MSTP, complexity increases.

Another challenge with STP is network behavior when links fail. When failures occur, the spanning tree needs to be redefined. This can take anywhere from five seconds with Rapid Spanning Tree (RSTP) up to several minutes with STP—and this convergence can vary unpredictably even with small topology changes. The demands for non-stop traffic flow increases with server virtualization, and consequently network convergence times have to shrink. STP does not provide an adequate solution for these requirements. Finally, when a spanning tree is reconverging, broadcast storms can occur and result in network slowdown. All of these limitations of STP are why Layer 2 networks are typically kept small in the data center.

Compared to classic hierarchical Ethernet architectures, Ethernet fabrics provide higher levels of performance, utilization, availability, and simplicity.

They have the following characteristics at a minimum:

Flatter

Ethernet fabrics eliminate the need for Spanning Tree Protocol, while still being completely interoperable with existing Ethernet networks.

Flexible

Can be architected in any topology to best meet the needs of any variety of workloads.

Resilient

Multiple "least cost” paths are used for high performance and high reliability.

Elastic

Easily scales up and down at need.

More advanced Ethernet fabrics borrow further from Fibre Channel fabric constructs:

• They are self-forming and function as a single logical entity, in which all switches automatically know about each other and all connected physical and logical devices.
• Management can then be domainbased rather than device-based, and defined by policy rather than repetitive procedures.
• These features, along with virtualization-specific enhancements, make it easier to explicitly address the challenges of VM automation within the network, thereby facilitating better IT automation.
• Protocol convergence (eg Fibre Channel over Ethernet, or FCOE) may also be a feature, intended as a means of better bridging LAN and SAN traffic.