Data Center Architecture: Where to Place the Server Access Switch

Data Center Architecture: Where to Place the Server Access Switch

A server access network switch serves a critical role in network design, as the point at which servers initially connect to the network. In a typical data center, conventional wisdom says this server access network switch should be placed on the top of a server rack (ToR). However, is this always the best solution? This article will discuss both ToR placement, and some viable alternatives.

Single-Rack Challenges and Solutions

The ToR placement provides excellent cable management, rack-level modularity, and scalability. However, there can be some downsides to this placement. To connect servers to the ToR switch, customers must maintain an inventory of several different lengths of patch cords or direct-attach cables (DAC) to cover the entire rack.

Instead of top of rack, placing the server access switch at the middle of rack (as pictured below) can help reduce the number of cord lengths by approximately 50%, thereby reducing operational complexity through simplifying sparing inventory.

Diagram depicting switch in the middle of a server rack.
Switch in the middle of a server rack

Going Above and Beyond the Rack

The ToR placement may lead to further challenges. For example, a ToR switch typically has 48 ports for server connections. If network operators wish to fully utilize all 48 ports to connect servers in the same server rack, the customer would need to install 48 1RU servers in the same rack. The first issue is a logistical inefficiency: a common 42RU rack or cabinet will necessitate each ToR switch can only operate at 87.5% capacity. Leaving six open ports on every ToR switch can have a significant impact on the overall ROI for this equipment.

Even if taller racks can be used, this can introduce power and cooling challenges. A fully-populated rack of 48 servers may consume approximately 30kW per rack, which is more than double the average data center power density of less than 15kW per rack. Additionally, 30kW per rack would present a great challenge to maintain sufficient cooling levels.

One solution to all these challenges is to place the server access switch on top of multiple racks (ToMR), using the space above server racks for a switch that can serve multiple racks. The ToMR placement maintains great cable management, modularity and scalability of a standard ToR placement, but enables the user to fully utilize all switch access ports and minimize the challenges presented by high heat and power density. This can be particularly effective when multiple networking speeds are involved, as shown below.

Switch at the top of multiple racks of servers with assorted data rates.
Top of Multiple Rack (ToMR) Switch

In the case illustrated above, a 100G switch sits above individual racks of 10G, 25G, and 50G servers. In this scenario, all ports of the switch can be used without overburdening the data center’s power distribution system.

Champion ONE Solutions

Champion ONE’s full line of Intelliport Open Networking Switches include solutions for each of these scenarios. Our 10G switch is an ideal top (or middle) of the rack solution, with 48 SFP+ ports for server access, plus 6 40G uplink ports. For ToMR solution, consider our 32-port 100G switch. Each of the 32 QSFP28 ports can be configured to support 4x10G, 4x25G, 1x40G, 2x50G, or 1x100G with appropriate breakout cables.

Contact us today to find out which solution is right for your data center.

Share on facebook
Share on twitter
Share on linkedin