A rack uplink design is not simply a switch-count or port-count decision.
In an AI cluster, first separate the frontend fabric that carries user, API, logging, and data-ingestion traffic from the backend fabric used for GPU collective communication. This article explains where Cisco’s architecture-specific oversubscription and 1:1 descriptions apply, why loss and delay tolerance differ between the two fabrics, how to review failure scope separately, and why optical-link selection does not calculate fabric capacity by itself.
Data center deep-dive series
Short answer: the same rack layout can need different uplink designs
Two racks can use the same physical switch placement yet require different uplink decisions. The key question is not only how many ports leave the rack, but what traffic leaves it, how much of that traffic may occur at once, and which delay, loss, or contention conditions are acceptable.
Cisco’s described AI-cluster architecture separates two fabrics. The frontend fabric is the gateway to the GPU cluster: it handles north-south user access, API calls, logging, and ingestion of data from storage or data lakes into GPUs. Cisco states that this fabric can carry both RDMA and non-RDMA traffic.
The backend, also called the scale-out fabric, has a different role. It is the high-speed, low-latency network for GPU-to-GPU collective east-west communication. A useful rack-uplink review therefore begins by splitting one broad question—"Is there enough uplink capacity?"—into two: what shared capacity is appropriate for frontend traffic, and what contention can be tolerated on paths used for GPU synchronization?
What oversubscription means—and where Cisco applies it
For the comparisons in this article, oversubscription is a working term for a design in which lower-level connections share uplink capacity. The term does not, by itself, establish acceptable performance. The design still depends on traffic concurrency, workload sensitivity to delay and loss, and the consequences when demand exceeds the shared path.
In Cisco’s described frontend fabric, traffic is often bursty. Cisco says oversubscription is acceptable and commonly implemented in that fabric. However, its description does not provide a numerical oversubscription ratio. It should therefore not be read as a fixed ratio for every AI workload or every data center.
Cisco describes its backend fabric differently: as lossless and non-blocking with 1:1 subscription for synchronized GPU communication. In this context, 1:1 describes the subscription structure of that architecture. It does not, on its own, prove that every link, switch, or path failure is tolerated.
Why frontend and backend uplinks should not share one rule
Both fabrics may connect equipment beyond the rack, but they are described as having different tolerance for degraded network conditions. In Cisco’s discussion of extending fabrics across sites, the backend is ultra-sensitive to packet loss and delay. As distance increases, bottlenecks can stall GPU synchronization.
Cisco says a certain degree of loss and latency may be tolerable in an extended frontend, depending on the workload. That is not a claim that frontend traffic is always resilient to loss or delay. User requests, data ingestion, API traffic, and logging may have different requirements, and the cited description does not establish a single common tolerance for all of them.
For scale-across designs, Cisco advises keeping backend oversubscription lower to support GPU-to-GPU communication while allowing a higher ratio for the frontend. It also says that deep-buffer requirements depend on the deployed distance profile and oversubscription ratio; there is no generic buffer answer.
This is conditional guidance for Cisco’s architecture discussion, not a universal target or a prescribed buffer quantity.
Review failure scope separately from subscription
A subscription ratio examines how much normal-state traffic may share an upstream path. Failure scope asks a different question: if one link, rack switch, upstream switch, or path fails, which servers, GPUs, or service flows are affected?
Terms such as non-blocking and 1:1 subscription do not answer that failure-scope question. Cisco’s discussion distinguishes frontend and backend sensitivity to loss and delay, but it does not state the blast radius of a failed link or switch. A design review should therefore record performance assumptions and failure assumptions as separate items.
Use this question map to keep the review boundaries clear:
Traffic purpose
- Frontend question
- Which user, API, logging, storage, or data-lake ingestion flows use this fabric?
- Backend question
- Which GPU collective communication paths use this fabric?
- Do not confuse this with
- Traffic leaving a rack does not necessarily have the same performance requirement.
Oversubscription
- Frontend question
- Under what concurrent burst conditions is shared uplink capacity acceptable?
- Backend question
- What subscription structure is needed to avoid contention that disrupts synchronization?
- Do not confuse this with
- Whether oversubscription is acceptable depends on the fabric and workload conditions.
Loss and delay
- Frontend question
- What level is tolerable for each relevant workload?
- Backend question
- How would loss or delay affect GPU synchronization?
- Do not confuse this with
- Frontend tolerance, where present, is workload-dependent rather than unconditional.
Failure scope
- Frontend question
- Which access or data flows stop after a link, switch, or path failure?
- Backend question
- Which GPU communication group is affected by the same failure?
- Do not confuse this with
- 1:1 or non-blocking terminology does not independently demonstrate resilience.
Physical link
- Frontend question
- What bitrate, distance, and optical-loss conditions must this connection meet?
- Backend question
- What bitrate, distance, and optical-loss conditions must this connection meet?
- Do not confuse this with
- A viable optical link does not set a fabric-wide subscription ratio.
Keep optical-link selection separate from rack-capacity calculations
An optical uplink also has a physical-link design question. According to the Fiber Optic Association’s educational guidance, the required transceiver and cable-plant components depend on the named link’s digital bitrate, its length, and the bandwidth and optical-loss conditions of the fiber cable plant.
Those conditions help determine whether a particular optical connection is suitable. They do not determine aggregate rack-to-uplink capacity, the simultaneous use of multiple ports, or the fabric’s oversubscription ratio. A transceiver supporting a given bitrate is not, by itself, evidence that the rack or fabric has sufficient capacity for its traffic pattern.
Treat the physical-link check and the traffic-and-failure review as connected but distinct decisions. One establishes the conditions for an individual connection; the other establishes how the fabric is expected to behave under demand and disruption.
The Fiber Optic Association — The FOA Reference For Fiber Optics - Fiber Optic Data Links -
The practical implication: start with unacceptable contention and failures
For AI-cluster rack uplinks, the useful starting point is the role of each fabric rather than a port total. Cisco’s architecture describes a bursty mixed-traffic frontend where oversubscription may be used, and a GPU synchronization backend described as lossless, non-blocking, and 1:1 subscribed. These descriptions are useful design distinctions, not universal rules or fixed numerical ratios.
Before selecting an uplink ratio, identify the traffic expected to coincide, the loss and delay each fabric can tolerate, and the set of communications that would be interrupted by one failed component or path. Then assess optical bitrate, distance, and cable-plant conditions for each named link.
That sequence prevents a physically valid uplink or a nominally non-blocking fabric from being mistaken for proof of adequate workload performance or acceptable failure impact.
Sources
- The Fiber Optic Association — The FOA Reference For Fiber Optics - Fiber Optic Data Links -
- blogs.cisco.com — Scale-across: Why the future of distributed AI isn’t in one data center - Cisco Blogs Search
No comments:
Post a Comment