●     Cisco UCS® C885A Rack Servers with NVIDIA HGX™ H200 and Spectrum™-X Ethernet.

●     Cisco® Silicon One® NPU-based 8000 Series SONiC Switches.

●     Cisco Optics and cables.

●     Cisco provisioning, observability and security frameworks.

●     C: Number of CPUs in the node.

●     G: Number of GPUs in the node.

●     N: Number of network adapters (NICs), categorized into:

●     B: Average network bandwidth per GPU in gigabits per second (GbE).

●     East-West Compute Network

●     Converged North-South Storage and Management Network

●     Provides access to high performance storage from compute nodes

●     Provides host management related access to compute nodes from Management nodes

●     Interconnects with border leaf exit switches to forward traffic in and out of cluster

●     Allows interconnecting to additional customer infrastructure such as data lakes, and other nodes for support, monitoring, log collection, etc., that a cloud provider wishes to add.

●     The compute nodes are grouped into two SUs to create a half SU-group consisting of 64 nodes. Each half SU-group uses two 8122-64EHF-O front-end leaf switches. A total of eight half SU-groups (four SU-groups) are required to deploy 4K GPUs. Each front-end leaf switch 8122-64EHF-O uses 64 400GE downlinks to the compute nodes and 32 400GE uplinks (8 400GE to each spine switch) to provide greater than 12.5 Gbps target storage bandwidth per H200 GPU.

●     On the storage side, networking will be pre-provisioned to deliver at least the amount of bandwidth requirement as mentioned above. Five redundant leaf-switch pairs, each with 64 200GE downlink ports per leaf pair, will be used for a total of 10 8101-32FH-O storage leaf switches and 320 200GE downlink ports. For uplinks, per storage leaf, 16 400GE ports will be used (2 400GE links per spine) for a total of 160 400GE ports. The number of storage nodes to connect to the storage leaf switches will vary based on cloud partners throughput and capacity needs on top of the minimum requirements.

●     The management node network consists of two parts:

●     Each core-group will consist of a single 128 400GE port 8122-64EHF-O switch (for a total of four switches in four core-groups) to meet the scale of 8K GPUs.

●     128 compute nodes (1K GPUs) are grouped into an SU-group connected to four leaf switches. Two SU-groups (2K GPUs) connect to eight 8122-64EHF-O leaf switches, which in turn connect to four 8122-64EHF-O spine switches. Four such parallel planes each with two SU-groups, are required to deploy 8K GPUs. In this design, each leaf switch with 64 400GE downlink ports connects to each spine switch via 8 400GE ports, and will still allow 12.5 Gbps of target storage bandwidth to each GPU. Each of the four spine switches connect to their respective core-group switches via 16 400GE ports.

●     The storage network consists of 10 pairs of 8101-32FH-O leaf switches, each connected to four 8122-64EHF-O spine switches. Each redundant leaf-switch pair has 64 200GE downlink ports and 32 400GE uplink ports (8 to each spine switch) for a total of 20 leaf switches, 640 200GE downlink ports and 320 400GE uplink ports. Each spine switch connects to its corresponding core-group switch via 40 400GE ports. On an aggregate basis, this provides around 64 Tbps of storage bandwith, or 8 Gbps to each GPU.

●     The management node network consists of two parts:

●     The number of switches within a core-group are doubled to two for a total of eight core switches.

●     Eight parallel planes each with two SU-groups of compute nodes (2K GPUs) are used to deploy 16K GPUs. Spine switches in each of these parallel planes connect to their respective core-group switches via 16 400GE links (8 400GE per core switch).

●     The storage network consists of 20 pairs of 8101-32FH-O leaf switches, each connected to 8 8122-64EHF-O spine switches. Each redundant leaf-switch pair has 64 200GE downlink ports and 32 400GE uplink ports (4 to each spine switch) for a total of 40 leaf switches, 1280 200GE downlink ports and 640 400GE uplink ports. Each spine switch connects to its corresponding core-group switch via 40 400GE ports. On an aggregate basis, this provides around 128 Tbps of storage bandwith, or 8 Gbps to each GPU.

●     The management node network consists of two parts:

●     The number of switches within a core-group is doubled to 4 for a total of 16 core switches.

●     16 parallel planes each with two SU-groups of compute nodes (2K GPUs), are used to deploy 32K GPUs. Spine switches in each of these parallel planes connect to their respective core-group switches via 16 400GE links (4 400GE links per core switch).

●     The storage network consists of 40 pairs of 8101-32FH-O leaf switches, each connected to 16 8122-64EHF-O spine switches. Each redundant leaf-switch pair has 64 200GE downlink ports and 32 400GE uplink ports (2 to each spine switch) for a total of 80 leaf switches, 2560 200GE downlink ports, and 1280 400GE uplink ports. Each spine switch connects to its corresponding core-group switches via 40 400GE ports. On an aggregate basis, this provides around 256 Tbps of storage bandwith, or 8 Gbps to each GPU.

●     The management node network consists of two parts:

●     Provisioning the compute nodes either via Cisco Intersight or NVIDIA Base Command Manager (BCM) or additional provisioning tools/frameworks.

●     Setup Slurm and/or Kubernetes control nodes for orchestrating jobs on worker compute nodes

●     Additional infrastructure for observability, monitoring, and logs collections

●     Cisco Secure Firewall

●     Cisco Isovalent

●     Cisco Hypershield

●     Cisco AI Defense