Share via

Configuring RDMA/RoCE v2 Bandwidth & DCB for 26-Node Hyper-V 2025 Cluster via SCVMM

Ahmed Essam 290 Reputation points
2026-03-16T11:17:40.73+00:00

Hello,

I am implementing a 26-node Hyper-V cluster on Windows Server 2025 using Cisco UCS M7 blades (100Gbps VIC 15000). We are using SCVMM to manage a converged network with Switch Embedded Teaming (SET).

Current Configuration: We have successfully deployed the following MinimumBandwidthWeight values to all 26 hosts via SCVMM to manage our 100Gbps converged fabric:

Live Migration (RDMA/RoCE v2): 40

Backup (RDMA/RoCE v2): 40

Cluster Management: 10

Host Management: 10

Our Intersight QoS Policy is set to Platinum (CoS 5) with an MTU of 9000. On the host side, Get-NetAdapterRdma currently shows Operational: True, but PFC and ETS are False.

My Questions

  1. Software SMB Limits: I see documentation suggesting the use of Set-SMBBandwidthLimit -Category LiveMigration. Since we have 100Gbps links and have already defined 40% weights in the SET switch, is it recommended to apply an SMB bandwidth cap as well? Does this conflict with the SET weight algorithm?.
  2. DCB Feature Requirement: To flip the PFC/ETS status to True, do I need to manually install the Data-Center-Bridging feature and create New-NetQosTrafficClass policies on each host, or should the SCVMM weights and the Nexus switch configuration (CoS 5/DCBX) handle this automatically in Server 2025?. Thanks
Windows for business | Windows Server | Storage high availability | Virtualization and Hyper-V
0 comments
Answer accepted by question author
  1. Domic Vo 19,660 Reputation points Independent Advisor
    2026-03-16T13:59:15.91+00:00

    Hello Ahmed Essam,

    On the SMB bandwidth limits: the Set-SMBBandwidthLimit cmdlet applies caps at the SMB protocol layer, independent of the SET weights. In your case, since you already have MinimumBandwidthWeight configured in the SET switch (Live Migration 40, Backup 40, Cluster/Host Management 10 each), applying an SMB bandwidth cap would override the dynamic allocation logic and force a hard ceiling. That can be counterproductive on 100Gbps links because SET weights are designed to proportionally share bandwidth across workloads. Unless you have a strict compliance requirement to guarantee Live Migration never exceeds a certain throughput, it is best practice to rely on SET weights alone. Microsoft’s documentation confirms that SMB bandwidth limits are optional and primarily useful in mixed or oversubscribed environments. In a converged 100Gbps fabric with RDMA enabled, you should not combine both unless you want to enforce a hard cap.

    On the DCB feature requirement: the fact that Get-NetAdapterRdma shows Operational: True but PFC/ETS: False means RDMA is functional but Data Center Bridging is not active. Windows Server does not automatically enable PFC/ETS just because SCVMM weights or Nexus switch CoS/DCBX are configured. You must explicitly install the Data-Center-Bridging feature (Install-WindowsFeature Data-Center-Bridging) and then define QoS policies with New-NetQosTrafficClass and Enable-NetQosFlowControl. SCVMM can push QoS policies, but the host OS still requires the DCB feature to interpret and enforce them. In other words, the Nexus side handles CoS/DCBX advertisement, but the Windows host must have DCB enabled to negotiate and flip those flags to True. Without it, you’ll see RDMA working but no PFC/ETS enforcement, which can lead to congestion under load.

    So the recommended path is: keep SET weights as your primary bandwidth governance, avoid SMB caps unless you need hard limits, and install/configure DCB on each host to align with your Intersight QoS policy. That will ensure PFC/ETS negotiation succeeds and your RDMA traffic is lossless across the fabric.

    I hope you've found something useful here. If it helps you get more insight into the issue, it's appreciated to accept the answer. Should you have more questions, feel free to leave a message. Have a nice day!

    Domic Vo.

    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Q&A Assist
    2026-03-16T11:18:08.72+00:00
    AI generated content. This question contains an answer created with AI from Q&A Assist. Learn more

    SMB bandwidth caps and SET minimum bandwidth weights are independent mechanisms and do not conflict. MinimumBandwidthWeight on the Hyper‑V virtual switch partitions link capacity among traffic classes under congestion, while Set-SMBBandwidthLimit caps SMB categories (such as Live Migration) at a defined ceiling.

    For Hyper‑V failover clusters, guidance is to rely on Hyper‑V QoS minimum bandwidth in weight mode on the vSwitch rather than absolute bandwidth caps, because weight mode is more flexible and compatible with Live Migration and NIC Teaming/SET. The recommendation is to configure minimum bandwidth in weight mode on the vSwitch and assign weights to all virtual adapters, including Live Migration and storage/backup traffic, as already done. There is no requirement in the provided guidance to additionally cap SMB Live Migration with Set-SMBBandwidthLimit when weights are correctly configured on a high‑bandwidth fabric.

    Regarding DCB and RDMA/RoCE v2, the documentation distinguishes between:

    • Hyper‑V QoS Minimum Bandwidth (software, enforced by the QoS Packet Scheduler and vSwitch), and
    • Data Center Bridging (DCB, hardware‑enforced on DCB‑capable NICs and switches, used with RDMA).

    These two are not designed to work together for the same networking stack or NIC. The guidance explicitly states not to enable both Minimum Bandwidth and DCB for workloads that share the same networking stack or NIC. They can both be present on a server only when they apply to separate networking stacks or separate NICs.

    For RoCE‑based RDMA fabrics, DCB is required and must be configured consistently end‑to‑end (hosts and switches). The RDMA guidance for Azure Local and Storage Spaces Direct specifies that if DCB is implemented, PFC and ETS must be configured correctly on every network port, including switches, and that RoCE‑based implementations require specific PFC traffic classes (for system/heartbeat, RDMA/SMB Direct, and default traffic).

    In System Center VMM scenarios with Storage Spaces Direct, DCB configuration is not automatic from QoS weights alone. VMM provides an explicit DCB configuration workflow where priorities and bandwidth percentages for SMB‑Direct and Cluster Heartbeat are defined, and the RDMA‑capable adapters (or storage vNICs in a converged NIC scenario) are selected. VMM then pushes DCB parameters to all S2D nodes. The documentation also notes that DCB settings must be consistent across all hosts and fabric switches, and that out‑of‑band changes can cause non‑compliance, which VMM can remediate.

    From the provided material:

    • Hyper‑V QoS minimum bandwidth by weight is the preferred mechanism for partitioning bandwidth on converged fabrics with NIC Teaming/SET and Live Migration.
    • DCB (PFC/ETS) is required for RoCE RDMA and must be configured explicitly and consistently; it is not implied or automatically enabled by Hyper‑V QoS weights.
    • Minimum Bandwidth and DCB should not be enabled together on the same NIC/stack for the same workloads.

    Therefore, enabling PFC/ETS (DCB) for RoCE RDMA requires explicit DCB configuration on the hosts and switches. In VMM‑managed S2D scenarios, this is done via the DCB configuration workflow; the presence of SET weights alone does not turn PFC/ETS on.

    References:

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.