MetroCluster ISL requirements in shared networks
When sharing ISL traffic in a shared network, you must ensure that you have adequate capacity and size the ISLs appropriately. Low latency is critical for replication of data between the MetroCluster sites. Latency issues on these connections can impact client I/O.
You should review these sections to correctly calculate the required end-to-end capacity of the ISLs. Continuous nonvolatile cache and storage replication with low latency is critical for MetroCluster configurations. The latency in the back-end network impacts the latency and throughput seen by client IO.
Latency and packet loss limits in the ISLs
The following requirements must be met for round-trip traffic between the MetroCluster IP switches at site_A and site_B, with the MetroCluster configuration in steady state operation:
Round trip latency must be less than or equal to 7 ms.
The maximum distance is 700 km, so the distance between the sites is limited by the latency or the maximum distance, whichever is reached first.
As the distance between two MetroCluster sites increases, latency increases, usually in the range of 1 ms round-trip delay time per 100 km (62 miles). This latency also depends on the network service level agreement (SLA) in terms of the bandwidth of the ISL links, packet drop rate, and jitter on the network. Low bandwidth, high jitter, and random packet drops lead to different recovery mechanisms by the switches or the TCP engine on the controller modules for successful packet delivery. These recovery mechanisms can increase overall latency.
Any device that contributes to latency must be accounted for.
Packet loss must be less than or equal to 0.01%.
Packet loss includes physical loss or loss due to congestion or over-subscription.
Packet drops can cause retransmissions and a reduced congestion window.
The supported jitter value is 3 ms for round trip (or 1.5 ms for one way).
The network should allocate and maintain the SLA for the bandwidth required for MetroCluster traffic, accounting for microbursts and spikes in the traffic.
Low bandwidth can cause queuing delays and tail drops on switches. If you are using ONTAP 9.7 or later, the network intermediate between the two sites must provide a minimum bandwidth of 4.5 Gbps for the MetroCluster configuration.
MetroCluster traffic should not consume the complete bandwidth and have negative impact on non-MetroCluster traffic.
The shared network should have network monitoring configured to monitor the ISLs for utilization, errors (drops, link flaps, corruption, etc.) and failures.
Connection limits and trunking in the customer switches
The intermediate customer-provided switches must meet the following requirements:
The number of intermediate switches is not limited, and more than two switches between the MetroCluster IP switches is supported.
The MetroCluster IP switches should be located as close as possible to the intermediate switches providing the long-haul link. All of the ISL connections along the route must meet all of the requirements for MetroCluster ISL.
The ISLs in the customer network (the ISLs between the customer switches) must be configured in such way that sufficient bandwidth is provided and order of delivery is preserved.
This can be done with trunking a sufficient number of links and enforcing load balancing policies to preserve order.
Other network requirements
The intermediate, customer-provided switches must meet the following requirements:
The MetroCluster traffic uses fixed VLAN IDs that are set in the provided RCF files.
Layer 2 VLANs with IDs that match the MetroCluster VLAN IDs must span the shared network. AFA DM5000F and DM5000H systems require VLAN 10 and VLAN 20 unless they are changed during interface creation. Other systems are not restricted to specific VLAN IDs.
The MTU size must be set to 9216 on all devices in the end-to-end network.
No other traffic can be configured with a higher priority than class of service (COS) five.
ECN (explicit congestion notification) must be configured on all end-to-end paths.