Skip to main content

Recovering from a multi-controller or storage failure

If the controller failure extends to all controller modules on one side of a DR group in a four or eight-node MetroCluster configuration, or storage has been replaced, you must replace the equipment and reassign ownership of drives to recover from the disaster.

Before you begin

About this task

The examples in this procedure show four-node configurations. If you have an eight-node configuration (two DR groups), you must take into account any failures and perform the required recovery task on the additional controller modules.

This procedure uses the following workflow:



  1. Replacing hardware at the disaster site
    If hardware components have to be replaced, you must replace them using their individual hardware replacement and installation guides.
  2. Determining the system IDs of the old controller modules
    After you have replaced all hardware at the disaster site, you must determine the system IDs of the replaced controller modules. You need the old system IDs when you reassign disks to the new controller modules.
  3. Isolating replacement drives from the surviving site (MetroCluster IP configurations)
    You must isolate any replacement drives by taking down the MetroCluster iSCSI initiator connections from the surviving nodes.
  4. Clearing the configuration on a controller module that was previously used
    Before using a controller module as a replacement in the MetroCluster configuration, you must clear the preexisting configuration if it was previously used and not received directly from Lenovo.
  5. Netbooting the new controller modules
    If the new controller modules have a different version of ONTAP from the version on the surviving controller modules, you must netboot the new controller modules.
  6. Determining the system IDs of the replacement controller modules
    After you have replaced all hardware at the disaster site, you must determine the system ID of the newly installed storage controller module or modules.
  7. Verifying the ha-config state of components
    In a MetroCluster configuration, the ha-config state of the controller module and chassis components must be set to mcc so they boot up properly.
  8. Preparing the disaster site for switchback
    You must take steps to prepare for the switchback operation. The steps you take depend on your configuration.
  9. Reestablishing object stores for FabricPool configurations (MetroCluster IP configurations)
    If one of the object stores in a FabricPool mirror was co-located with the MetroCluster disaster site and was destroyed, you must reestablish the object store and the FabricPool mirror.
  10. Verifying licenses on the replaced nodes
    You must install new licenses for the replacement nodes if the impaired nodes were using ONTAP features that require a standard (node-locked) license. For features with standard licenses, each node in the cluster should have its own key for the feature.
  11. Performing a switchback
    After you heal the MetroCluster configuration, you can perform the MetroCluster switchback operation. The MetroCluster switchback operation returns the configuration to its normal operating state, with the sync-source storage virtual machines (SVMs) on the disaster site active and serving data from the local disk pools.
  12. Verifying a successful switchback
    After performing the switchback, you want to confirm that all aggregates and storage virtual machines (SVMs) are switched back and online.
  13. Mirroring the root aggregates of the replacement nodes (MetroCluster IP configurations)
    If disks were replaced, you must mirror the root aggregates of the new nodes on the disaster site.
  14. Reconfiguring the ONTAP Mediator service (MetroCluster IP configurations)
    If you have a MetroCluster IP configuration that was configured with the ONTAP Mediator service, you must remove and reconfigure the association with the mediator.
  15. Verifying the health of the MetroCluster configuration
    You should check the health of the MetroCluster configuration to verify proper operation.