Install the B200 NVSwitch and retimer cold plate module

Follow instructions in this section to install the B200 NVSwitch and retimer cold plate module. The procedure must be executed by a trained technician.

About this task

Attention

• Read Installation Guidelines and Safety inspection checklist to ensure that you work safely.
• Touch the static-protective package that contains the component to any unpainted metal surface on the server; then, remove it from the package and place it on a static-protective surface.
• Two people and one lifting device on site that can support up to 400 lb (181 kg) are required to perform this procedure. If you do not already have a lifting device available, Lenovo offers the Genie Lift GL-8 material lift that can be purchased at Data Center Solution Configurator. Make sure to include the Foot-release brake and the Load Platform when ordering the Genie Lift GL-8 material lift.
• A torque screwdriver is available for request if you do not have one at hand.

Note

Make sure you have the required tools listed below available to properly replace the component:

• Torx T15 head screwdriver
• Torx T15 200mm extension bit
• Phillips #1 head screwdriver
• Phillips #2 head screwdriver
• Alcohol cleaning pad
• 2 x B200 PCM
• 2 x B200 SXM6 PAD-1
• 2 x B200 SXM6 PAD-2
• B200 GPU F&R Shipping bkt Kit
• B200 GPU Service Kit
• B200 Retimer NVSwitch Service Kit
• B200 Retimer NVSwitch Shipping Kit
• B200 NVSWITCH PCM
• B200 NVSWITCH PAD-1
• B200 NVSWITCH PAD-2

B200 (GPU & Retimer NVSwitch) (service & shipping bkt) Kit are reusable and mandatory when servicing GPUs and GPU cold plate modules. It is recommended to keep them at the facility where the server operates for future replacement needs.

Important

Putty pad/phase change material (PCM) replacement guidelines

• Before replacing the putty pad/PCM, gently clean the hardware surface with an alcohol cleaning pad.
• Hold the putty pad/PCM carefully to avoid deformation. Make sure no screw hole or opening is blocked by the putty pad/PCM.
• Do not use expired putty pad/PCM. Check the expiry date on putty pad/PCM package. If the putty pads/PCM are expired, acquire new ones to properly replace them.

The following illustration shows the B200 GPU numbering and corresponding slot numbering in XCC.

Figure 1. B200 GPU numbering

Physical GPU socket	Slot numbering in XCC	Logical number in nvidia-smi
GPU 1	Slot 21	4
GPU 2	Slot 24	7
GPU 3	Slot 22	5
GPU 4	Slot 23	6
GPU 5	Slot 17	0
GPU 6	Slot 20	3
GPU 7	Slot 18	1
GPU 8	Slot 19	2

The following illustration shows the components for NVSwitch and retimer cold plate module.

Figure 2. NVSwitch and retimer cold plate module components identification

Table 1. NVSwitch and retimer cold plate module components

1 Retimer cold plate torque label	2 Leakage sensor module
3 Handle	4 Hose tie
5 NVSwitch cold plate	6 TIM breaker screw
7 NVSwitch slot number label	8 NVSwitch cold plate torque label
9 Retimer cold plate

Procedure

1. Make sure the GPU complex is installed in the chassis.
2. Replace the Phase Change Material (PCM) on the NVSwitch cold plates.
   Attention

   • PCM cannot be reused. PCM must be replaced with new ones every time the water loop is removed.

   1. Ensure the shipping bracket is attached to the NVSwitch and retimer cold plate module. Flip over the module and place it on a surface with the cold plate facing upward.
   2. Install the jig onto the NVSwitch cold plate.
   3. Remove the liner from one side of the pad. Align the PCM with the jig and place it onto the cold plate. Then, apply finger pressure across the entire surface area of the PCM to remove any trapped air, and allow 1-2 minutes dwell time until it is firmly attached. Carefully remove the remaining top liner.
      Figure 3. PCM application
      
      
   4. Repeat to replace the PCM on the other NVSwitch cold plate. Remove the jig after application.
3. Replace the Phase Change Material (PCM) and putty pads on the retimer.
   Attention

   • PCM cannot be reused. PCM must be replaced with new ones every time the water loop is removed.

   1. Bend the jig along the bending line 1.
      Figure 4. Bending the jig
      
      
   2. Align the jig to the components and carefully apply it onto the GPU baseboard.
      Figure 5. Jig application on retimer
      
      
   3. Follow the application instructions and use a plastic tweezer to attach putty pad to each component. Then, apply light finger pressure across the entire surface area of the pad to ensure adhesion. Carefully remove the remaining top liner.
      Note

      • Apply the putty pads from B200 NVSWITCH PAD-1 to the eight locations marked with number 1 (pink color)
      • Apply the putty pads from B200 NVSWITCH PAD-2 to the ten locations marked with number 2
      • Apply the putty pads from B200 NVSWITCH PAD-2 to the two locations marked with number 3

      Figure 6. Putty pads application on retimer
      
      
4. Replace the putty pads on the NVSwitches.
   Attention

   • PCM cannot be reused. PCM must be replaced with new ones every time the water loop is removed.

   1. Remove the liner from one side of the pad. Align the putty pads with the marking as illustrated and attach it to the NVSwitch, then apply light finger pressure across the entire surface area of the pad to ensure adhesion. Carefully remove the remaining top liner.
   2. Repeat to replace all the putty pads.
      Figure 7. Putty pads application on NVSwitch
      
      
5. Install the jigs for NVSwitch and retimer module installation.
   1. Place the jig in the slot in between the retimer and front GPUs; then, fasten the two screws by 360 degrees alternately (PH2, 0.5 newton-meters, 4.3 inch-pounds) to install the jig on the GPU baseboard.
      Figure 8. Installing the jig
      
      
   2. Install the other two jigs. Place the jig in the slot in between the GPUs; then, fasten the two screws by 360 degrees alternately (PH2, 0.5 newton-meters, 4.3 inch-pounds) to install the jig on the GPUs. Repeat to install the other one.
      Figure 9. Installing the jigs
      
      
6. Install the NVSwitch and retimer cold plate module.
   1. Before installing the cold plate module, ensure the TIM breaker screw is in its initial position.
      Note

      • (Except for the brand-new cold plate module) Ensure the TIM breaker screw is loosened to its initial position before tightening the cold plate screws.

      • Loosen the TIM breaker screw to return it to its initial position.
        
        
      • Close the lid. If the lid cannot be closed, the TIM breaker screw needs to be further loosened.
        
        
   2. Lift the NVSwitch and retimer cold plate module by the handles; then, align the cold plates with the NVSwitch and retimer on the GPU baseboard, and gently place it onto the NVSwitch and retimer.
   3. Adjust the cold plates until they are securely seated in the NVSwitch and retimer sockets.
      Figure 10. Installing the NVSwitch and retimer cold plate module
      
      
7. With a torque screwdriver set to the proper torque, follow the screw sequence specified on the cold plate label, and tighten the eighteen Torx T15 screws by 360 degrees for one round to position the cold plates before removing the shipping brackets.
   1. Set the torque screwdriver to 1.0±0.1 inch-pounds, 0.112±0.0112 newton-meters.
   2. Tighten the NVSwitch cold plate screws by 360 degrees for one round following the screw sequence: → → → → → → →
      Figure 11. Positioning the NVSwitch cold plates
      
      

      Note

      • Make sure to follow screw sequence to prevent cold plate tilting.
   3. Tighten the retimer cold plate screws by 360 degrees for one round following the screw sequence: → → → →
      Figure 12. Positioning the retimer cold plates
      
      

      Note

      • Make sure to follow screw sequence to prevent cold plate tilting.
8. Remove the shipping brackets.
   1. Loosen the fourteen captive screws that secure the shipping brackets to the NVSwitch and retimer cold plate module.
   2. Lift the shipping bracket out of the chassis.
      Figure 13. Removing the shipping brackets
      
      
9. Follow the screw sequence specified on the cold plate label, and repeat to fully fasten the eighteen Torx T15 screws with a torque screwdriver set to the proper torque.
   1. First set the torque screwdriver to 1.0±0.1 inch-pounds, 0.112±0.0112 newton-meters to fasten the screws for a few rounds. Then set the torque screwdriver to 5.22±0.2 inch-pounds, 0.59±0.024 newton-meters to fully fasten the screws.
   2. Fasten the NVSwitch cold plate screws by 360 degrees following the screw sequence: → → → → → → →
      Figure 14. Installing the NVSwitch cold plates
      
      

      Note

      • Make sure to follow screw sequence to prevent cold plate tilting.
   3. First set the torque screwdriver to 1.0±0.1 inch-pounds, 0.112±0.0112 newton-meters to fasten the screws for a few rounds. Then set the torque screwdriver to 5.22±0.2 inch-pounds, 0.59±0.024 newton-meters to fully fasten the screws.
   4. Fasten the retimer cold plate screws by 360 degrees following the screw sequence: → → → →
      Figure 15. Installing the retimer cold plates
      
      

      Note

      • Make sure to follow screw sequence to prevent cold plate tilting.
10. Remove the jigs after installing NVSwitch and retimer cold plate module.
    Note

    • Make sure to follow screw sequence to prevent cold plate tilting.
    1. Loosen the two screws by 360 degrees alternately to remove the first jig from the GPU baseboard.
       Figure 16. Removing the jig
       
       
    2. Loosen the two screws by 360 degrees alternately to remove the jig from the GPUs. Repeat to remove the other jig.
       Note

       Be careful not to drag and damage the leakage module sensor cable.

       Figure 17. Removing the jig
       
       
11. The following illustration shows the hose holder location.
    Figure 18. Hose holder location
    
    
12. Place the NVSwitch and retimer cold plate module hoses on (1) hose holder A.
    Important

    • Check the guiding labels on the hoses and hose holders to ensure they match.

    Figure 19. Placing the hoses
    
    

    1 Hose holder A

13. Align the hose holder B/C with the two screw holes on hose holder A; then, fasten the two captive screws (PH1, 2 x M3, 0.5 newton-meters, 4.3 inch-pounds) to secure hose holder B/C on top of hose holder A. Repeat to install hose holder B/C on the other side.
    Figure 20. Installing hose holder B/C
    
    
14. Install the front B200 GPU cold plate module. See Install the front B200 GPU cold plate module.
15. Install the rear B200 GPU cold plate module. See Install the rear B200 GPU cold plate module.
16. Secure the hoses and cables to the hose guides. See Fan control board cable routing and Leakage sensor module cable routing.
    Note

    When securing cables on the hose holder, ensure not to route the cables on top of the hoses.

    Important

    • Check the guiding labels on the hoses and hose holders before installation.

    • Ensure to keep the front GPU cold plate module hose to the right of the rear GPU cold plate module hose.

      
      
      

    • Ensure not to cover the joints with the hoses.

      
      
      

    Figure 21. Securing the hoses and cables on the hose guides
    
    
17. Install the rear fan cage support bracket.
    1. Align the rear fan cage support bracket with the corresponding screw holes; then, install the rear fan cage support bracket on top of hose holder B/C as illustrated.
    2. Fasten the four M3 screws (PH2, 4 x M3, 0.5 newton-meters, 4.3 inch-pounds) to secure the rear fan cage support bracket to the fan cage.
    3. Fasten the eight M3 screws (PH2, 8 x M3, 0.5 newton-meters, 4.3 inch-pounds) to secure the rear fan cage support bracket to the chassis.
       Figure 22. Installing the rear fan cage support bracket