Skip to main content

Install the network board (trained technician only)

Use this information to install the network board.

If replacing the OSFP module and network board together, refer to OSFP module and network board replacement (trained technician only).

About this task

Important
Gap pad/putty pad replacement guidelines
  • To identify the gap pad/putty pad location and orientation, see:

  • Before replacing the gap pad/putty pad, gently clean the interface plate or the hardware surface with an alcohol cleaning pad.

  • Hold the gap pad/putty pad carefully to avoid deformation. Make sure no screw hole or opening is blocked by the gap pad/putty pad material.

  • Do not use expired putty pad. Check the expiry date on putty pad package. If the putty pads are expired, acquire new ones to properly replace them.

Required tools

Make sure you have the required tools listed below in hand to properly replace the component.

  • SD665-N V3 Water Loop Service Kit (The water loop carrier in the Service Kit is reusable, it is recommended to keep it at the facility where the server operates for future replacement needs.)

  • SD665-N V3 Water Loop Putty Pad Kit

  • SD665-N V3 SXM5 PCM Fixture

  • SXM5 PCM Kit (for removing PCM from GPU)

    Putty pad cannot be reused. Whenever the water loop is removed, putty pads must be replaced with new ones before reinstalling the water loop.

  • Screws and screwdrivers

    Prepare the following screwdrivers to ensure you can install and remove corresponding screws properly.
    Screw TypeScrewdriver Type
    Hex screwdriver6 mm hex head screwdriver
    Torx T10 screwTorx T10 head screwdriver
    Torx T15 screwTorx T15 head screwdriver
    Phillips #1 screwPhillips #1 head screwdriver
    Phillips #2 screwPhillips #2 head screwdriver
Attention
  • Read Installation Guidelines and Safety inspection checklist to ensure that you work safely.

  • Turn off the corresponding DWC tray that you are going to perform the task on.

  • Disconnect all external cables from the enclosure.

  • Use extra force to disconnect QSFP cables if they are connected to the solution.

  • To avoid damaging the water loop, always use the water loop carrier when removing, installing or folding the water loop.

  • A torque screwdriver is available for request if you do not have one at hand.

Prepare the following screwdrivers to ensure you can install and remove corresponding screws properly.
Screw TypeScrewdriver Type
Hex screwdriver6 mm hex head screwdriver
Torx T10 screwTorx T10 head screwdriver
Torx T15 screwTorx T15 head screwdriver
Phillips #1 screwPhillips #1 head screwdriver
Phillips #2 screwPhillips #2 head screwdriver
Firmware and driver download: You might need to update the firmware or driver after replacing a component.

Procedure

  1. Align the network board to the guide holes on the carrier board; then, place the network board onto the carrier board.
    Figure 1. Installing the network board
    Installing the network board
  2. Follow the sequence shown in the illustration below to fasten the network board screws (x4 Torx T15 screws) to the carrier board.
    Note
    For reference, the torque required for the screws to be fully tightened/removed is 0.9 newton-meters, 7.96 inch-pounds.
    Figure 2. Network board screws installation
    Network board screws installation

GPU node water loop gap pad/putty pad replacement. Make sure to follow Gap pad/putty pad replacement guidelines.

  1. Replace the Phase Change Material (PCM) and putty pads on the GPU node water loop with new ones.
    1. Install the PCM jig on the GPU cold plate.
    2. Attach the PCM to the square opening of the jig.
    3. Repeat to replace the PCM of all four GPU cold plates.
      Attention
      • PCM cannot be reused. PCM must be replaced with new ones every time the water loop is removed.

      • After PCM is replaced, there is an expected short duration of throttling before the GPU returns to normal operation. This is due to the PCM requiring a break-in period after being replaced.

      Figure 3. Water loop GPU cold plate PCM replacement (GPU node)
      Water loop GPU cold plate PCM replacement
    4. Replace the putty pads on the GPU node water loop.
      Attention
      Putty pad cannot be reused. Whenever the water loop is removed, putty pads must be replaced with new ones before reinstalling the water loop.
      Figure 4. Water loop putty pads replacement (GPU node)
      Putty pads on the GPU node water loop putty pads replacement
    5. Replace the putty pads (x5) on the GPU. Make sure to align the putty pads to the GPU VR (1) and the markings on GPU. Repeat to replace all putty pads on the four GPUs.
      1 GPU VR (Cover the GPU VR with putty pad)
      Attention
      Putty pad cannot be reused. Whenever the water loop is removed, putty pads must be replaced with new ones before reinstalling the water loop.
      Figure 5. GPU putty pads replacement
      GPU putty pads replacement
  2. Unfold the water loop and place it onto the GPU node.
    Figure 6. Unfolding the water loop to GPU node
    Unfolding the water loop to GPU node
  3. Loosen water loop carrier screws (x20 Phillips #2 screws).
    Figure 7. Water loop screws and quick connect screws installation (GPU node)
    Water loop screws and quick connect screws installation (GPU node)
  4. Remove the water loop carrier from the GPU node.
    Figure 8. Water loop carrier removal (GPU node)
    Water loop carrier removal (GPU node)
  5. Install GPU cold plate screws (x16 T10 screws).
    1. Install GPU cold plate screws following the GPU cold plates installation sequence: GPU 2 > GPU 4 > GPU 1 > GPU 3
      Figure 9. GPU numbering
      GPU numbering
    2. Set the torque screwdriver to 1.5 +/- 0.5 lb-In (0.1 +/- 0.06 N-m); then, fasten the GPU cold plate screws.
    3. Set the torque screwdriver to 3.5 +/- 0.5 lb-In (0.4 +/- 0.06 N-m); then, fasten the GPU cold plate screws until all screws are fully tightened.
      Note
      Make sure to follow the screw sequence on the GPU cold plate label:
      Figure 10. GPU cold plate screw installation
      GPU cold plate screw installation
  6. Follow the screw installation sequence specified on the network board label, and install network cold plate screws (x8 Torx T10 screws) with a torque screwdriver set to the proper torque.
    Note
    For reference, the torque required for the screws to be fully tightened/removed is 5.0+/- 0.5 lbf-in, 0.55+/- 0.05 N-M.
    Figure 11. Network card screw installation
    Network card screw installation
  7. Install the quick connect screws (x4 Torx T10) with a torque screwdriver set to the proper torque.
    Note
    For reference, the torque required for the screws to be fully tightened/removed is 5.0+/- 0.5 lbf-in, 0.55+/- 0.05 N-M.
    Figure 12. Quick connect screw installation (GPU node)
    Quick connect screw installation (GPU node)
  8. Install water loop screws and quick connect screws (x13 Torx T10 screws) with a torque screwdriver set to the proper torque.
    Note
    For reference, the torque required for the screws to be fully tightened/removed is 5.0+/- 0.5 lbf-in, 0.55+/- 0.05 N-M.
    Figure 13. Water loop Torx T10 screws installation (GPU node)
    Water loop Torx T10 screws installation (GPU node)
  9. Install the Hex screw (x1) and the PH1 screws (x3).
    Note
    For reference, the torque required for the screws to be fully tightened/removed is 5.0+/- 0.5 lbf-in, 0.55+/- 0.05 N-M.
    Figure 14. Water loop Hex and PH1 screws installation (GPU node)
    Water loop Hex and PH1 screws installation (GPU node)
  10. Install the cable tie to the GPU board.
    Figure 15. Installing the cable tie
    Installing the cable tie
  11. Connect the carrier board power cable.
    Figure 16. Connecting carrier board power cable
    Connecting carrier board power cable
    CableFrom (carrier board)To (GPU node power distribution board)
    1 Carrier board power cablePower and side band connectorPower connector
  12. Connect the OSFP module cables to the network board on the GPU node.
    CAUTION
    • Hold the cable connector at 45 degree angle when inserting it into the port.

    • Have extremely precaution when reseating the connectors that have a 45 degree label on them as they are fragile and will get damage if not installed in the correct 45 degree angle.



    Figure 17. Connecting the OSFP module cables to the network board
    Connecting the OSFP module cables to the network board
    CableFrom

    OSFP module connectors

    To

    Network board connectors

    1 Network cable (long)Network connector (near GPU node)Port 0
    2 Power cablePower connector P1OOB port 0
    3 Power cablePower connector P2OOB port 1 (on network board) and power connector (on carrier board)
    4 Network cable (short)Network connector (near Compute node)Port 1
  13. Press the network cable connector to properly connect the cable plug to the connector. Make sure the latch is fully engaged to the connector.
    Figure 18. Pressing network cable plug

After you finish
  1. Install the MCIO cables. Follow the guidance and routing information in Internal cable routing.

  2. Install the bus bar. See Install the bus bar.

  3. Install the cross braces. See Install the cross braces.

  4. Install the tray cover. See Install the tray cover.

  5. Install the tray into the enclosure. See Install a DWC tray in the enclosure.

  6. Connect all required external cables to the solution.
    Note
    Use extra force to connect QSFP cables to the solution.
  7. Check the power LED on each node to make sure it changes from fast blink to slow blink to indicate all nodes are ready to be powered on.

Demo video

Watch the procedure on YouTube