NDR InfiniBand Network Upgrade

Posted on August 21, 2023 by Dani Cannella

0.0.0.1 Maintenance completion brings major upgrades to CARC systems.

The completion of our extended maintenance period from August 14 - August 21 brings major improvements to our network!

During maintenance, we upgraded from FDR InfiniBand to NDR InfiniBand. This change increases interconnect “communication” speeds from 56 Gbps to 200 Gpbs. The new speeds apply to newer compute nodes in the epyc-64, gpu, and largemem partitions.

We also retired all compute nodes in the oneweek partition and replaced them with:

  • 35 20-core 64 GB nodes
  • 10 24-core 192 GB nodes.

The GCC 11.3.0 software stack is now compatible with all CARC compute nodes.

Thank you for your patience while we performed these major upgrades to our systems. Stay tuned for further announcements on our improved systems and services. If you have any questions or concerns, please submit a help ticket.