Discovery Resource Overview

CARC's general-use HPC cluster Discovery has over 21,000 cores across 600 compute nodes available for researchers to use.

Note: Discovery is a shared resource, so there are limits in place on size and duration of jobs. This ensures that everyone has a chance to run jobs. For details on the limits, see Running Jobs.

For general CARC system specifications, see our High Performance Computing page.

Partitions and compute nodes

There are a few Slurm partitions available on Discovery, each with a separate job queue. These are general-use partitions available to all researchers. The table below describes the intended purpose for each partition:

PartitionPurpose
mainSerial and small-to-medium parallel jobs (single node or multiple nodes)
epyc-64Medium-to-large parallel jobs (single node or multiple nodes)
gpuJobs requiring GPU nodes
oneweekLong-running jobs (up to 7 days)
largememJobs requiring larger amounts of memory (up to 1 TB)
debugShort-running jobs for debugging purposes

Each partition has a different mix of compute nodes. The table below describes the available nodes by partition. Each node typically has two sockets with one multi-core processor each and an equal number of cores per processor. In the table below, the CPUs/node column refers to logical CPUs such that 1 logical CPU = 1 core = 1 thread.

PartitionCPU ModelCPUs/nodeGPU modelGPUs/nodeMemory (GB)/nodeNodes
mainxeon-411624--9439
mainxeon-411624--19229
mainxeon-2640v420--6416
mainxeon-2640v420k4026445
mainxeon-2640v316--6432
epyc-64epyc-754264--25632
epyc-64epyc-751364--256139
gpuxeon-613032v100219129
gpuxeon-2640v420p100212838
gpuepyc-751364a100 (80 GB)225612
gpuepyc-751364a100 (40 GB)225612
gpuepyc-731332a40225617
gpuepyc-728232a40225612
oneweekxeon-411624--19210
oneweekxeon-2640v420--6435
largememepyc-751364--10244
debugxeon-411624--1922
debugxeon-2640v420p10021281
debugepyc-731332a4022561

This table was last updated on February, 26, 2024.

Note: Use the nodeinfo and gpuinfo commands for similar real-time information.

There are a few commands you can use for more detailed node information. For CPUs, the lscpu command will provide information about CPUs. For nodes with GPUs, the nvidia-smi command and its various options will provide information about GPUs. Alternatively, after module load nvhpc, use the nvaccelinfo command to view information about GPUs. After module load gcc/11.3.0 hwloc, use the lstopo command to view a node's topology.

CPU microarchitectures and instruction set extensions

Different CPU models also offer different CPU instruction set extensions. Compiled programs can use these extensions to boost performance. The following is a summary table:

CPU modelMicroarchitecturePartitionsAVXAVX2AVX-512
xeon-2650v2ivybridgeoneweek
xeon-2640v3haswellmain, debug
xeon-2640v4broadwellmain, gpu, debug
xeon-4116skylake_avx512main
xeon-6130skylake_avx512gpu
epyc-7542zen2epyc-64
epyc-7513zen3epyc-64, gpu, largemem
epyc-7282zen2gpu
epyc-7313zen3gpu

Use the lscpu command while logged in to a compute node to list all available CPU flags.

GPU specifications

The following is a summary table for GPU specifications:

GPU ModelPartitionsArchitectureMemoryMemory BandwidthBase Clock SpeedCUDA CoresTensor CoresSingle Precision Performance (FP32)Double Precision Performance (FP64)
A100gpuampere80 GB1.9 TB/s1065 MHz691243219.5 TFLOPS9.7 TFLOPS
A100gpuampere40 GB1.6 TB/s765 MHz691243219.5 TFLOPS9.7 TFLOPS
A40gpuampere48 GB696 GB/s1305 MHz1075233637.4 TFLOPS584.6 GFLOPS
V100gpuvolta32 GB900 GB/s1230 MHz512064014 TFLOPS7 TFLOPS
P100gpu, debugpascal16 GB732 GB/s1189 MHz3584n/a9.3 TFLOPS4.7 TFLOPS
K40main, debugkepler12 GB288 GB/s745 MHz2880n/a4.29 TFLOPS1.43 TFLOPS
Back to top