HPC with Julia

Julia is an open-source programming language designed for high-performance scientific and numerical computing.

0.0.1 Using Julia on CARC systems

Begin by logging in. You can find instructions for this in the Getting Started with Discovery or Getting Started with Endeavour user guides.

You can use Julia in either interactive or batch modes. In either mode, first load a corresponding software module:

module purge
module load julia/1.10.2

To see all available versions of Julia, enter:

module spider julia

0.0.1.1 Installing a different version of Julia

If you require a different version of Julia that is not currently installed, please submit a help ticket and we will install it for you. Alternatively, you could install a version of Julia from official binaries in one of your directories or use the juliaup installer.

0.0.1.2 Installing Julia packages

See the section on installing packages below, with special notes for the MPI.jl and CUDA.jl packages.

0.0.1.3 Integrated development environments

VSCode, JupyterLab, and other integrated development environments (IDEs) can be used on compute nodes via our OnDemand service. To install Jupyter kernels, see our guide here. You can also use Pluto notebooks via the GUI Terminal app.

0.0.2 Running Julia in interactive mode

To run Julia interactively on a compute node, follow these two steps:

1. Reserve job resources on a node using Slurm’s salloc command
2. Once resources are allocated, enter julia to start a new Julia REPL session

[user@discovery1 ~]$ salloc --time=1:00:00 --ntasks=1 --cpus-per-task=8 --mem=16G --account=<project_id>
salloc: Pending job allocation 24737
salloc: job 24737 queued and waiting for resources
salloc: job 24737 has been allocated resources
salloc: Granted job allocation 24737
salloc: Waiting for resource configuration
salloc: Nodes d05-04 are ready for job

Change the resource requests (the --time=1:00:00 --ntasks=1 --cpus-per-task=8 --mem=16G --account=<project_id> part after your salloc command) as needed, such as the number of cores and memory required. Also substitute your project ID (<PI_username>_<id>); enter myaccount to view your available project IDs.

Once you are granted the resources and logged in to a compute node, enter julia:

[user@d05-04 ~]$ module purge
[user@d05-04 ~]$ module load julia/1.10.2
[user@d05-04 ~]$ julia
               _
   _       _ _(_)_     |  Documentation: <https://docs.julialang.org>
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.2 (2024-03-01)
 _/ |\__'_|_|_|\__'_|  |  Official <https://julialang.org/> release
|__/                   |

julia>

The shell prompt changes from user@discovery1 to user@<nodename> to indicate that you are now on a compute node (e.g., d05-04).

To run Julia scripts from within Julia, use the include() function. Alternatively, to run Julia scripts from the shell, use the julia command (e.g., julia script.jl).

To exit the node and relinquish the job resources, enter exit() in Julia and then enter exit in the shell. This will return you to the login node:

julia> exit()
[user@d05-04 ~]$ exit
exit
salloc: Relinquishing job allocation 24737
[user@discovery1 ~]$

0.0.3 Running Julia in batch mode

To submit jobs to the Slurm job scheduler, use Julia in batch mode:

1. Create a Julia script
2. Create a Slurm job script that runs the Julia script
3. Submit the job script to the job scheduler using sbatch

Your Julia script should consist of the sequence of Julia commands needed for your analysis. The julia command runs Julia scripts.

A Slurm job script is a special type of Bash shell script that the Slurm job scheduler recognizes as a job. For a job running Julia, a Slurm job script should look something like the following:

#!/bin/bash

#SBATCH --account=<project_id>
#SBATCH --partition=main
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=1:00:00

module purge
module load julia/1.10.2

julia --threads $SLURM_CPUS_PER_TASK script.jl

Each line is described below:

Adjust the resources requested based on your needs, but remember that fewer resources requested leads to less queue time for your job. Note that to fully utilize the resources, especially the number of cores, you may need to explicitly change your Julia code to do so (see the section on parallel programming below).

Develop and edit Julia scripts and job scripts to run on CARC clusters:

• on your local computer and then transfer the files to one of your directories on CARC file systems.
• with the Files app available on our OnDemand service.
• with one of the available text editor modules (nano, micro, vim, or emacs).

Save the job script as jl.job, for example, and then submit it to the job scheduler with Slurm’s sbatch command:

[user@discovery1 ~]$ sbatch jl.job
Submitted batch job 13587

To check the status of your job, enter myqueue. If there is no job status listed, then this means the job has completed.

The results of the job will be logged and, by default, saved to a file of the form slurm-<jobid>.out in the same directory where the job script is located. To view the contents of this file, enter less slurm-<jobid>.out, and then enter q to exit the viewer.

For more information on job status and running jobs, see the Running Jobs user guide.

0.0.4 Installing Julia packages

To install Julia packages, start an interactive Julia session and enter the ] key to toggle package mode:

(@v1.10.2) pkg>

By default, packages will be installed to ~/.julia/packages.

To install a registered package, use the add command together with the package name:

(@v1.10.2) pkg> add DataFrames

To install unregistered or development versions of packages, such as from GitHub or GitLab, use the URL path to the Git repository:

(@v1.10.2) pkg> add <https://github.com/JuliaData/DataFrames.jl>

To exit package mode, enter the Backspace key on an empty line. The shell prompt will toggle back to the standard Julia prompt julia>.

You can also install packages to other locations, such as for use in a shared group or project library. We recommend using the /project or /scratch1 file systems, especially for large parallel jobs. To change the Julia depot path, set the relevant environment variable in the shell before starting Julia:

export JULIA_DEPOT_PATH=/path/to/dir

This changes the Julia depot location to the specified directory, and then packages will be installed to and loaded from a /path/to/dir/packages directory. After exporting this variable, you can start Julia and install and load packages like normal. Note that this line also needs to be added to Slurm job scripts in order to load packages from that location. Alternatively, add it to your ~/.bashrc to set the depot location automatically every time you log in.

To clear this environment variable and return to the default depot location in your home directory, enter unset JULIA_DEPOT_PATH in the shell.

Also consider using Julia environments to create reproducible, project-specific package environments.

0.0.4.1 Installing MPI.jl

Configuring the MPI.jl package to use the MPI libraries optimized for the CARC HPC clusters requires special steps. First, load the relevant software modules:

module purge
module load julia/1.10.2
module load gcc/11.3.0
module load openmpi/4.1.4

Then, start an interactive Julia session, install the MPIPreferences package, specify to use a system binary, and exit:

(@v1.10.2) pkg> add MPIPreferences
julia> using MPIPreferences
julia> MPIPreferences.use_system_binary()
julia> exit()

This preference is saved in the current package environment (global or local), so it only needs to be completed once (per project).

Next, start another interactive Julia session and install the MPI package:

(@v1.10.2) pkg> add MPI

The MPI package will be installed and configured to use the openmpi/4.1.4 module.

For more information about running MPI jobs, see the Using MPI user guide.

0.0.4.2 Installing CUDA.jl

To install the CUDA.jl package for use with NVIDIA GPUs, first log on to a GPU node via salloc:

salloc --partition=gpu --time=1:00:00 --ntasks=1 --cpus-per-task=8 --gpus-per-task=1 --mem=16G --account=<project_id>

Then, start an interactive Julia session and install the CUDA package:

(@v1.10.2) pkg> add CUDA
julia> using CUDA
julia> CUDA.versioninfo()

The currently installed NVIDIA driver on the node will be detected and the latest compatible CUDA toolkit will be downloaded and installed as an artifact for use with the CUDA package.

0.0.5 Parallel programming with Julia

Julia uses only one thread by default, but it also supports both implicit and explicit parallel programming to enable full use of multi-core processors and compute nodes. This also includes the use of shared memory on a single node or distributed memory on multiple nodes. On CARC systems, 1 thread = 1 core = 1 logical CPU (requested with Slurm’s --cpus-per-task option).

Parallelizing your code to use multiple cores or nodes can reduce the execution time of your Julia jobs, but the speedup does not necessarily increase in a proportional manner. The speedup depends on the scale and types of computations that are involved. Furthermore, sometimes using a single core is optimal. There is a cost to setting up parallel computation (e.g., modifying code, communications overhead, etc.), and that cost may be greater than the achieved speedup, if any, of the parallelized version of the code. Some experimentation will be needed to optimize your code and resource requests (optimal number of cores and amount of memory). Also keep in mind that your project account will be charged CPU-minutes based on the cores reserved for a job, even if all those cores are not actually used during the job.

0.0.5.1 Implicit parallelism

Implicit parallelism is typically based on multi-threading, so that you do not need to explicitly call for parallel computation in your Julia code. To set the number of threads to use, start Julia with the --threads option (e.g., julia --threads 8) or set the environment variable JULIA_NUM_THREADS (e.g., export JULIA_NUM_THREADS=8) before starting Julia. In job scripts, you can also use the SLURM_CPUS_PER_TASK variable to set threads (e.g., julia --threads $SLURM_CPUS_PER_TASK). Multi-threaded Julia packages and functions will automatically detect and use the number of threads that are set when Julia is started.

0.0.5.2 Explicit parallelism

Explicit parallelism means explicitly calling for parallel computation in your Julia code, either in relatively simple ways or potentially in more complex ways depending on the tasks to be performed. Many Julia packages exist for explicit parallelism, designed for different types of tasks that can be parallelized.

The main Julia packages for explicit parallelism are summarized in the following table:

Please review the linked documentation above for examples and more information about how to use these packages and their functions.

If using BLAS functions with both explicit and implicit parallelism, it is typically best to restrict the number of BLAS threads with export OPENBLAS_NUM_THREADS=1.

For more information about high-performance computing with Julia, see our workshop materials for HPC with Julia, as well as the resources linked below.

0.0.6 Additional resources

If you have questions about or need help with Julia, please submit a help ticket and we will assist you.

• Julia language
• Julia documentation
• Julia cheat sheet
• Julia style guide
• Julia performance tips
• JuliaHub
• JuliaParallel
• JuliaGPU
• SciML
• Flux
• JuMP
• JuliaStats
• BioJulia
• EcoJulia
• JuliaHealth
• JuliaGeo
• JuliaAstro

Tutorials:

• Julia Learning
• JuliaAcademy

Web books:

• Think Julia: How to Think Like a Computer Scientist
• Julia Data Science