Using Apptainer

Last updated January 12, 2025

Apptainer (formerly Singularity) is an open-source application for creating and running software containers, designed primarily for high-performance computing on shared Linux-based computing clusters like CARC systems.

Apptainer containers provide a custom user space and enable portable, reproducible, stable, and secure software environments on Linux systems. An Apptainer container bundles a primary application and all of its dependencies into a single image file, which can also include data, scripts, and other files if desired. In addition, Apptainer containers have direct access to the Linux kernel on the host system (e.g., Discovery or Endeavour compute nodes), so there is no substantial performance penalty when using a container compared to using natively installed software on the host system.

With Apptainer, you can:

  • Install anything you want (based on any Linux operating system)
  • Ease installation issues by using pre-built container images
  • Ensure the same software stack is used among a research group
  • Use the same software stack across Linux systems (e.g., any HPC center or cloud computing service)

0.0.1 Using Apptainer on CARC systems

Begin by logging in. You can find instructions for this in the Getting Started with Discovery or Getting Started with Endeavour user guides.

You can use Apptainer in either interactive or batch modes. In either mode, first load a corresponding software module:

module purge
module load apptainer

To see the help pages, enter apptainer help.

0.0.2 Getting Apptainer container images

A container image is a single executable file that defines the software environment for a container and also runs the container. A single container image can be used to run multiple instances of the same container simultaneously for different jobs.

To get a container image for use on CARC systems, you can either pull (i.e., download) pre-built container images into one of your directories or build a custom container image from a definition file.

When pulling images, Apptainer will create a ~/.apptainer/cache directory by default to store cache files, which can quickly use a lot of storage space. We recommend changing this location to a directory with more storage space (e.g., one of your scratch directories) by setting the environment variable APPTAINER_CACHEDIR. For example:

export APPTAINER_CACHEDIR=/scratch1/<username>/apptainer

Add this line to your ~/.bashrc to automatically set this variable every time you log in.

Enter apptainer cache clean to clear the cache and free up storage space.

0.0.2.1 Pulling pre-built images

You can find pre-built container images at container registries, such as Docker Hub. You may also find them as file downloads from software repositories or websites — you can use a wget <url> command to download these.

If you have identified a pre-built container image from a registry that includes all the software you need, then you can download this image into one of your directories using the apptainer pull command.

For Docker Hub, use the docker:// registry prefix. Most Docker container images will convert to the image format used by Apptainer. For example, to get an image containing base R from the Rocker Project, enter:

apptainer pull docker://rocker/r-ver

This will pull an image that contains the latest version of R into the current directory and name it r-base_latest.sif.

To get a specific version of an image, instead of the latest, you can append a tag ID to the image name. For example, to get an image containing R version 4.4.2, enter:

apptainer pull docker://rocker/r-ver:4.4.2

Explore the registry page for images to obtain available tags.

Note that Apptainer uses temporary space when pulling and converting Docker images. If you have set the TMPDIR environment variable to one of your directories and you get errors when pulling Docker images, enter unset TMPDIR to unset the variable and then pull again.

For larger images, you may need to log on to a compute node requesting all available memory via salloc --mem=0 and use export TMPDIR=/dev/shm before pulling.

Once an image is downloaded, it is ready to be used on CARC systems.

0.0.2.2 Building images from a definition file

You can build a custom container image by using a definition file that instructs the build process

0.0.2.2.1 1. Create a definition file

Next, create a definition file. The following is an example definition file that uses a base image with R already installed and then installs additional R packages:

Bootstrap: docker
From: rocker/r-ver:4.4.2

%post

# Install and update OS packages
apt-get -y update
apt-get -y upgrade
apt-get -y clean

# Install R packages
Rscript -e 'install.packages("skimr")'

%test

R --version

%help

Rocker-based image for R 4.4.2.

Save your definition file as a text file (e.g., r.def). For more details on creating definition files, see the official Apptainer documentation.

0.0.2.2.2 2. Build a container image using the definition file
apptainer build r.sif r.def

It will then begin to build and display a build log. The build will either succeed or fail. Sometimes successful builds may still contain errors, so it is a good idea to review the build log each time. If the build fails, consult the build log and modify your definition file as needed to correct any errors.

To see the definition file that was used to build an image, enter:

apptainer inspect --deffile <container_image>

0.0.3 Running Apptainer container images

You can run Apptainer containers using one of the following commands:

  • apptainer shell — for an interactive shell within a container
  • apptainer exec — for executing commands within a container
  • apptainer run — for running a pre-defined runscript within a container

To start an interactive shell within a container, enter:

apptainer shell <container_image>
Apptainer>

Notice that your shell prompt changes to Apptainer> to indicate that you are now working within the container. Enter exit to exit the container.

To non-interactively execute commands within a container, enter:

apptainer exec <container_image> <commands to execute>

This will run the given commands using the software stack within the container. For example, to run an R script using the version of R installed within a container, enter:

apptainer exec r.sif Rscript script.R

To run a pre-defined runscript within a container, enter:

apptainer run <container_image>

This will execute the defined sequence of commands in the runscript. There may be additional arguments that you can pass to the runscript.

To see the runscript for a container, enter:

apptainer inspect --runscript <container_image>

There are a number of options that can be used with each of these commands. Enter apptainer help exec, for example, to see the options available for a given command.

Use the --cleanenv (or shorter -e) option to establish a clean shell environment within a container. For example:

apptainer exec --cleanenv r.sif Rscript script.R

By default, most environment variables in your shell will be passed into the container, which can sometimes cause problems with using software installed within the container. However, note that using the --cleanenv option will result in Slurm environment variables not being included within the container. For more information on environment variables, see the official documentation.

There is a separate root file system within containers to store installed software and files, but containers can still access and interact with file systems on the host system. Apptainer will automatically mount the following directories to a container:

  • /home1/<username>
  • /tmp
  • /var/tmp
  • /proc
  • /sys
  • /dev

You will automatically have access to these directories as needed. You may want to exclude your home directory from binding to the container because of software packages installed there (e.g., for Python or R containers). If so, add the --no-home option to the apptainer command you are using.

To mount additional directories, such as your current working directory or project or scratch directories, use the --bind option. For example:

apptainer exec --bind $PWD,/scratch1/<username> r.sif Rscript script.R

Instead of the --bind option, you can also set the environment variable APPTAINER_BIND to automatically include any directory paths that you want mounted in your containers. For example:

export APPTAINER_BIND=$PWD,/scratch1/<username>

Add this line to your ~/.bashrc to automatically set this variable every time you log in. For more information on bind mounting, see the official documentation.

0.0.4 Running Slurm jobs with Apptainer

To use Apptainer in interactive Slurm jobs, first request resources with Slurm’s salloc command. Once the resources are granted, you will be logged in to a compute node. Then you can run the apptainer commands interactively.

To use Apptainer in batch Slurm jobs, load the apptainer module and include the apptainer commands to be run in your job scripts. For example:

#!/bin/bash

#SBATCH --account=<project_id>
#SBATCH --partition=main
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=1:00:00

module purge
module load apptainer

apptainer exec --bind $PWD r.sif Rscript script.R

The container will automatically have access to the requested resources.

0.0.4.1 Running GPU jobs with Apptainer

To run GPU jobs with Apptainer, you can use GPU-enabled container images with the CUDA toolkit installed. For example, you can convert official NVIDIA Docker container images to Apptainer format. Make sure that the versions of the image you pull (e.g., the version of the CUDA library within the container) are compatible with the currently installed NVIDIA drivers on the GPU nodes on CARC systems (use the nvidia-smi command on a GPU node to see the current driver version). For example, if the driver supports up to CUDA version 12.6, make sure the container image does not have a CUDA version greater than 12.6.

To run a GPU-enabled container on a GPU node, use the --nv option to allow the container to access the NVIDIA driver on the node:

apptainer exec --cleanenv --nv pytorch.sif python script.py

For more information on using GPUs with Apptainer, see the official documentation page for GPU support.

0.0.4.2 Running MPI jobs with Apptainer

It is possible to run MPI jobs with Apptainer, and there are different approaches to do so. For more information, see the official documentation page for Apptainer and MPI applications.

0.0.5 Additional resources

If you have questions about or need help with Apptainer, please submit a help ticket and we will assist you.