Software Module System

On both the Discovery cluster and the Endeavour condo cluster, users can find and load software using the Lmod module system. Lmod dynamically changes your shell environment using module files to ensure the software applications and libraries you use are compatible. Module files are configuration files, written as Lua scripts, that instruct how to make an application or library available during your shell session. Typically, a module file contains instructions to initialize or modify environment variables, such as PATH.

Using a module system like Lmod is helpful because applications and libraries compiled with one compiler are not necessarily compatible with applications and libraries compiled with a different compiler, and your shell environment must be changed to accommodate these incompatibilities. With Lmod, resetting your environment is done dynamically when you load new modules. Loading a module will make available only compatible software for you to use. In this way, modules are organized in a hierarchy based on compilers.

Software Management on CARC Systems video

0.0.1 Finding software

When you log in, we automatically load a module named usc for you, which is actually a collection of modules. You can enter the module list command to view them:

[user@discovery1 ~]$ module list

Currently Loaded Modules:
  1) gcc/11.3.0   2) openblas/0.3.21   3) openmpi/4.1.4   4) pmix/3.2.3   5) usc/11.3.0

This is our default software stack and the recommended one for most users, based on the GCC 11.3.0 compiler.

The naming convention for modules is <software_name>/<version> (e.g., gcc/11.3.0).

If you do not want these modules to be automatically loaded when you log in, add module purge to your ~/.bashrc.

0.0.1.1 Using module avail

To see what modules you can load into your environment, enter the command module avail. With gcc/11.3.0 loaded, this will print a large number of available modules:

[user@discovery1 ~]$ module avail

------ /spack/2206/apps/lmod/linux-centos7-x86_64/openmpi/4.1.4-4w23jca/gcc/11.3.0 -------
   fftw/3.3.10    ior/3.3.0         netcdf-combined/4.8.1
   hdf5/1.12.2    meme/5.5.0        netcdf-fortran/4.5.4
   hpcg/3.1       netcdf-c/4.8.1    parallel-netcdf/1.12.2

----- /spack/2206/apps/lmod/linux-centos7-x86_64/openblas/0.3.21-afcoykn/gcc/11.3.0 ------
   plink/1.9-beta7 (D)

----------------- /spack/2206/apps/lmod/linux-centos7-x86_64/gcc/11.3.0 ------------------
   alsa-lib/1.2.3.2                           libxrandr/1.5.0
   amdblis/3.1                                libxrender/0.9.10
   ant/1.10.7                                 libxscrnsaver/1.2.3
   aom/v1.0.0-errata1                         libxshmfence/1.3
   argtable/2-13                              libxslt/1.1.33
   aria2/1.36.0                               libxt/1.1.5
...

To unload all your loaded modules, enter the command module purge. Then module list will return No modules loaded. If you enter module avail again, then you will see only the core modules. These are primarily compilers but can also include applications like MATLAB that are pre-built and have no dependencies:

[user@discovery1 ~]$ module avail

----------------------- /spack/apps/lmod/linux-centos7-x86_64/Core -----------------------
   aocc/3.1.0                   intel-oneapi/2021.3        matlab/2022a        (D)
   cctools/7.2.0                intel/18.0.4               micro/2.0.11
   cellranger-arc/2.0.0         intel/19.0.4        (D)    nnn/4.7
   cellranger-atac/2.1.0        jdk/17.0.5                 nvhpc/22.11
   comsol/5.6                   julia/1.8.3                nvidia-hpc-sdk/21.7
   conda/4.12.0                 julia/1.8.4                pgi-nvhpc/20.7
   conda/23.3.1          (D)    julia/1.8.5         (D)    plink2/2.00a3.7LM
   gcc/4.9.4                    launcher/carc              qchem/6.0.1
   gcc/8.3.0                    llvm/12.0.0-dev            sas/9.4
   gcc/9.2.0                    llvm/14.0.2         (D)    spaceranger/2.0.1
   gcc/11.3.0            (D)    mathematica/12.3           usc-amd/9.2.0
   gurobi/9.5.2                 matlab/2019a               usc/8.3.0
   gurobi/10.0.0         (D)    matlab/2020b               usc/11.3.0          (D)

  Where:
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

To explore the software stacks available, load one of these core modules and then see which new ones are unlocked. For example, to see all applications built with a certain compiler, load that compiler module and then enter module avail.

0.0.1.2 Using module spider

If you know the name of a software package, use the module spider command to find out if it is available and how to load it.

For example, to search for Python modules:

[user@discovery1 ~]$ module spider python

------------------------------------------------------------------------
  python:
------------------------------------------------------------------------
     Versions:
        python/2.7.16
        python/3.6.8
        python/3.7.6
        python/3.9.2
        python/3.9.6
     Other possible modules matches:
        intel-oneapi-python  py-python-dateutil

------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*python.*'

------------------------------------------------------------------------
  For detailed information about a specific "python" package (including
  how to load the modules) use the module's full name. Note that names
  that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider python/3.9.6
------------------------------------------------------------------------

This shows that there are multiple versions of Python available. For more specific information, add the version to your command as given in the example:

[user@discovery1 ~]$ module spider python/3.7.6

------------------------------------------------------------------------
  python: python/3.7.6
------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below
    before the "python/3.7.6" module is available to load.

      gcc/4.9.4
      gcc/8.3.0
      gcc/8.3.0  intel/18.0.4
      gcc/8.3.0  intel/19.0.4
      gcc/9.2.0

    Help:
      The Python programming language.

This indicates, for example, that the gcc/8.3.0 module is required in order to load python/3.7.6. Alternatively, if you want to use the gcc/9.2.0 compiler, then you need to load this module first in order to load python/3.7.6.

0.0.1.3 Using module keyword

If you do not know the exact name of a software package, use the module keyword command instead to search for modules.

For example, to search with the keyword sam:

[user@discovery1 ~]$ module keyword sam
------------------------------------------------------------------------

The following modules match your search criteria: "sam"
------------------------------------------------------------------------

  bamutil: bamutil/1.0.13

  cufflinks: cufflinks/2.2.1

  jags: jags/4.3.0

  libbsd: libbsd/0.9.1, libbsd/0.10.0

  pcre: pcre/8.42, pcre/8.43

  pcre2: pcre2/10.31

  perl-text-soundex: perl-text-soundex/3.05

  picard: picard/2.20.8

  samtools: samtools/1.10, samtools/18.0.4

------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package, specify the version if there is more than one:

   $ module spider Foo/11.1

------------------------------------------------------------------------

0.0.2 Loading and unloading software

Typically, loading modules is as simple as entering module load <software_name>. The <software_name> must be visible when you run module avail.

By entering:

module load <software_name>

Lmod will set your environment such that the software specified in <software_name> will be placed in your PATH, and then you can run the commands associated with <software_name>.

If there are multiple versions of <software_name>, you can specify a version:

module load <software_name>/<version>

For example, to load gcc version 8.3.0, enter:

module load gcc/8.3.0

If no version is specified:

module load gcc

then the default version will be loaded. The default version is indicated with a (D) next to it after running module avail.

To unload a specific module, enter:

module unload <software_name>

To see the modules that you currently have loaded, enter:

module list

To see all modules that are compatible with the currently loaded modules and available to be loaded, enter:

module avail

To unload all currently loaded modules and reset your environment, enter:

module purge

To reload the default usc module collection, enter:

module load usc

Enter module help to view more information about the available module commands.

0.0.3 Environment management

One of the best features of Lmod is that it tracks software dependencies automatically. Importantly, there are two safety features when using the module system:

1. Only one version of a module can be loaded at once
2. Only one compiler, BLAS/LAPACK, and MPI stack can be loaded at once

This ensures that the loaded modules are compatible with one another.

For example, let’s say you want to use the jellyfish package compiled with gcc/8.3.0. You would load it with:

[user@discovery1 ~]$ module load gcc/8.3.0 jellyfish

If for some reason you need to switch to the intel compiler set, you can use the module swap command to swap out the gcc compiler:

[user@discovery1 ~]$ module swap gcc intel

Due to MODULEPATH changes, the following have been reloaded:
  1) jellyfish/2.3.0

Lmod automatically changes the jellyfish module to one that was compiled with intel.

The module system can also automatically replace or deactivate modules to ensure the packages that are loaded are compatible with each other. For example, switching from gcc/8.3.0 to gcc/9.2.0:

[user@discovery1 ~]$ module load gcc/9.2.0

Inactive Modules:
  1) openblas/0.3.8

Due to MODULEPATH changes, the following have been reloaded:
  1) openmpi/4.0.2     2) pmix/3.1.3

The following have been reloaded with a version change:
  1) gcc/8.3.0 => gcc/9.2.0

0.0.4 Module settings

Loading the desired module will make some changes to your shell environment. Some common settings in module files include:

Every module is different; some will have extra environment variables, while others will have fewer.

To see what a module changes, use the module show command:

module show <software_name>

This will print the contents of the module file. For example:

[user@discovery1 ~]$ module show gcc
-----------------------------------------------------------------------------------
    /spack/apps/lmod/linux-centos7-x86_64/Core/gcc/11.3.0.lua:
-----------------------------------------------------------------------------------
whatis("Name: gcc")
whatis("Version: 11.3.0")
whatis("Category: compiler")
whatis("Keywords: compiler, gcc, g++, c++, c, fortran")
whatis("Description: GNU Compiler Collection including gcc, g++, and gfortran.")
whatis("URL: https://gcc.gnu.org/")
family("compiler")
setenv("GCC_ROOT","/spack/apps/gcc/11.3.0")
prepend_path("LD_LIBRARY_PATH","/spack/apps/gcc/11.3.0/lib64")
prepend_path("PATH","/spack/apps/gcc/11.3.0/bin")
prepend_path("MANPATH","/spack/apps/gcc/11.3.0/share/man")
prepend_path("CMAKE_PREFIX_PATH","/spack/apps/gcc/11.3.0")
prepend_path("MODULEPATH","/spack/2206/apps/lmod/linux-centos7-x86_64/gcc/11.3.0")

In this case, loading the GCC module will modify your PATH environment variable by prepending it with the path to the GCC binaries. Additionally, it creates a new variable named GCC_ROOT that you could then use if needed:

[user@discovery1 ~]$ echo $GCC_ROOT
/spack/apps/gcc/11.3.0

Every module sets a <SOFTWARE_NAME>_ROOT variable, which is a useful shortcut pointing to the root directory of the installation.

0.0.5 Saving module collections

Another useful feature of Lmod is saving collections of modules for easier loading and unloading.

With a set of desired modules loaded, use module save to save them as your default collection:

[user@discovery1 ~]$ module purge
[user@discovery1 ~]$ module load gcc/11.3.0 openblas/0.3.20 r/4.2.3
[user@discovery1 ~]$ module save
Saved current collection of modules to: "default"

Then the next time you log in use module restore to load your default collection:

[user@discovery1 ~]$ module restore
Restoring modules from user's default

You can also save multiple collections by specifying a collection name:

[user@discovery1 ~]$ module save mymodules
Saved current collection of modules to: "mymodules"

Then use the collection name when restoring:

[user@discovery1 ~]$ module restore mymodules
Restoring modules from user's mymodules

0.0.6 Additional resources

If you have questions about or need help with software modules, please submit a help ticket and we will assist you.

• Lmod
• User Guide for Lmod
• Advanced User Guide for Personal Modulefiles