Getting Started with Discovery

macOS users can connect to Discovery using the Terminal application that is natively installed. Linux users can similarly use the natively installed terminal application that comes with their distribution of Linux (e.g., Terminal on Ubuntu).

To connect, open a new terminal window and enter:

ssh <username>@discovery.usc.edu

Make sure to substitute your USC NetID as the username. This is the same username for your USC email account (e.g., ttrojan@usc.edu’s NetID is ttrojan). After entering the command, you will then be prompted to enter your USC NetID password. This is the same password for your USC email account.

Windows users may need to download and install a third-party SSH client to connect to Discovery. A popular client is PuTTY, which is available through the developer’s website. Another popular option is MobaXterm, which also provides file transfer capabilities. It is also available through the developer’s website. On Windows 10, there is a natively available Windows Terminal that has a built-in SSH client.

Whichever option you choose, you will log in using your USC NetID and password. Your NetID is the same username for your USC email account (e.g., ttrojan@usc.edu’s NetID is ttrojan). Your USC NetID password is the same password for your USC email account.

0.0.3.0.1 Windows Terminal

To connect using Windows Terminal, open a new terminal window and enter:

ssh <username>@discovery.usc.edu

Make sure to substitute your USC NetID as the username. You will then be prompted to enter your USC NetID password.

0.0.3.0.2 PuTTY

To connect using PuTTY, start in the Configuration window under the Session category and enter discovery.usc.edu as the hostname with port 22 and select Connection type: SSH. Then select Open to open a connection window, enter your USC NetID as the username, and finally enter your USC NetID password.

0.0.3.0.3 MobaXterm

There are two ways to connect using MobaXterm. If you prefer the command-line interface, select “Start local terminal” and follow the instructions above using the Windows Terminal. If you prefer a GUI interface, select the icon “Session” on the top left corner (or, Sessions > New Session from the menu at the very top of the window). When prompted to choose a session type, choose “SSH.” Under “Basic SSH Settings” enter discovery.usc.edu as “Remote Host” and enter your USC NetID as the username after checking “Specify Username”. Keep the port as 22. There are additional settings below that you may want to adjust as necessary, particularly the Bookmark settings if you want a shortcut. Click OK to open your SSH session. Enter your USC NetID password when prompted for a password to finish logging in.

If you are having issues with the MobaXterm SSH session disconnecting due to timeout/inactivity, click the “Settings” icon from the top menu. Then click to the SSH tab and under “SSH Settings” make sure to check “SSH keepalive” and then click OK. Restart MobaXterm and try again.

The /home1 file system consists of personal directories for CARC users. Each CARC user has a home directory with 100 GB of disk space intended for storing personal files, configuration files, and software. I/O-intensive jobs should not be run directly from your home directory.

We keep two weeks of snapshots for home directories. You can think of these snapshots as semi-backups, meaning that if you accidentally deleted some data we would be able to recover it within two weeks. If the file was created and deleted within a one-day period, then the snapshot cannot recover it. You should always keep extra backups of your important data and other files because of this.

When you log in to Discovery, you will always start in your home directory. Your home directory is located at:

/home1/<username>

Use the cd command to quickly change to your home directory.

The /project file system is a high-performing, parallel file system and consists of directories for different research project groups. Project directories are intended for sharing files among a research group and should be used for most of your CARC work. Each project member has access to their group’s project directory, where they can store data, scripts, and related files and install software. Users affiliated with multiple CARC projects will have access to multiple project directories.

The default quota for each project directory is 5 TB. Each PI can request up to 10 TB of storage across their project(s) at no cost in the CARC user portal. If more than 10 TB is needed, a PI can request additional storage space in 5 TB increments at a cost of $40/TB/year. For more information on storage quotas and pricing, see the Project and Allocation Management pages.

We keep two weeks of snapshots for project directories. You can think of these snapshots as semi-backups, meaning that if you accidentally deleted some data we would be able to recover it within two weeks. If the file was created and deleted within a one-day period, then the snapshot cannot recover it. You should always keep extra backups of your important data and other files because of this.

Project directories are located at:

/project/<PI_username>_<id>

where <PI_username> is the username of the project owner and <id> is a 2 or 3 digit project ID number (e.g., ttrojan_123).

You can also find the project ID and path on the project page in the user portal.

To create your own subdirectory within a project directory, enter a command like the following:

mkdir /project/<PI_username>_<id>/<username>

where <username> is your USC NetID (your email address without “@usc.edu”).

The /scratch1 file system is a high-performance, parallel file systems intended for temporary files and I/O operations. Each CARC user has a /scratch1 directory with 10 TB of disk space.

The files stored in both scratch file systems are not backed up in any way. Files on these file systems may be purged every so often (with advanced warning). If needed, files stored here should be periodically backed up to decrease the risk of data loss.

Your /scratch1 directory is located at:

/scratch1/<username>

Use the cds command to quickly change to your /scratch1 directory.

We recommend that you first perform some basic tests of your job interactively on a compute node before submitting it as a batch job to the Slurm job scheduler, ensuring that you will have quality results after the job completes. You can do this by requesting an interactive session with a compute node using the salloc command.

For example, to request 1 CPU and 2 GB of memory for one hour (the default allocation request), enter:

salloc --account=<project_id>

Make sure to use the correct account for your jobs. Without the --account option, your default account will be used. This is fine if you only have one project account.

After running the command, the job scheduler will add your interactive job to the job queue. Once your job starts, you can then test by loading the software environment and running scripts or programs. Once you are confident that your job will run correctly, you are ready to submit a batch job through Slurm.

After you have tested your job interactively and achieved the results you want, you can now submit your batch job to Slurm.

To submit a job, first create a Slurm job script using one of the previously mentioned text editors.

Your job script should look something like this:

#!/bin/bash

#SBATCH --account=<project_id>
#SBATCH --partition=main
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=1:00:00

module purge
module load julia/1.9.3

julia --threads $SLURM_CPUS_PER_TASK script.jl

The top few lines of the file (that begin with #SBATCH) specify the requested resources for your program and other options. Make sure to use the correct account in your job scripts. Without the --account option, your default account will be used. This is fine if you only have one project account.

The next set of lines loads the required software modules (module load ...). After that, the remaining lines are the commands that run your programs.

To submit a batch job, use the sbatch command. For example:

sbatch my.job

where the argument to the command is the job script’s file name (e.g., my.job).

Submitted jobs are processed remotely. The process is recorded and written to an output file in the same directory where you submitted the job from. By default, this output file is named slurm-<jobid>.out. This is a plain-text file, so you can view it using the less command:

less slurm-<jobid>.out

There are several commands you can use to monitor a job after it has been submitted.

The first thing you will want to check is if your job request was queued. Use the squeue command to view the status of your jobs:

squeue --me

Each job is assigned a unique job identifier. It is sufficient to use only the numeric portion of the job ID when referencing a job or submitting a support ticket.

In the example below, the job 3271 has been placed in the default “main” partition (PARTITION):

[ttrojan@discovery1 ~]$ squeue --me
  JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
   3271      main   my.job  ttrojan  R      35:58      1 d05-09

This job has been running for 35 minutes and 58 seconds (TIME). It has requested 4 CPUs and was allocated 1 node (NODES), which is d05-09 (NODELIST). The status of the job is “R” (running) (ST).

If you wish to cancel and remove your job from the queue, you can use the scancel command:

scancel <job_id>

Your job may remain in the queue for a short time, but its status will change to “CG” (completing) or “CA” (canceled).