File Systems
Every user has access to three file systems through their CARC account: /home1, /project, and /scratch1. Users conducting cryo-EM research at USC can access /cryoem2.
Nearly 100 GB in a ZFS/NFS file system of storage space is provided for each user to store important code and configuration files in their home directories. Additionally, Principal Investigators (PI) receive a maximum of 10 TB to be distributed across their projects in the ZFS/BeeGFS parallel /project file system. A rapidly growing fleet of state-of-the art multicore compute nodes is available for data-intensive research jobs, backed by two high-throughput scratch file systems. A low-latency, high-bandwidth 200 gigabits-per-second (Gbps) InfiniBand network fabric facilitates intense research workloads on large data sets.
CARC clusters are shared resources. As a result, there are quotas on usage to help ensure fair access to all USC researchers as well as to maintain the performance of the file systems. For more information on storage quotas, allocations, and pricing, see the Project and Allocation Management pages.
0.0.1 Home file system
The /home1 file system has a total capacity of 136 TB, running NFS/ZFS on dedicated storage machines. It consists of personal directories for CARC users. Each home directory has a quota of 100 GB of disk space and 1.9 million files. It is intended for storing personal files, configuration files, and software. Production jobs of I/O-intensive jobs should not be run directly from the home directory.
Data in the home directory are backed up using snapshots. Snapshots are kept for two weeks.
0.0.2 Project file system
The /project file system has a total capacity of 8.4 PB and consists of directories for different research project groups. It offers high-performance, parallel I/O, running ZFS/BeeGFS on dedicated storage machines. The default quota for each project directory is 5 TB of disk space and 30 million files. Each PI can request up to 10 TB of storage across their project(s) at no cost. If additional storage space is required, PIs can request more in 5 TB increments. Additional storage is charged annually at a rate of $40/TB/year.
Each project member has access to their group’s project directory, where they can store data, scripts, and related files and install software. The project directory should be used for most of your CARC work, and it’s also where you can collaborate with your research project group. Users affiliated with multiple CARC projects will have access to multiple project directories so they can easily share their files with the appropriate groups.
See the Request a New Allocation user guide for instructions on requesting project storage.
0.0.3 Cryo-EM file system
Cryo-EM research has a dedicated /cryoem2 file system with 1.4 PB of space. This storage option functions similarly to the project file system and allocations are requested in the user portal in the same way. See the Request a New Allocation user guide for instructions. Cryo-EM storage allocations can be requested in 5 TB increments.
Cyro-EM storage space does not have a free tier like the project file system. The current rate for cryo-EM storage is $40/TB/year. Users can request a decrease in their project storage allocation to offset costs, if needed.
0.0.4 Scratch file system
The /scratch1 file system offers high-performance, parallel I/O, running ZFS/BeeGFS on dedicated storage machines. /scratch1 has a total capacity of 1.6 PB. Each CARC user gets a personal directory in /scratch1. The quota in the directory is 10 TB of disk space and 20 million files.
The scratch file system is intended for temporary and intermediate files, so it is not backed up.
A data purge is conducted on the /scratch1 file systems every 6 months—or whenever the /scratch1 file system’s capacities are greater than 80%—to ensure fair and efficient use for all CARC users.
0.0.5 Temporary files
For temporary files, each compute node has a local /tmp directory, implemented as a RAM-based file system (tmpfs). They are restricted to 1 GB of space that is shared among jobs running on the same node. Users can also use the scratch directories for temporary files, but read/write speeds may be slower because files are saved to disk.