Research Data Overview
The Center for Advanced Research Computing (CARC) provides a state-of-the-art data management system for our users. This overview provides introductory information to our data management systems and workflows. All research projects should have a data management plan to maximize research productivity on CARC systems.
Currently, CARC systems do not support the use or storage of sensitive data. If your research work includes sensitive data, including but not limited to HIPAA-, FERPA-, or CUI-regulated data, contact us at carc-support@usc.edu before using our systems.
0.0.1 File systems
All CARC account holders are assigned three directories on three file systems where they can store data and run programs: /home1, /project, and /scratch1. These file systems are global—you can access them from any Discovery, Endeavour, or transfer node.
For detailed information on the available file systems, see the Storage File Systems user guide.
0.0.2 Managing and sharing data
Data management involves organizing, sharing, backing up, and archiving files. On CARC systems, this can be achieved using various command-line tools.
For more information, see the Managing Data user guide.
0.0.3 Dedicated data transfer nodes
CARC has two dedicated, high-speed, 100 Gbps data transfer nodes at hpc-transfer1.usc.edu and hpc-transfer2.usc.edu. These nodes are especially useful for larger transfers. If needed, you can log in to them using, for example, ssh <username>@hpc-transfer1.usc.edu. The Discovery and Endeavour login nodes have a 40 Gbps connection speed and are adequate for most transfers.
0.0.4 Transferring data
Secure and efficient data transfer to and from CARC systems can be achieved with a variety of useful tools, the choice of which depends on whether the storage location is a personal computer or an external site (e.g., cloud storage). The choice also depends heavily on the requirement of data sensitivity as well as the familiarity of the user. There are three methods of data transfer that we recommend for use with CARC systems: command-line tools, graphical tools, and the Globus service.
0.0.4.1 Which method should I use?
Below are four example scenarios that provide some insight into which data transfer method you might use for a given situation:
| System 1 | System 2 | Example Scenarios | Method |
|---|---|---|---|
| Personal computer | CARC file system for small-medium transfers | When transferring files from a personal computer to your CARC project folder that takes a moderate amount of time | GUI, CLI |
| Personal computer | CARC file system for large or secure transfers | When transferring files from a personal computer to your CARC project directory that takes a large amount of time or needs to be encrypted | Globus |
| Amazon Web Services (AWS) | Any CARC file system | When transferring files from an AWS server to your CARC project directory | CLI |
| Other HPC center | Any CARC file system | When transferring files from another university or research institution to your CARC project directory | Globus |
For more information, see the Transferring Data user guides.