Data Management
The Center for Advanced Research Computing (CARC) provides high-speed data transfer nodes and a variety of useful tools to achieve secure and efficient data transfer depending on whether the storage location is a personal computer or an external site (e.g., cloud storage). Different tools offer solutions for data sensitivity requirements and varying levels of user familiarity.
The three main methods of data transfer that CARC supports are command-line tools, graphical tools, and the Globus service. Our Research Data Management user guides provide more in-depth details for each method.
Due to security risks, please be mindful of the type of information being transferred. Where possible, omit all information that may be considered confidential. For examples of confidential information that requires additional consideration, visit the ITS Sensitive and Confidential Information page.
0.0.1 Data transfer nodes
CARC has two dedicated, high-speed, 100 Gbps data transfer nodes at hpc-transfer1.usc.edu and hpc-transfer2.usc.edu. These nodes are especially useful for larger transfers. The Discovery and Endeavour login nodes have a 40 Gbps connection speed and are adequate for most transfers. To log in to a transfer node, enter a command like the following:
ssh <username>@hpc-transfer1.usc.edu
Transfer speeds are determined by a number of factors, such as the network speed at your location, router and firewall settings, etc. If you experience slower than expected transfers, try to troubleshoot these issues first. If you continue to experience slow transfer speeds, submit a help ticket.
0.0.2 Command-line tools
We offer a variety of command-line interface (CLI) tools to transfer data to and from CARC storage systems, depending on the specific use case.
CLI tools available include:
sftprsyncrclone
See the guide for Transferring Files Using the Command Line.
0.0.3 Graphical tools
CARC gives users the option to use applications that provide a graphical user interface (GUI). These tools can be used to transfer data between a personal computer and a storage solution that allows SFTP connections, including CARC storage systems. They offer drag-and-drop capability, but transfer speeds may be slower compared to using a command-line tool.
GUI tools available include:
- Cyberduck
- FileZilla
- WinSCP
- CARC Open OnDemand
See the guide for Transferring Files Using a Graphical User Interface.
0.0.4 Globus service
Globus is a data management and transfer service that gives researchers unified access to their data across systems through a web-based GUI. It can be used for data transfers from a personal computer or another HPC center to CARC storage systems. Relative to other tools, it is useful for large transfers and will provide the best transfer speeds. A CLI for Globus can also be used, if desired.
See the guide for Transferring Files Using Globus.
0.0.5 Use cases
Below are four example scenarios that provide some insight into which data transfer method you might use for a given situation:
| Start Location | End Location | Example Scenario | Transfer Method |
|---|---|---|---|
| Personal Computer | CARC file system for small-to-medium transfers | When transferring files from a personal computer to your CARC project folder that takes a moderate amount of time. | GUI, OnDemand, CLI |
| Personal Computer | CARC file system for large or secure transfers | When transferring files from a personal computer to your CARC project directory that takes a large amount of time or needs to be encrypted. | Globus |
| Amazon Web Services (AWS) | Any CARC file system | When transferring files from an AWS server to your CARC project directory. | CLI |
| Other HPC centers | Any CARC file system | When transferring files from another university or research institution to your CARC project directory. | Globus, CLI |