Data Storage on ReD Parallel Cluster

Last Update: 2025-02-28

Home Directory

Your initial working directory upon login is your home directory (aka homedir). This location is for your use in storing everyday data for configuration files, scripts, documentation, etc. Importanly this is also where your .bashrc reside, which you can update your environment variables. The homedir is on a shared EFS volume /shared and is also where software and containers are served out of. You have a ___GB quota in your homedir.

Syntax

Home directories paths look like /shared/home/XXXX where XXXX is your username. For example:

[syockel_test@ip-10-3-2-176 ~]$ pwd
/shared/home/syockel_test

TIP: You can also reach your home directory using the Unix shortcut ~, as in: cd ~

Permisions

Permissions set to read/write ofor individual users only.

[syockel_test@ip-10-3-2-176 home]$ ls -ld syockel_test/
drwx------. 5 syockel_test Domain Users 6144 Jan 15 21:38 syockel_test/

Where the "d" represents a directory. The next 3 triads represent the user, group, other permission sets. So in this case rwx represent read-write-execute for the user, while "---" is unset for group, and the last "---" is unset for others. For more on permissions see this nice tutorial.

Proper Usage

Your homedir has good performance for most simple tasks. However, I/O intensive or large numbers of jobs should not be processed in home directories. Widespread computation against home directories would result in poor performance for all users. For these types of tasks, your Lab Share /data/labs is best suited.# Lab Share The main working directory for your research data. This location is for researchers to set up and organize data within the lab. We strongly advise to create top level project folders that are representative of your DUAs.

The main working directory for your research data. This location is for researchers to set up and organize data within the lab. We strongly advise to create top level project folders that are representative of your DUAs.

File Tiers

Historically, in high-performance computing, only filesystems such as "scratch" or "temp" were performant, and your lab share was on a seperate NFS volume. Within the ReD Environment, we have simplified storage using performant storage for your Lab Share, so there is no need for moving data back and forth. We leverage FSx for Lustre as an file cache in front of S3. Files are first written to a SSD layer, before migrating to S3. To keep data costs manageable, our S3 layer has a policy engine to move between tiers. Files older than 30 days are migrated to S3 Infrequent. Retrieval time will not be affected. Files older than 90 days are migrated to Glacier. Retrieval times will be slowed. If this becomes problematic please send in a ticket.

Syntax

Lab Share paths look like /data/labs/XXXX_lab where XXXX is your typically your PIs last name, Center, or project group. For example, Univ RCD has a test project "urcdtest":

[syockel_test@ip-10-3-2-176 urcdtest_lab]$ pwd
/data/labs/urcdtest_lab

Permisions

Permissions are set to read/write for lab groups by default. If more granular permissions are needed, please submit a request ticket.

[syockel_test@ip-10-3-2-176 labs]$ ls -ld urcdtest_lab/
drwxrws---+ 5 root urc-urcdtest-lab 33280 Jan  8 22:13 urcdtest_lab/

Where the "d" represents a directory. The next 3 triads represent the user, group, other permission sets. So in this case rwx represent read-write-execute for the user, while rws is set for group, and the last "---" is unset for others. The (s)pecial bit on the group is commonly noted as SGID, and provides the utility of:

If set on a file, it allows the file to be executed as the group that owns the file (similar to SUID)
If set on a directory, any files created in the directory will have their group ownership set to that of the directory owner

Proper Usage

Your Lab Share is capable of handling I/O intensive or large numbers of jobs. Since Lustre is a parallel filesystem, you can leverage MPI I/O and other RDMA functions for increased performance.