Storage on Research and Engineering Studio

This document describes the type of storage you have access to during your Research and Engineering Studio Sessions.

Windows Sessions:

The storage you can use in your Linux sessions is of two types:

1. Local Storage:

This is the “C:” drive on the system that is on the server.

Please note that this includes your User folder. Its size is the size of the storage that you selected when launching your desktop session.*

We recommend limiting the size to what you need, as this storage is more expensive than the persistent lab storage.

Features:

• Ephemeral. This storage will disappear when you terminate the system (see details on sessions lifecycle on main doc page for Research and Engineering Studio).
• Not shared across sessions. If you have more than one session active you will not be able to see and access from session B any data stored on local storage on session A.
• High performance – particularly for small files. While we do not necessarily recommend to use this as location to point the main IO for your large scale calculations, it can certainly be a good solution for calculations in particular those involving small files.

2. Remote Lab Share:

Additional Storage (Lab folder and additional software repositories) can be mounted as remote network drives to your Windows system.

• Software repositories. This share contains additional software resources, like Stata packages, R and python extra repositories that can be used to add packages for those languages to your environment. (please see details on the document page about scientific software)

This share can be mounted using the following procedures:

• Open the Windows Command Prompt

• Map the share to a drive of your choice executing the following command in the command prompt - in this example we are using drive P: but you can use any currently unused drive letter.

  net use P: \\10.7.34.49\res-urc-software-pgks # note misspelling of pkgs

• Lab/Project Share: This share constitutes the main storage repository for the lab. It provides persistent data storage and a location to point the IO of your calculations to. We are aware the storage performance is not ideal for IO bound calculations, and we are working at defining alternate solutions. The specific share for your lab can be mounted with the same procedure as described in the section about the software package share. The exact share address to use for each lab and project is provided in the following table:

Note:

• Feel free to use any drive letter G: or higher that is currently not already in use.
• Append /persistent:yes to the above command to have the connection restored at next login.

Linux Session:

The storage you can use in your Linux sessions is of three types :

1. Linux Home Folder:

This points at /home/$USER

Features: - It is persistent across sessions. This is the right place to store your configuration files, scripts or programs and potentially some small volume of personal data. - Not suitable for high performance IO. Your software should not point at this folder when doing reads and writes as it will impact performance. - Not suitable for storing large amount of data.

2. Local Storage:

This is the local root disk for the system and its size is the size of the storage that you selected when launching your desktop session. We recommend limiting the size to what you actually need, as this storage is more expensive than the persistent lab storage.

Features:

• Ephemeral. This storage will disappear when you terminate the system (see details on sessions lifecycle on main doc page for Research and Engineering Studio).
• Not shared across sessions. If you have more than one session active you will not be able to see and access from session B any data stored on local storage on session A.
• High performance – particularly for small files. While we do not necessarily recommend to use this as location to point the main IO for your large scale calculations, it can certainly be a good solution for calculations in particular those involving small files.

3. Lab Share - Lustre

Automatically mounted on /data/labs/your_labname. This storage is a Lustre filesystem cache which uses AWS S3 storage as a backend for storing large amount of data.

Features:

• Persistent: The data in this storage is under the hood stored on S3, versioned and protected by S3 internal replication.
• Shared across sessions: the filesystem is mounted across all Linux sessions, so you can see the same data from different sessions.
• High Performance: the Lustre cache offers tunable high performance, and it is the recommended location for your main IO during your calculations. If you feel you need additional performance, please send in a request and URC engineers will evaluate with you and if necessary will increase size or performance tier of the lustre cache system.
• Large Data: Data is automatically offloaded to S3, which offers ideal storage solution for large amount of data.