Skip to content

Choosing Resources

(Borrowing from other authoritative sources & our partners..)

If working on a compute cluster like ReD's Parallel Cluster/Open OnDemand (PC/OOD), it is important to accurately request your job resources, especially memory. Failure to do so can result in jobs crashing, you or others waiting longer for results, and (worst case) a large bill from AWS. If working in Research and Engineering Studio (RES), choosing the appropriate-sized and -typed compute instance type is important for keeping the costs low, ensuring your session operates as smoothly and efficiently as possible , and is not automatically stopped as if it appears idle.

Assumption:

  • For PC/OOD, you are understand how compute clusters work and are familiar with job scheduling
  • For RES, you understand the basics of how computer software runs, and the importance and difference between CPU cores, RAM, and (disk) storage.

Parallel Cluster/Open OnDemand

The SLURM scheduler must have an accurate idea of the resources your program(s) will use so that it can effectively schedule your jobs in coordination with others in your group (and in other cases, with everyone else on a given cluster). If insufficient memory is allocated, your program may crash (often in an unintelligible way); if too much memory is allocated, resources that could be used for other jobs – yours or someone else's – will be wasted, often resulting in longer wait times and large compute usage charges. Requestin CPUs is similar: under-request and your job runs slowly; over-request and resources are wasted (which you pay for). Both are sub-optimal and aren't conducive to maximizing your research output.

An analogy that might help with this process is making a car reservation for the Martha's Vineyard Ferry: You are asked the number of people, the size of your car, type of car, etc. It is important that this be as accurate as possible: the crew needs to fit your car (with you in it) on a ferry of limited space and capacity. If you over-inflate the size, then space is wasted and fewer people can be ferried. If you under-report your size, your vehicle may not fit and there is a good likelihood that you may be rejected from that scheduled service.

CPU Requirements

This resource selection is relatively straightforward in some cases, and 'it depends' in others. Briefly:

  • Be thoughtful of on how many CPU cores to request. Careless choices may result in others waiting to execute their own work. A mindful approach means everyone shares the resources.
  • Use fewer or one CPU core for interactive work: interactive == the computer is waiting for you. Some exceptions on this rule:
  • If you're executing and finishing your work in a short period of time
  • If most of your time is computing instead of coding
  • R and Python can only use one CPU core unless you have written code that is parallelized OR you are using R packages or Python modules that are explicitly parallelized
  • If parallelizing your code or running parallelized code, running basic scaling tests should inform the best number of CPU cores to choose.

Note: If using software, packages, or modules that are parallelize, it is extremely important to read the documentation to understand how the parallelization works. Failure to do so could result in jobs running poorly, crashing compute nodes, or jobs being terminated.

### RAM Requirements

There are several strategies, in priority order, that one can use to correctly size your work requirements, esp. RAM. This will seem tedious and present uncertainty at first, but in short time, it will seem very second nature:

  • Repeating History (or, Being Historically Informed)
  • It It Fits, Use It
  • Guestimate
  • Try and Resize

Repeating History

This approach is rather straightforward: if you've been able to run something successfully, review the resource usage metrics from these past runs, adjust the RAM (and CPU) footprint appropriately, and run again.

If It Fits, Use It

If one can run the analysis on a local machine, then request resources that match the RAM and CPU core count for this machine. This information is available on Macs when selecting the About This Mac menu item (select Apple menu > About This Mac) or on Pcs when selecting the Properties pop-up menu item for This PC in the Windows File Browser (File Browser > This PC > Properties).

Guestimate

Here, you are making an estimate based on various factors that inform the size, and adding a little extra room for padding:

  • If importing text files (a one-time operation for your workflows), you could require up to 10X the size of a compressed text file or 4 - 6x an uncompressed one.
  • For working with binary data files:
  • If doing work with existing data structures, ask for the size of the file ≤ * 1.5
  • If creating new data structures, consider increasing this amount by 2x or 3x, depending on the amount of data being manipulated, duplicated, or transformed
  • Both R and Python offer detailed guidance on RAM usage, appropriate data structures, and how to work with large datasets with efficiency. Don't pass up on this helpful information!

Note: We highly discourage ongoing reading and writing text files as the main way to work with data. Not only is it time consuming, but significantly more RAM is needed for this pattern than with binary files.*

Try and Resize

If you really don’t know how much RAM to ask for or when starting a new workflow with different requirements than before:

  • Pick a memory amount and verify that everything works. On the next round of work, reduce your resource requests.
  • if your work crashes or your software reports out of memory errors, consider small increases in the memory amount, say 25% to 50%, instead of double or triple – the latter approach results quickly in overasks, wasting resources, and increasing costs unnecessarily.
  • Help yourself by using native functions in your tool of choice to determine the current memory usage of data structures loaded into memory. Some examples include:
  • hpy() from Python's guppy module
  • mem_used() from R's pryr package

Other Considerations

Some additional guidelines when considering what and how to execute your work:

  • One should refactor differently-shaped tasks into separate jobs. This is especially imperative for long-running jobs/workflows: this reduces the likelihood that the largest task balloons the resource requests for all parts of the job, and the smaller tasks subsequently wasted these resources
  • To facilitate this, one should take the opportunity to leverage workflows, use job dependency capabilities, and script, scheduler, and framework/platform automations to reduce your cognitive burden in coordinating compute logistics.
  • Data imports to- and from- text formats should be one-off tasks: typically the import process requires a large amount of RAM while ingesting and transforming text data into the binary representation of the data, esp. if the software must guess at the data types for your variables.
  • Optimizing RES instance selection

  • Monitoring RAM and CPU usage in RES

  • Determining job efficiency in PC/OOD

  • Performing scaling tests

Updated: 4/1/2026