Introduction to RSPH Compute Cluster

Cluster Structure

120+ compute nodes

• 24 nodes have 32 cores and 196 GB RAM each
• 1 "large memory node" with 1.5 TB RAM
• The other cluster systems consist of 70+ reserved use faculty nodes, which are also available for general, preemptive use when they are not in active use by their research groups.
• 4 private nodes with partition "yanglab" (32 cores, 188GB RAM) available for yang lab members
• These systems are all connected together via 25GB Ethernet high-speed network, and all have access to a shared 1.2 Petabyte Panasas parallel file system.
  • Extra 20TB storage for yanglab /projects/YangLabData/
• Support thousands of concurrent running job managed by SLURM
• IT help: help@sph.emory.edu

Login

• First login to Emory VPN
• Login using ssh by typing the following command in respective terminal ssh <your_Emory_NetID>@clogin01.sph.emory.edu
  1. On Mac OS X system, open a terminal app and type ssh
  2. Under Linux, open a terminal and type ssh
  3. Under Windows 10, open up a Powershell (search for “Powershell” in the search tool next to the Start menu) and type ssh. (Older versions of the operating system may require the installation of a third party ssh client, like Putty.)
• Login without password

Mount Remote Directory to Local Computer

• Use WinSCP for Windows system
• Use macFUSE/SSHFS for MAC
  1. Install SSHFS
  2. Install macFUSE
  3. Mount RHPC server directory to your local mount directory Example command to mount remote directory /home/jyang51/ to local mount directory /Users/jyang51/RHPC/ sshfs jyang@clogin01.sph.emory.edu:/home/jyang51/ /Users/jyang51/RHPC/ -o auto_cache -ovolname=RHPC -o follow_symlinks 

Storage on the Cluster

Sub volumns all mounted across the cluster

• /home where user home direcotry is located
  • Each user have a 25GB quota for home directory, suitable for running computations
  • Retains two snapshots of data at /home/.snapshot. If a user accidentally deletes a file, one of two possible copies of the file can be restored from the snapshot directory
• /scratch where user may load data for immediate computation
  • Usable for large computations with the two-week maximum retention policy
  • 100GB/user
• /projects where groups may store project data and run computations. Requestable in increments of 1TB, $75 per year
  • 20TB storage space is available for yang lab under /projects/YangLabData/
  • You may creat a working directory under this lab storage space for all of your work
  • **Please be mindful with your storage usage. **
    • Avoid creating duplecates of data files.
    • Delete files that you no longer need to use.
    • gzip large text data files.
• /apps and /projects/YangLabData/bin where all softwares can be found

Monitoring storage usage With command pan_quota

All files are not backed up on the cluster

Handle Data Files on Linux Cluster

Command man [command] would give help page for the following commands, e.g., man rsync. Type q to exit from the help page.

1. Command rsync is recommended for copying data on cluster
2. Command cp also copys data within cluster
3. Delete data or directory by rm
4. Make directories by mkdir
5. Move a directory by mv
6. List all files under a directory by ls
7. List all files with their sizes by ls -l -h
8. Use vi or nano to edit text files on the cluster
   
9. Read text file by less, cat
10. Consider gzip your text file to save space by gzip
11. Open gzipped text file by zcat [file_name.gz] | less -S
12. Command man to see user manual/instructions of Linux tools, e.g., man ls
13. Use pipe | to take output from the command before | as the input for the command after |
14. Command awk is highlly recomended for handling large text files.
15. Split your screen by tmux. See tmux cheat sheet.

Copying Data to/from the Cluster

1. Command rsync is recommended

   • Transfer data from local machine to cluster, or from cluster to local machine
   • Transfer data within cluster or across clusters
   • See Example usage
   • rsync [options] [SOURCE] [DESTINATION]

2. General command by using scp as follows:

   • See Example usage
   • scp <path_to_file> <username>@<remote_system_name>:<destination_path>

Using a Software

Check if a software is available on the cluster

• which [software_name]
• module spider to list all installed software modules
• module spider [software_name] to find detailed information about the software

Using a software installed on the cluster

Please never run a software directly on gateway node. First login to a compute node using command srun -p yanglab --pty --preserve-env bash.

• module load [software_name] to load the software.
• Use software command to run commands
  • Provide full directory to the data files you want to read or write or plot.
  • For example, type command R to start using R after loading R by module load R/4.2.2. You may copy and paste your R commands from a text script to the session window on RHPC.
  • Save figures as pdf into the disk to view your plots by R function ggsave("temp.pdf") or pdf("temp.pdf"); plot(); dev.off();
  • Type command python in the session to start a python session.
  • One can open the script on RHPC by local Editor (e.g., Sublime, Visual Studio Code) after mounting cloud directory to local machine, and just copy/paste command into these R or python sessions
  • Type command /projects/YangLabData/bin/plink -h to see plink user mannual.
• Command which can also be used to check if a software is available under the ${PATH} directory in current session
• Unload a software module by module unload [software_name]
• Unload all software modules by module purge

Install a software without root access

• Install software under ~/.local/
• Install python libraries by pip install --user [package]. Or search "install without root access", "install with specificed installation directory". The installation directory should be those you have writting access to.
• Install python libraries under virtual environment.

Share installed software with the lab

• Make a soft link of the executible tool directory under /projects/YangLabData/bin by the following commands

cd /projects/YangLabData/bin/
ln -s [tool directory]

• Add /projects/YangLabData/bin into the PATH environment variable by including the following command line in your ~/.bashrc file

export PATH=/projects/YangLabData/bin:$PATH

Setup one's .bashrc file

• See useful things that you can setup your own bashrc file under your home directory to save tying of commands.
  • Set up PATH system variables. See example.
  • Create alias by alias in .bashrc to set up your own shortcup of commands. See Examples.
  • Setup environment variables. See example.
  • Load software modules

Submit Jobs by SLURM

Basic SLURM commands

• sbatch to submit jobs (like qsub in Grid Engine)
  • Set arguments to sbatch in a wrapper shell (job submission) script, for example, you may use sbatch normal.sh with the following normal.sh script:

#!/bin/bash
#SBATCH --job-name=normal.R
#SBATCH —partition=yanglab
## This puts all output files in a separate directory.
#SBATCH --output=Out/normal.%A_%a.out
#SBATCH —error=Err/normal.%A_%a.err
## Submitting 100 instances of srun commands listed below
#SBATCH —array=0-100
## For notification purposes. Use your Emory email address only!
#SBATCH —mail-user=<your_email_address>@emory.edu
#SBATCH --mail-type=END,FAIL
module purge
module load R
srun /home/<user>/normal.R

• squeue to display information about the run queue
• scancel [jobid] used to cancel or kill a job
• scontrol used to show information about running or pending jobs
  • scontrol show job [jobid] to show system details of a submitted job
  • scontrol show partition day-long-cpu to show partition configuration

• srun used to run an interactive instance: srun --pty bash.

  • Open an interactive session to test your R code:

  srun -p interactive-cpu --pty bash
  module load R
  R

  • Simply exit the interactive session by exit
  • Do not test running R scripts on the login node clogin01

• sinfo used to report the state of the cluster partition and nodes

Additional Resources

• Emory VPN
• The Linux Command Line book is recommended to read ahead
• Documents about Modules
• SLURM Quickstart