Getting Started with HiPerGator
Posted: Sun Oct 10, 2021 6:04 am
BASICS
Alright, we've worked through basic principles of AI using P5.js and ML5.js. Now let's work with AI using some tools that allow us to interact with the outstanding resources available via: UF's HiPerGator AI.Python is commonly used in AI work and many resources are available online. A common technique is to prototype and develop an AI application using a development tool like Google Collab or Jupyter. Once the app is designed, tested, and optimized, it is typically run as a Python program in production. Many of the programs with which we'll experiment are intended to run on Collab or via a local installation. To get them to work on HiPerGator we'll need to make a few adjustments.
Since HiPerGator is a cluster of computing resources, it manages the distribution of work to the various nodes using the SLURM workload management system. If you are reading this and you're in the Art + Technology program, you can think of SLURM as something similar to our own Thinkbox Deadline renderfarm submission system, but SLURM is more commonly available in the world of high-performance computing systems. HiPerGator provides resources for using Jupyter, which we'll focus on below. You should recognize that you can install and run Jupyter, locally, on your personal computer; you just won't have ready access to all the CPUs and GPUs available via HiPerGator. Fortunately, you can remotely access HiPerGator from your personal computer and we'll show you how to do that as well.
REFERENCE
Pre-recorded TrainingJupyter Notebooks
AI HELP
TensorFlow
Modules (see also Basic Usage)
Locally available Datasets (i.e. on the HiPerGator system)
CONFIG
Let's log into UF HiPerGator using the Secure Shell protocol (SSH). Substitute your gatorlink username for <gatorusername> below:Code: Select all
ssh <gatorusername>@hpg.rc.ufl.edu
Code: Select all
hgator
The default shell on the Linux login nodes on HiPerGator is BASH. Let's set up a basic shell configuration by adding some things to your ".bash_profile" using the "nano" text editor.
Code: Select all
nano .bash_profile
Code: Select all
EDITOR=nano
#make data accessible to all group members, but not to the users outside of the group
umask 007
#configure the command prompt
export PS1='\[\033[32m\][\u@\h]\[\033[34m\](\w)\$ \[\033[00m\]'
#modify command-line-completion
bind 'set completion-ignore-case on'
bind 'set print-completions-horizontaly on'
bind 'set show-all-if-ambiguous on'
# ls with color, human readable file sizes, reversed, etc.
alias lr='ls -hartl'
alias ll='ls -halG'
alias l='ls -G'
# shorter cd ..
alias .='cd ..'
# nano without wrapping, which is essential for editing config files
alias edit='nano -w'
To make the changes register with the shell we need to tell the shell to re-read the configuration. Do that by typing:
Code: Select all
source .bash_profile
Next, recall in your HPC New User training, they want us to store our working files on the "blue" storage system rather than our home area or "orange," right? Also recall that these storage locations are not available to us (i.e. mounted) until we first access them by navigating to them. To make this easier for ourselves in the future, we're going to create a shortcut in our home area that points to our most commonly used storage directory; we're going to create a symbolic link (like an alias) to our class directory on the HiPerGator "blue" drive. Any time you log in, that's where we want to do most of our daily work.
First, we'll need to "mount" our "blue" storage system. Substitute your groupname (art4630, cota-dept, <username>, etc.) for "art4659" in the command below:
Code: Select all
cd /blue/art4659
Code: Select all
cd ~
Code: Select all
ln -s /blue/art4659/ blue_ai
Code: Select all
ls -al
Code: Select all
blue_ai -> /blue/art4659/
Code: Select all
cd blue_ai
Code: Select all
ls -al
Code: Select all
cd shared
mkdir <gatorusername>
cd <gatorusername>
mkdir code (or whatever you want to call it)
cd code
Configure Anaconda:
You'll probably want to use Anaconda to manage working environments for projects. You'll use HPC's module system to load Anaconda, but you should edit the .condarc file to store your packages and env files on "blue" instead of your home area, which will fill up quickly. If you've never loaded the Anaconda module, load it for the first time so it will create a default .condarc file for us to edit.Code: Select all
module load conda
Here's an example for my ART 4659 class:
Code: Select all
envs_dirs:
- /blue/art4659/${USER}/conda/envs
pkgs_dirs:
- /blue/art4659/${USER}/conda/pkgs
Code: Select all
envs_dirs:
- /blue/cota-dept/saah/${USER}/conda/envs
pkgs_dirs:
- /blue/cota-dept/saah/${USER}/conda/pkgs