Urika Training

Venue and Timings

Conference Room # 310, TAMUQ Building
Timings: 0830-0430pm
First Coffee Break at: 10am - 1015am
Lunch Break: 12pm to 01pm
Afternoon Coffee Break: 230pm - 0245pm

Path to shared files

/lustre/share/urika_training

Reservation

#SBATCH --account=ut
#SBATCH --reservation=ut
#SBATCH --partition=l_long
#SBATCH --qos=lcustom3

How to request allocation?

salloc -N 4 --account=ut --reservation=ut --partition=l_long --qos=lcustom3 start_analytics

Create tunnel for jupyter notebooks

Instructions for setting up tunneling on raad2 for Jupyter notebooks

1. Create a directory called "notebooks" (no quotes) in your home directory. Copy all the files from /lustre/share/urika_training/uxc_1.1/notebooks to this directory. Also, copy the file alldemo.yml from /lustre/share/urika_training/uxc_1.1/Anaconda to this directory.
2. Create tunnel from laptop through raad2 elogin node to raad2-login1 OR raad2-login2
3. Choose any 3 random number (port_num_1, port_num_2, port_num_3) b/w [10000 - 50000], these will be used as port numbers for tunneling.
4. From a terminal session on your laptop:

(use your own login instead of jamaltb30!)
laptop$ ssh -L <port_num_1>:raad2-login2:22 -N -f <user_name>@raad2.hbku.edu.qa

1. This will return you to your laptop prompt, leaving a tunnel active. Then log directly into the login2 node:

laptop$ ssh -l jamaltb30 -L <port_num_2>:localhost:<port_num_3> -p <port_num_1> localhost
jamaltb30@raad2-login2:~>

1. Load the analytics module

module load analytics

1. Start the analytics package on login2 with the -d flag

jamaltb30@raad2-login2:~> start_analytics -d

1. Create a conda environment to run the demos. You will only need to do this once.

jamaltb30@raad2-login2:~> conda env create -f alldemo.yml
jamaltb30@raad2-login2:~> exit

1. Start up the analytics cluster, specifying the communications ports and conda environment

jamaltb30@raad2-login2:~> salloc -N 4 -p l_long --qos=lcustom3 --account=ut --reservation=ut start_analytics --login-port <port_num_3> --ui-port <port_num_2> --dask-env alldemo

1. export the python environment to all the compute nodes in the cluster

a_student@nid00032:~> export PYSPARK_PYTHON=$(which python)

1. Run the jupyter notebook

jupyter notebook --port <port_num_2>

After some messages, it will give you a string with a passcode that you can paste into the address window of your local browser on your laptop.
The jupyter notebook should start up!