Raad2: Python

Using Python on raad2 is a 03 steps process.

a. Create Virtual Environment (Either with Python VirtualEnv or Anaconda Virtual Environments and install all the requirements.
b. Run interactively on a compute node and test your code.
c. Submit as batch job on the system for longer runs.

Decide on Python version and Packages requirements

Users should identify which version of Python is required for their application and the required packages. Once requirements are identified, users can create Python Virtual Environment using Anaconda Distribution of Python.

Users can use Python on raad2 in following ways;

a. Anaconda distribution of Python
b. Python via Modules (Not recommended)

Anaconda Distribution

Anaconda distribution allows users to create a Virtual Python environment with specific release and also makes easier for users to install custom packages which have multiple libraries dependencies. For example if your project 'X' requires Python 3.6.7 with TensorFlow 1.0, you can create a Conda Python Virtual Environment with required dependencies specific for this project. Similarly for another project 'Y', you can create another Virtual environment with totally different version of Python and TensorFlow version. Both Virtual environment will co-exist on your system and you can activate either of them as per project requirements.

In below example, we will demonstrate how you can create a Conda virtual environment and install required packages. Once the environment is created, you will learn how to use this environment in interactive session or in batch mode while working on raad2.

Note: If you were making use of old instructions to run anaconda3 on raad2, you may need to update your .bashrc. Edit your .bashrc file and remove line " source <old path>/etc/profile.d/conda.sh"

Step01: The first step is to load Anaconda module which will make conda executable available in your environment.

echo "source /ddn/sw/xc40/cle7/anaconda/2023.03-1/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc

Step 02: Load latest Python version

module load python/3113

python3.11 -V

Step 03: Now create a virtual environment and choose the python version you require. In this example, we are creating environment with Python version 3.6. During this process, Conda installer will install any dependencies which are required to make your Python version work. Requested version of Python and packages will be installed in your home directory.

muarif092@raad2a:~> conda create -n DLProj python=3.11

Step 04: Once the Python environment is created, you have to activate the environment to install new packages. Activating the environment sets user environment variables and make the respective version of Python available to use.

muarif092@raad2a:~> conda activate DLProj
(DLProj) muarif092@raad2a:~>

Step 05: In the terminal, you will notice that (DLProject) is added, this means that you are working inside the Conda python virtual environment. Now we can verify that the requested version of Python is correctly installed.

(DLProj) muarif092@raad2a:~> which python3
/lustre/home/muarif092/.conda/envs/DLProj/bin/python3

(DLProj) muarif092@raad2a:~> python3.11 --version
Python 3.11.3 :: Anaconda, Inc.
(DLProj) muarif092@raad2a:~>

Step 06: Install any additional Python Packages. Conda installer will list all the packages will be downloaded, upgraded and installed.

(DLProj) muarif092@raad2a:~> conda install tensorflow keras matplotlib

Step 07: Your Virtual Environment is ready and can be used for Testing codes and then Batch submission. You can navigate this page to find out how to submit Test Runs and Batch Jobs. At this point, you can deactivate your Virtual Environment.

(DLProj) muarif092@raad2a:~> conda deactivate

Tip 01: To see the list of Conda environments you have:

muarif092@raad2a:~> conda env list

Tip 02: To delete any Conda environment ("DLProj" in the example below) which is no longer required:

conda remove --name DLProj --all

Import/Export of Conda environment

Conda eases the process of porting any environment from a system to another. The magic of Conda is that you can smoothly port your environment to machines with different operating systems. I.e., you can migrate your tool chain from Windows to Windows, Linux to Linux, or even from Windows to Linux and vice versa in a convenient manner.
The main artifact used in such operation is called environment.yml , based on YAML format, which is a superset of the JSON markup language.

Export your Conda environment

From Linux

• Open your Linux terminal
• Create the environment.yml file out of the environment you want to export

albelfe17@raad2a:~> conda env export  --from-history -n my_env_to_export > my_env.yml

• Here my_env_to_export is the name of the Conda environment you wish to export,
• And my_env.yml will be the name of your environment.yml file.
• ==> That's it.<==

The --from-history option is important here, it will increase the portability of your environment. If you omit it, your environment.yml file will be super detailed, i.e. it will not only contain the versions of the tools, but also their build numbers. This will greatly increase the chances of your following import to fail, because of incompatibilities.

From Windows

• Get the Anaconda Navigator app
• Install it and open it
• Go to the Environment tab
• Click on the Play button of the environment you wish to export
• Select Open Terminal

• Type the same command as in Linux section, i,e:

 (my_env_to_export) C:\Users\abelferrag> conda env export  --from-history > my_env.yml

Manual Creation of the environment.yml file

• If for some reasons, the automatic creation of your environment.yml fails, you can still create it manually
• The format is the following:

name: my_new_env
channels:
  - defaults
  - conda-forge
  - Esri
dependencies:
  - python=3.10.13
  - pytorch==1.9.1
  - pandas
  - numpy
  - torchvision
  - pip
  - pip:
    - thermo==0.1.1

• name: specify the name of the Conda environment to be created
• channels: tell Conda where to look for the listed packages
• dependencies: the packages you want to install in your new environment
  • you can even specify the version of the package you want to install
• pip: the packages that you want to install that are not part of any Conda channel

Import your Conda environment

To Linux

• Open your Linux terminal
• Locate the path of the environment.yml file you created in the export section: my_env.yml
• Import the environment:

conda env create --name <name_of_your_new_env> --file <path/to/my_env.yml>

• ==> That's it <==

To Windows

• Open Anaconda Navigator app
• Go to Environment tab
• Click on Import (in the bottom left corner of the window)
• Click on the folder icon of the Local drive section and browse to your my_env.yml file
• Click on Import

• ==>Voila.<==

Some rules of thumbs

• In your import / export operation, the best practice is to rely only on Conda, as much as possible
• Only if there is no choice, use pip
• No choice here means that the package you are looking for, or the specific version of this package is not present in any Anaconda channel
• Ideal workflow:
  1. If you don't need specific tool versions, let Conda choose for you, do not specify it
  2. If you need a specific version specify it but don't mention the versions of the dependencies, by default Conda will resolve them for you
  3. When you do that and still encounter a version issue, then specify the versions of your tool and related dependencies
  4. If Conda is unable to find your specific versions, most likely it is because it does not know in which channel to search them, so you need to help him. To do that:
     1. Go to anaconda.org
     2. In the search bar, type the name of the package you are looking for, e.g. thermo. You will then find a list of channels
        • 
     3. Select a channel that contains your desired package, here it would be for instance conda-forge/thermo
     4. Click on the Files tab. Now you can filter by version, to see if your specific version is available in this channel
        • 
     5. If this channel contains the version you are looking for, add this channel to your Conda configuration:
        conda config --add channels channel-name
     6. If not, go back to the previous page (the list of channels), and inspect other channels
     7. Finally, if you can't find your package and/or its specific version in any anaconda.org channels, you can consider using pip to install it

Python via Modules

In HPC systems, multiple versions of application/packages are maintained via Modules. Users can use specific version of application by loading respective module. Modules ensure that different versions of the same application can co-exist on a system.

Available versions

a. Python 2.6.9
b. Python 3.7.9
c. Python 3.6.3
d. Python 3.11.3

Using Python via Modules

To use a specific version of Python, users can issue following;

muarif092@raad2a:~> module load python/3113
muarif092@raad2a:~> python3.11 --version
Python 3.11.3

Now if you want to use different version of Python, you can issue below;

muarif092@raad2a:~> module unload python/3113 

muarif092@raad2a:~> python3 --version
Python 3.6.15

Installing additional Python Packages

Users can install additional Python packages by creating a Python Virtual Environment. Below is an example of creating a Python Virtual Env which requires certain packages.

Step 01: Decide on base version of Python you want to use and load respective module. Here we are using Python 3.6.3

muarif092@raad2a:~> module load python/3113 
muarif092@raad2a:~> python3.11 --version
Python 3.11.3

Step 02: Create Virtual Environment

irtambo19@raad2a:~> mkdir myPythonProject

irtambo19@raad2a:~> cd myPythonProject/

irtambo19@raad2a:~/myPythonProject> virtualenv .
created virtual environment CPython3.11.3.final.0-64 in 3082ms
  creator CPython3Posix(dest=/ddn/home/irtambo19/myPythonProject, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/ddn/home/irtambo19/.local/share/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.0.0, wheel==0.41.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
irtambo19@raad2a:~/myPythonProject>

Step 03: Activate the Virtual Environment which was created in last step. Notice the change in terminal line which now shows that you are in a Virtual Environment and if you install any Python packages, those will be installed within this environment.

irtambo19@raad2a:~/myPythonProject> source bin/activate
(myPythonProject) irtambo19@raad2a:~/myPythonProject>

Step 04: Install any Python Packages requirements.

(myPythonProject) irtambo192@raad2a:~/myPythonProject> pip install matplotlib

Note: If your code has a lot of library dependencies and you require multiple additional Python packages, we recommend using Anaconda Distribution of Python.

Test your application via Interactive Job on raad2

When we created Python Virtual Environment, all the work was done on Login node i.e. raad2a/raad2b. The login nodes are meant for development purposes only, where you can build/compile your application. A login node is shared among many users of the HPC system. Users should submit Interactive jobs to get an allocation on a compute node for code testing.

Step 01: Request for an interactive session to a compute node. Interactive session will give you terminal access to a compute node where you can do test runs. Notice the change in node name when you issued sinteractive command. This confirms that now you are on one of the compute node i.e. nid00xx

irtambo19@raad2a:~> sinteractive
irtambo19@nid00444:~>

Step 02: Once you have allocation to a compute node, you can then activate the Python Virtual Environment which you created earlier.

muarif092@raad2a:~> sinteractive 
muarif092@nid00388:~> conda activate DLProj
(DLProj) muarif092@nid00388:~> which python3
/lustre/home/muarif092/.conda/envs/DLProj/bin/python3

(DLProj) muarif092@nid00388:~> python3
Python 3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 23:51:54) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

(DLProj) muarif092@nid00388:~> python3 mycode.py

Step 03: Once you are done testing your code and ready for production runs, you can exit from the current session.

(DLProj) muarif092@nid00388:~> exit
exit
Connection to raad2-login2 closed.
muarif092@raad2a:~>

Submit a batch job for longer runs

Batch Job allows you to package your application and define all the requirements in a file called as "job file". Then you submit this Job file to system which then runs your simulation. The results of the simulations are either written to file(s) you create in your source code Or slurm output/error files generated in the same directory from where you submitted your job file.

Step 01: Create a directory which will have all the input files.

muarif092@raad2a:~> mkdir myDlRun
muarif092@raad2a:~> cd myDlRun/

Step 02: Create slurm.job file with following contents;

#!/bin/bash
#SBATCH -J Python_job
#SBATCH -p s_debug
#SBATCH --qos sd
#SBATCH --time=08:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --output=slurm.o%j
#SBATCH --error=slurm.e%j
#SBATCH --hint=nomultithread
#SBATCH --gres=craynetwork:0 # Do not remove this line if you are submitting to s_* queues.

echo "Starting at "`date`
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR

# Source conda.sh to make conda executables available in the environment
source /ddn/sw/xc40/cle7/anaconda/2023.03-1/anaconda3/etc/profile.d/conda.sh

# Activate Conda environment which you created and tested earlier
conda activate DLProject

# Launch computation
srun --ntasks=1 python3 DLcode.py

echo "Simulation Ending.."
echo "Ending at "`date`

Step 03: Submit slurm.job file to raad2

muarif092@raad2a:~/myDlRun> sbatch slurm.job

Step 04: Monitor your job

You can monitor your job by inspecting output file created in the current directory.