• eResearch
    • Collaborative Technologies
      • ARDC (Australian Research Data Commons)
      • ARDC Nectar Research Cloud
      • Australian Access Federation
      • QRIScloud
      • Video Collaboration
    • Data Management
      • Research Data Management Plans
    • Data Services
      • Australian and International Data Portals
      • CQUni Research Data Storage Options
      • CQUniversity Research Data Storage
      • GEOSCIENCE DATA PORTALS
    • eResearch and Security: MFA and CyberSafety
      • Encrypting Data on Portable Devices
    • High Performance Computing
      • The History of CQU’s HPC Facilities
        • Ada Lovelace Cluster (Current)
        • Marie Curie Cluster (Decommissioning)
        • Einstein Cluster (Decommissioned)
        • Isaac Newton HPC Facility (Decommissioned)
      • HPC User Guides and FAQs
        • Basics of working on the HPC
        • Getting started on CQUniversity’s Ada Lovelace HPC System
        • Compiling Programs (and using the optimization flags)
        • Creating a Conda Enviroment
        • Finding Installed Software
        • Frequently Asked Questions
        • Graphical connection HPC via Open On Demand
        • HPC Trouble Shooting
        • LLM’s on Ada Cluster
        • Machine and Deep Learning modules on Ada
        • PBS to Slurm Command tables (HPC Scheduler)
        • Running Python on Ada
        • Simple Unix Commands
        • Slurm Commands
        • Software Module Information
        • Transferring Files to the HPC System (Ada)
        • Using Abaqus
        • Using ANSYS (Fluent) on the HPC System
        • Using APSIM
        • Using HPC Scheduler on Ada Lovelace Cluster
        • Using MATLAB
        • Using R
        • Virtualisation and Containers
      • HPC Community
      • HPC Related Links
      • HPC Sample Code Scripts
        • Multiple Job Submission – Slurm
        • Parameter sweep multiple job – Slurm
        • R Sample Scripts – Slurm
        • Sample Slurm Submission Script
      • HPC Software
    • Research Software
    • Scholarly Communication
    • Survey Tools
    • Training
      • QCIF – Queensland Cyber Infrastructure Foundation
      • Teaching Lab Skills for Scientific Computing

eResearch

Deploying your own Conda python environment

There is a software carpentry lesson currently being developed that might be useful in understanding how Conda Environments work.  See Introduction to Conda for (Data) Scientists help on understanding and using Conda.

There are a number of things to consider, before using this approach.

  • You should only deploy a full conda environment on the HPC system if you have hard or complex requirements.  The use and deployment of many “conda” environments can very quickly create 1000’s – 1,000,000’s of files and actually cause HPC filesystem performance issues.  Whilst there are no issues if you need to use a conda environment, it is recommended that not everyone creates one if the preinstalled python software modules or locally installed python modules options is not effective first in the first instance.

To deploy your own conda environment, you can use these instructions

Load a Miniconda software module

(in the example below, the latest version is used)

You need to load a conda software module to be able to create or activate a conda environment.

module load Miniconda3/24.7.1-0

Creating a conda environment

It is good practice to give your environment a meaningful name to help yourself remember the purpose of the environment. While naming things can be difficult, $PROJECT_NAME-env is a good convention to follow. You may also consider using the specific version of a package or why you had to create a new environment.

conda create --name python39-env python=3.9

# # To activate this environment, use # #

conda activate python36-env

# # To deactivate an active environment, use # #

conda deactivate

Activating a conda environment

To activate a conda environment, @ada $ conda activate python36-env (python36-env) bash-4.2$

Deactivate the current environment

To deactivate the environment

conda deactivate python36-env

To install additional python packages within the conda environment

Note you need to be in activated conda environment to run these commands

python36-env) $ conda install matplotlib

Collecting package metadata (current_repodata.json): done

Solving environment: done

Support

eresearch@cqu.edu.au

tasac@cqu.edu.au OR 1300 666 620

Hacky Hour (3pm – 4pm every Tuesday)

High Performance Computing Teams site