Using ANSYS (Fluent) on the HPC System

USING ANSYS (FLUENT) ON THE HPC SYSTEM

To use the ANSYS software on the HPC, you will need the following:

Access to the HPC system (Contact HPC support if you need an account created).
A connection to the HPC System, see Connecting to the Marie Curie Cluster for information on how to do this.
If you plan on using running the ANSYS Graphical Software (such as workbench and Fluent) on the HPC system, you will require a graphical interface to the HPC system. See Graphical Connection to the HPC System for information on how to do this.

Selecting a different version of the ANSYS Software

For those who already have a HPC account and are using a “graphical” connection, you should be able to start ANSYS Workbench by issuing the following commands (inside a terminal session and within the directory you wish to stored your ANSYS files):

On our HPC system, we actually have different versions of the ANSYS software installed. Please visit the HPC software Engineering applications listings which lists the different versions of the ANSYS software available on our system.

Therefore if you want to use a different versions of the ansys software, you will need to issue the following commands to unload current version of the software and then load the version you want.

module unload ansys

module load <module name>

example

module load ansys-2019-r2

Using ANSYS Workbench (Interactively)

For those who already have a HPC account and are using a “graphical” connection, you can start an interactive session to run the ANSYS software .

Note, if you are having issues, you can “purge” all of the loaded software, in which will reduce conflicts, etc. But you should have to issue these commands for normal usage.

module purge
module load ansys
or
module load <ansys module name>
module load intel-parallel-studio-2018

To start ANSYS Workbench by issuing the following command (inside a terminal session and within the directory you wish to stored your ANSYS files):

runwb2

runwb2 -oglmesa

Use the above command if you want to force software rendering for the ansys software.

When using ANSYS workbench, you are able to solve computational fluid dynamics (fluent), or finite element analysis (Mechanical) models. To ensure ANSYS licenses are not wasted, please close the application down when finished.

Which “Graphical” connection site should I use?

On the CQUniversity HPC facilities, we provide a number of options when connecting to the HPC system using a graphical user interface.

Using the “Strudel software – see Graphical Connection to the HPC System for instructions”, you have a number of selection options.

HPC Login Node – Marie;
HPC Login Node – Curie;
GPU Interactive Session; or
Standard Interactive Session.

Users are advised to use one of the “Login Node options”, if they are developing a model, or processing a quick model. If users plan on running any processing jobs, they should select the GPU Interactive Session and Standard Interactive Session option.

The benefit for using the either the “GPU or Standard Interactive Session” option, is that the processing occurs on the HPC Compute nodes, thus not overloading the login nodes. If you are using either Interactive Session option, ensure you select the “Stop the Desktop” option, when your modelling/processing has been completed. This will ensure the resources are freed up for other users to use.

It has been found that selecting the “GPU Interactive Session” preforms significantly better when using complex 3D models (especially in generating and displaying the model).

Running ANSYS fluent application across multiple CPU cores

There is a bug in which if you try to run there ANSYS fluent application across multiple CPU cores on the HPC system within apply the suggested workaround detailed before, it will causes fluent to slow dramatically down (with some jobs taking being 100 times slower)!

Therefore the following information provides a guide on how to correctly configure the ANSYS software to utilise multiple CPU’s to achieve significant perform improved in solving the simulations!

Start strudel and select the ‘Standard Interactive Session’ site option.
Enter in the required resources required, If you want to consume an entire HPC compute node, you can select the example options listed below.

It should be noted that more CPU’s won’t guarantee it will be faster and in some cases, the more CPU’s you select the slower it goes, due to the inter-process communication overhead.
Open terminal and then issue the command runwb2 -oglmesa (to start ANSYS Workbench in software rendering mode)

Open your saved workbench project
Double click on setup and select to open the ‘launcher setup options’
Click on the ‘Processing Options’ parallel and select you requested CPU’s. It is recommended to save one or two CPU’s for the communications between the processes. Therefore, in the above example where I have asked for 24 CPU’s, within the ANSYS setup configurations, select 23 CPU’s, as this will make solving a lot more efficient.
Now for the most important part, click on the ‘Show More Options’ option and then select the ‘Environment tab (If you don’t add this environment setting, the program will perform extremely slow.
Click on the “Parallel Settings Tab and select the following options:
- Interconnects – infiniband
- MPI Types – intel
- Run types, you can experiment with this, but you can try either Shared Memory or Distributed Memory option

Once the Parallel Settings environment variables have been selected, click on the OK button to proceed with starting the simulation.

Using the Standalone ANSYS solver launcher application

The Ansys software has an application that allows you to submit jobs to be solved through their graphical application “Launcher“. To use this application, simply follow these steps,

Start a terminal session and issue the launcher command
You will be presented with the “stand alone ANSYS solver launcher application”

Click on the “Browse button” within the Job Name section and then you can search for your “ANSYS” programs

You can also select the different types of files to look for

It should then show which program you would like to solve manually.

You can select the “High Performance Computing Setup – to make the job run faster by using my CPU resource to solve the problem. For example, the image below show the problem being solved across 8 CPU’s.

Once you are ready, click on the Run button to fire off your job to be solved.

You will see the progress of your program in a XTERM window.

After the time required, your problem will be solved. You can then use the workbench interface to look at the solved model.

Solving ANSYS jobs non-interactively

One of the benefits of using the HPC system is that you can submit 1 to many jobs to the HPC scheduler. Using the Scheduler, you can request more resources (such as CPU’s) which can dramatically improve the processing execution time.

To solve an ANSYS Jobs on the HPC system, you will need to create an input file. In ANSYS Workbench this can be done by selecting Tools -> Write Input File.

An example of a ANSYS HPC scheduler script – make sure you change any of the requested resources – such as memory – your problem being solve will require:

ansys.pbs

#!/bin/bash -l #PBS -N AnsysTest #PBS -l walltime=24:00:00 #PBS -l mem=8000mb #PBS -l ncpus=8 #PBS -l ANSYS=1

cd ~/YOUR_PROJECT ansys192 -np 8 < YOUR_INPUT_FILE.inp > YOUR_OUTPUT_FILE.out

Fluent input files reference a CASE file created using the Fluent GUI.

fluent.in

; Fluent Example Input File ;---------------- ; Read case file /file/read-case YOUR_CASE_FILE.cas.gz ;---------------- ; Set the number of time steps and iterations/step /solve/iterate 2 ;---------------- ; Save Case & Data files /file/write-data YOUR_OUTPUT_DATA_FILE.dat.gz /exit yes

An example ANSYS Fluent HPC scheduler script – make sure you change any of the requested resources – such as memory – your problem being solve will require:

fluent.pbs

#!/bin/bash -l #PBS -N FluentTest #PBS -l walltime=24:00:00 #PBS -l mem=4000mb #PBS -l ncpus=8 #PBS -l ANSYS=1

cd ~/DIRECTORY_WITH_CAS_FILE fluent 3d -g -t 8 -i fluent.in > YOUR_OUTPUT_FILE.out

Trouble shooting

On some older versions of the ANSYS software, there is a minor bug that can cause the software to crash upon start-up. During the first time you start up the software the “ANSYS Product Improvement Program” window will pop up. If you select the “No, I would not like to participate” the software will crash. Please either click on the cross to close the windows or select the “yes” option. Once the software has started correctly, you can then go into the Help >> ANSYS Product Improvement Program option and then select the “no” option. This seems to then work fine.