Minnesota Supercomputing Institute
OpenFOAM is an open source software package which is designed to scale well on large distributed memory systems. While originally written for CFD, with it's open source strategy and a large user base OpenFOAM has grown to be a general purpose finite element toolbox.
MSI currently provide OpenFOAM outside of the module system. See the additional tabs on this page for instructions on how to access this software on Mesabi.
OpenFOAM v6 is available on Mesabi. Several working examples are posted, which illustrate how to run interactive with graphics using ParaFOAM, and how to run non-interactively in the PBS batch queue.
Run Interactively with GraphicsIn a terminal window on an MSI NICE or NX desktopssh -X mesabi/home/dhp/public/sing/openfoam6/run
This will start an instance of the openfoam6-paraview54 singularity container. At the prompt in this container, enter you can get and run backward facing step with simpleFoam with:cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .cd pitzDailyblockMeshsimpleFoam
View results with Paraview
Run in PBS batchFrom a login shell on Mesabi, get a copy of the OpenFOAM v6 tutorial pitzDaily example, PBS run script, and submit it to the queue:cp -r /panfs/roc/msisoft/openfoam/6.0/pitzDaily_v6 .cd pitzDaily_v6cp /panfs/roc/msisoft/openfoam/6.0/pbs_run .qsub pbs_run
This is a tiny example which runs for a very short time. The PBS resource request only asks for 2 cores and 10 min of run time. In fact, it’s running OpenFOAM single-threaded, and only asks for 2 cores to ensure it has enough memory. To run larger problems, request all 24 cores on a node and a longer wall clock time. For example, to run for as long as 6 hours and have the full memory of the Mesabi compute node, use a resource request like
#PBS -l nodes=1:ppn=24,pmem=2580mb,walltime=06:00:00
This example runs blockMesh, simpleFoam, and foamToVTK non-interactively in a PBS job, using a singularity container. It does so by first creating a bash script (sing_input.sh) and then running the container on this script with
singularity exec $container_file $wrkdir/sing_input.sh
You can modify run_pbs to run your own OpenFOAM commands for your own use case.
This example is setup to work as-is for any user, so you do not need to make any modifications to see it run. It has you copy the OpenFOAM v6 tutorial pitzDaily directory to a location anywhere under your home directory. You then use this directory as a working directory (i.e.,. where you run your job, and OpenFOAM does its work). The PBS script (see the file pbs_run copied below) generates a path to the working directory in the shell variable wrkdir with
wrkdir=$HOME/$(echo $PWD | cut -d'/' -f8-)
You can leave this as is if you want to have your PBS run script in the OpenFOAM project directory.
Example PBS Batch File (pbs_run)
#!/bin/bash -l#PBS -l nodes=1:ppn=2,pmem=2580mb,walltime=00:10:00#PBS -j oecd $PBS_O_WORKDIR# Construct working directory path that will be valid in containerwrkdir=$HOME/$(echo $PWD | cut -d'/' -f8-)# Create bash shell scripti with commands you want to run in containerecho "#!/bin/bash -l# Set OpenFOAM6 environment, and working directory, inside of containersource /opt/openfoam6/etc/bashrccd $wrkdir# Your OpenFOAM commands go here (must be non-interactive)blockMeshsimpleFoamfoamToVTK" > sing_input.shchmod 755 sing_input.shmodule load singularitycontainer_file=/panfs/roc/msisoft/openfoam/6.0/openfoam6-paraview54.simgsingularity exec $container_file $wrkdir/sing_input.sh