Repository navigation
Installing on mesocentre
You have to load the following modules.
$ module list
Currently Loaded Modulefiles:
1) openmpi/intel/1.6.3 2) intel/13.1.2
the commands for this are
module load openmpi/intel/1.6.3
module unload intel/12.1 intel/13.0
module load intel/13.1.2
note that loading openmpi/intel/1.6.3 loads also different, unwanted versions of intel compiler, you have to be sure that only the two modules are loaded.
The first step is to install FFTW from source. Avoid using preinstalled versions as they might be changed in the future.
wget http://www.fftw.org/fftw-3.3.4.tar.gz
tar vxzf fftw-3.3.4.tar.gz
cd fftw-3.3.4/
./configure --prefix=$PWD --enable-sse2 --enable-openmp --enable-mpi
make clean
make -j12
make install
export FFT_ROOT=$PWD
echo $FFT_ROOT
cd ..
do not forget to add
export FFT_ROOT=....
directly to your .bashrc. Logout and login and test if
echo $FFT=ROOT
module list
really point to the directory you were install FFTW to, and that your really have only two module files
Second step is installing p3dfft. We realized (15. oct 2015) that p3dfft moved to github, with google code dying out.
git clone https://github.com/sdsc/p3dfft.git
cd p3dfft
./configure --prefix=$PWD --enable-intel --enable-fftw --with-fftw=$FFT_ROOT --enable-measure --enable-stride1 LDFLAGS="-lm"
make clean ; make -j1 ; make install
export P3DFFT_ROOT=$PWD
echo $P3DFFT_ROOT
On success, you should find in the directory ${HOME}/src/p3dfft-dist.2.7.1-openmpi/
./include:
p3dfft.h p3dfft.mod
./lib:
libp3dfft.a
Please add then to your basrc the location of p3dfft:
export P3DFFT_ROOT=...
I did not manage to use any of the preinstalled hdf5 versions, so I had to install my own.
wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.15-patch1.tar.gz
tar vxzf hdf5-1.8.15-patch1.tar.gz
cd hdf5-1.8.15-patch1
CC=mpicc FC=mpif90 ./configure --prefix=$PWD --enable-fortran --enable-parallel
make clean; make -j12 ; make install
export HDF_ROOT=$PWD
echo HDF_ROOT
now besides the usual ensuring HDF_ROOT to be correct, we have to add the path for the linker.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HDF_ROOT/lib
if you do not do that, you get runtime errors.
Then you can go ahead and clone flusi and make it
Create a test directory run1 and run2 Create therein the file exeflusi.sh with the following content:
#!/bin/bash
ulimit -s unlimited
./flusi "$@"
Create the jobfile:
#!/bin/bash
#OAR -n FLUSI
#OAR -p cluster
#OAR -l nodes=1, walltime=00:30:00
#OAR -O FLUSI.%jobid%.output
#OAR -E FLUSI.%jobid%.ERROR
#cat $OAR_NODEFILE
module load openmpi/intel/1.6.3
module unload intel/12.1 intel/13.0
module load intel/13.1.2
CPUS=$(wc -l ${OAR_NODEFILE} | awk '{print $1}')
echo we run on ${CPUS} CPUs
mpirun -n "${CPUS}" -machinefile "${OAR_NODEFILE}" --bind-to-core ./exeflusi.sh test1.ini
Create a parameter file:
; Define the resolution
[Resolution]
nx=384;
ny=384;
nz=384;
[Time]
; final time
Tmax=1.0;
; maximum number of time steps. if not set, unlimited
nt=100;
; Time stepper for Fluid
iTimeMethodFluid=AB2; RK2, RK4, FSI_AB2_iteration, FSI_AB2_staggered, FSI_AB2_semiimplicit
CFL=0.1;
[ReynoldsNumber]
nu=1.0e-2; nu is the inverse of Re if properly scaled
[InitialCondition]
inicond=turbulence;
nu_smoothing=1.0e-5;
omega1=200.0;
[Penalization]
; use penalization method or not? (superseeds following parameters)
iPenalization=1;
; does the obstacle move?
iMoving=1;
; which mask to set? (sphere, Flapper, Insect, noncircular_cylinder)
; note noncircular_cylinder reads in shape.in
iMask=sphere;
; penalization parameter
eps=1.0e-3;
[Geometry]
; domain size
xl=3.0; new: default value is 2*pi
yl=3.0;
zl=3.0;
Size=1.0; generic size parameter
alpha=0.0; generic angle parameter
; Position of obstacle (not used for insects)
x0=2.0;
y0=2.0;
z0=2.0;
[MeanFlow]
iMeanFlow_x=fixed;
iMeanFlow_y=fixed;
iMeanFlow_z=fixed;
ux=0.0;
uy=0.0;
uz=0.0;
[Saving]
; Time interval for saving fields
tsave=0.05;
; Time interval for saving integral quantities
tintegral=0.01;
; Save drag every itdrag time steps
itdrag=10;
; save beam (if in use) every itbeam time steps
itbeam=2;
; make a backup or dont
iDoBackup=1;
; what fields to save?
iSaveVelocity=1;
iSaveMask=1;
or any other parameter file will do, as long as you include output and a sufficient number of points then run on one and two nodes and see if there is speedup by looking at timestep.t:
[ethomas@login run1]$ tail timestep.t
38 3.64200000E-02 9.90000000E-04 6.62180310E+00 5.68590498E+00
39 3.74100000E-02 9.90000000E-04 6.59797218E+00 5.69216609E+00
40 3.84000000E-02 9.90000000E-04 6.69983185E+00 1.06721330E+01
41 3.93900000E-02 9.90000000E-04 6.67511907E+00 5.68651605E+00
42 4.03800000E-02 9.90000000E-04 6.77047991E+00 1.06800492E+01
43 4.13700000E-02 9.90000000E-04 6.74532093E+00 5.68854904E+00
44 4.23600000E-02 9.90000000E-04 6.72130782E+00 5.68851900E+00
45 4.33500000E-02 9.90000000E-04 6.69831991E+00 5.68661714E+00
46 4.43400000E-02 9.90000000E-04 6.67627671E+00 5.68072391E+00
47 4.53300000E-02 9.90000000E-04 6.65526087E+00 5.68830299E+00
[ethomas@login run1]$ cd ../run2/
[ethomas@login run2]$ tail timestep.t
67 6.39400000E-02 9.90000000E-04 3.49401115E+00 2.95297503E+00
68 6.49300000E-02 9.90000000E-04 3.48601604E+00 2.95011997E+00
69 6.59200000E-02 9.90000000E-04 3.47822672E+00 2.94830298E+00
70 6.69100000E-02 9.90000000E-04 3.50671630E+00 5.47228003E+00
71 6.79000000E-02 9.90000000E-04 3.49886582E+00 2.94923496E+00
72 6.88900000E-02 9.90000000E-04 3.49128656E+00 2.95293903E+00
73 6.98800000E-02 9.90000000E-04 3.48390081E+00 2.95191312E+00
74 7.08700000E-02 9.90000000E-04 3.51084581E+00 5.47760892E+00
75 7.18600000E-02 9.90000000E-04 3.50328751E+00 2.94388413E+00
76 7.28500000E-02 9.90000000E-04 3.49596621E+00 2.94664788E+00
in this case, there is speedup (5.68830299E+00 vs 2.94664788E+00 for short time steps (i.e. without forces etc)