Welcome to the forums. Please post in English or French.

You are not logged in.

#1 2011-02-18 03:53:21

ppeetteerr
Member
Registered: 2009-06-30
Posts: 234

[How to] Mpi Parallel Code Aster, acml, parmetis

After some work I believe I have repeatable steps for installing a Parallel version of Code-Aster. If someone can verify that this works then I'll add it to the wiki. However, we have far to many pages that don't work there. Starting from a fresh install of Ubuntu 10.04

1st start by downloading acml-4-4-0-gfortran-64bit.tgz, aster-full-src-10.3.0-3.noarch.tar.gz, ParMetis-3.1.1.tar.gz, mpich2-1.3.2.tar.gz to ~/Install_Files

then install the necessary programs

sudo apt-get install gfortran g++ python-dev python-numpy zlib1g-dev libatlas-base-dev grace bison flex python-qt4 checkinstall

Prepare the installation directory, extract and install acml libs, extract and configure code aster, run code-aster install script

sudo chown peter /opt/
#Change peter to your username
#install acml
cd ~/Install_Files/
mkdir ./acml
cp acml-4-4-0-gfortran-64bit.tgz ./acml
cd /opt/acml
tar xfvz acml-4-4-0-gfortran-64bit.tgz
./install-acml-4-4-0-gfortran-64bit.sh -accept -installdir=/opt/acml4.4.0
echo /opt/acml4.4.0/gfortran64/lib | sudo tee -a /etc/ld.so.conf.d/amd_math.conf
sudo ldconfig
cd ~/Install_Files
tar xfvz aster-full-src-10.3.0-3.noarch.tar.gz
cd aster-full-src-10.3.0/
#replace PREFER_COMPILER = 'GNU' with PREFER_COMPILER = 'GNU_without_MATH'
#add MATHLIB= '-L/opt/acml4.4.0/gfortran64/lib -lacml'
sed -i "s:PREFER_COMPILER\ =\ 'GNU':PREFER_COMPILER\ =\ 'GNU_without_MATH'\nMATHLIB=\ '-L/opt/acml4.4.0/gfortran64/lib -lacml':g" setup.cfg
#python setup.py test
python setup.py install

Accept Code-Aster installation, Then setup mpich2 using mpd and create and install via checkinstall for easy handeling

#install mpich2
cd ~/Install_Files
cp mpich2-1.3.2.tar.gz /opt/
cd /opt
tar xfvz mpich2-1.3.2.tar.gz 
cd mpich2-1.3.2
./configure --with-pm=mpd
make
#Create mpich2 package
sudo checkinstall -D make install
# Accept Defaults, This make the include directory /usr/local/include/ and lib /usr/local/lib

Accept all defaults [Enter] in the packaging system, Then Setup .mpd.conf, install scalapack using the acml libs, Extract ParMetis

cd ~/
echo MPD_SECRETWORD=mr-secret > .mpd.conf
chmod 600 .mpd.conf
echo CSTM4500 >> /opt/aster/etc/codeaster/aster-mpihosts
#install blacs and scalapack
cd ~/Install_Files
tar xfvz scalapack_installer.tgz
cd scalapack_installer_0.96/
./setup.py  --blaslib=/opt/acml4.4.0/gfortran64/lib/libacml.a --downblacs --lapacklib=/opt/acml4.4.0/gfortran64/lib/libacml.a --mpicc=mpicc --mpif77=mpif77 --mpiincdir=/usr/local/include/ --prefix=/opt/scalapack --f90=/usr/bin/gfortran
#install ParMetis
cp ~/Install_Files/ParMetis-3.1.1.tar.gz /opt
cd /opt
tar xfvz ParMetis-3.1.1.tar.gz
cd ParMetis-3.1.1

After changing the Makefile.in to include the line 'XTRALIBS = -lmpich'

make
cd /opt/aster
cp -ax STA10.3 PAR10.3
cp ~/Install_Files/aster-full-src-10.3.0/SRC/mumps-4.9.2.tar.gz /opt/
cd /opt
tar xfvz mumps-4.9.2.tar.gz
mv mumps-4.9.2 mumps-4.9.2_mpi
cd mumps-4.9.2_mpi/

edit Makefile.inc see below.

make all

edit asrun to fit your number of processor
edit profile.sh (see below)
edit config.txt (see below)

cd /opt/aster/PAR10.3
../bin/as_run --vers=PAR10.3 --make clean
../bin/as_run --vers=PAR10.3 --make clean bibf90/mumps
../bin/as_run --vers=PAR10.3 --make

And that should be everything. You will have to start mpd by hand or make the appropriate edit to asrun

Changes to mumps-4.9.2_mpi Makefile.inc

diff Makefile.inc /opt/mumps-4.9.2_mpi/Makefile.inc
39c39,40
< LMETISDIR = ?HOME_METIS?/lib
---
> #LMETISDIR = /opt/aster/public/metis-4.0/lib
> LMETISDIR = /opt/ParMetis-3.1.1
48,49c49,50
< LMETIS    = -L$(LMETISDIR) -lmetis
< #LMETIS    = -L$(LMETISDIR) -lparmetis -lmetis
---
> #LMETIS    = -L$(LMETISDIR) -lmetis
> LMETIS    = -L$(LMETISDIR) -lparmetis -lmetis
54c55,56
< ORDERINGSF  = -Dpord -Dmetis
---
> #ORDERINGSF  = -Dpord -Dmetis
> ORDERINGSF  = -Dpord -Dparmetis -Dmetis
67,69c69,71
< CC      = ?CC?
< FC      = ?F90?
< FL      = ?LD?
---
> CC      = mpicc
> FC      = mpif90
> FL      = mpif90
72a75,85
> 
> SCALAP  = /opt/scalapack/lib/libscalapack.a \
>         /opt/scalapack/lib/blacs.a \
>         /opt/scalapack/lib/blacsF77.a \
>         /opt/scalapack/lib/blacs.a \
>         /opt/acml4.4.0/gfortran64/lib/libacml.a
> 
> 
> INCPAR = -I/usr/local/include $(IPORD)
> LIBPAR = $(SCALAP)  -L/usr/local/lib -lmpich
> 
77c90
< LIBBLAS = ?MATHLIB?
---
> LIBBLAS = -L/opt/acml4.4.0/gfortran64/lib/libacml.a
83,85c96,98
< OPTF    = -O -Dintel_ -DALLOW_NON_INIT ?F90FLAGS?
< OPTL    = -O ?LDFLAGS?
< OPTC    = -O ?CFLAGS?
---
> OPTF    = -O -DALLOW_NON_INIT 
> OPTL    = -O 
> OPTC    = -O 
87,89c100,102
< INC = $(INCSEQ)
< LIB = $(LIBSEQ)
< LIBSEQNEEDED = libseqneeded
---
> INC = $(INCPAR)
> LIB = $(LIBPAR)
> LIBSEQNEEDED =

Changes to Profile.sh

/opt/aster$ diff PAR10.3/profile.sh STA10.3/profile.sh 
7c7
< PATH=/opt/ParMetis-3.1.1:\
---
> PATH=/opt/aster/public/metis-edf-4.1:\
19c19
< /opt/mumps-4.9.2_mpi/lib:\
---
> /opt/aster/public/mumps-4.9.2/lib:\
42c42

Changes to config.txt

diff PAR10.3/config.txt STA10.3/config.txt 
46,48c46
< BIBL           | mumps   | ?     | -L/opt/mumps-4.9.2_mpi/lib -ldmumps -lzmumps -lsmumps -lcmumps -lmumps_common -lpord 
< BIBL           | scalap  | 1.8.0 | /opt/scalapack/lib/libscalapack.a /opt/scalapack/lib/blacs.a /opt/scalapack/lib/blacsF77.a /opt/scalapack/lib/blacs.a /opt/acml4.4.0/gfortran64/lib/libacml.a
< BIBL         | metis   | ?     | -L/opt/ParMetis-3.1.1 -lparmetis -lmetis
---
> BIBL           | mumps   | ?     | /opt/aster/public/mumps-4.9.2/lib/libdmumps.a /opt/aster/public/mumps-4.9.2/lib/libzmumps.a /opt/aster/public/mumps-4.9.2/lib/libsmumps.a /opt/aster/public/mumps-4.9.2/lib/libcmumps.a /opt/aster/public/mumps-4.9.2/lib/libmumps_common.a /opt/aster/public/mumps-4.9.2/lib/libpord.a /opt/aster/public/mumps-4.9.2/lib/libmpiseq.a -L/opt/aster/public/metis-4.0/lib -lmetis
50,51c48,49
< BIBL           | math    | ?     | /opt/acml4.4.0/gfortran64/lib/libacml.a
< BIBL           | mpi     | 2     | -L/usr/local/lib -lmpich
---
> BIBL           | math    | ?     | -L/opt/acml4.4.0/gfortran64/lib -lacml
> BIBL           | mpi     | 2     | 
53c51
< BIBL           | sys     | ?     | -Wl,--export-dynamic -lpthread -lz
---
> BIBL           | sys     | ?     | -Wl,--allow-multiple-definition -Wl,--export-dynamic -L/usr/lib -ldl -L/usr/lib -lutil -L/usr/lib -lm -L/usr/lib -lpthread -L/usr/lib -lz
55c53
< DEFS           | defined | ?     | LINUX64 _HAVE_MUMPS _USE_MPI _USE_MPI_MUMPS _DISABLE_SCOTCH
---
> DEFS           | defined | ?     | LINUX64 _HAVE_MUMPS
57c55
< #NOBUILD        | option  | ?     | bibfor/from_c
---
> NOBUILD        | option  | ?     | 
61,62c59,60
< LINK           | link    | ?     | mpif90
< OPTL           | link    | ?     | --verbose
---
> LINK           | link    | ?     | /usr/bin/gfortran
> OPTL           | link    | ?     | -fopenmp
64,71c62,69
< CC             | cc      | ?     | mpicc
< OPTC_D         | cc      | ?     | -c -g  -fno-stack-protector 
< OPTC_O         | cc      | ?     | -c -O2 -fno-stack-protector 
< INCL           | include | ?     | -I/opt/aster/PAR10.3/bibc/include -I/usr/include/python2.6 -I/usr/lib/python2.6/dist-packages/numpy/core/include -I/opt/aster/public/hdf5-1.6.9/include -I/opt/aster/public/scotch_4.0/include
< #
< F77            | f77     | ?     | mpif90
< OPTF_D         | f77     | ?     | -c -g -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
< OPTF_O         | f77     | ?     | -c -O2 -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
---
> CC             | cc      | ?     | /usr/bin/gcc
> OPTC_D         | cc      | ?     | -c -g  -fno-stack-protector -fPIC -fopenmp
> OPTC_O         | cc      | ?     | -c -O2 -fno-stack-protector -fPIC -fopenmp
> INCL           | include | ?     | -I/opt/aster/STA10.3/bibc/include -I/usr/include/python2.6 -I/usr/lib/python2.6/dist-packages/numpy/core/include -I/opt/aster/public/hdf5-1.6.9/include -I/opt/aster/public/scotch_4.0/include
> #
> F77            | f77     | ?     | /usr/bin/gfortran
> OPTF_D         | f77     | ?     | -c -g  -fPIC  -fopenmp  -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
> OPTF_O         | f77     | ?     | -c -O2 -fPIC  -fopenmp  -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
74,77c72,75
< F90            | f90     | ?     | mpif90
< OPTF90_D       | f90     | ?     | -c -g  -ffixed-line-length-0 -x f77-cpp-input -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
< OPTF90_O       | f90     | ?     | -c -O2 -ffixed-line-length-0 -x f77-cpp-input -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
< INCLF90        | include | ?     | -I/opt/aster/PAR10.3/bibf90/include_mumps-4.9.2_mpi
---
> F90            | f90     | ?     | /usr/bin/gfortran
> OPTF90_D       | f90     | ?     | -c -g  -ffixed-line-length-0 -x f77-cpp-input -fPIC  -fopenmp  -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
> OPTF90_O       | f90     | ?     | -c -O2 -ffixed-line-length-0 -x f77-cpp-input -fPIC  -fopenmp  -fdefault-double-8 -fdefault-integer-8 -fdefault-real-8
> INCLF90        | include | ?     |  -I/opt/aster/STA10.3/bibf90/include_mumps-4.9.2
115,116c113,114
< REPMAT         | exec    | 02-05 | /opt/aster/PAR10.3/materiau
< REPDEX         | exec    | 02-05 | /opt/aster/PAR10.3/datg
---
> REPMAT         | exec    | 02-05 | /opt/aster/STA10.3/materiau
> REPDEX         | exec    | 02-05 | /opt/aster/STA10.3/datg

Last edited by ppeetteerr (2011-02-18 03:55:20)

Offline

#2 2011-02-18 10:19:22

apalazzi
Member
Registered: 2010-05-11
Posts: 283

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

Hi,

just a few notes about your procedure... and when I'll get my new PC I'll test it on a fresh install - on Debian, however.

ppeetteerr wrote:

After some work I believe I have repeatable steps for installing a Parallel version of Code-Aster. If someone can verify that this works then I'll add it to the wiki. However, we have far to many pages that don't work there. Starting from a fresh install of Ubuntu 10.04

I agree with you, there are many pages, describing different procedures and many have problems in some steps... it would be nice to write a procedure as unique as possible - let alone that the best option would be to have a package shipped with the distribution.

1st start by downloading acml-4-4-0-gfortran-64bit.tgz, aster-full-src-10.3.0-3.noarch.tar.gz, ParMetis-3.1.1.tar.gz, mpich2-1.3.2.tar.gz to ~/Install_Files

I would rather use the distro's packages if available:
- on Debian, mpich2 is there, anyway the default mpi implementation is openmpi, so I rather use this, since AFAIK the performances are about the same.
- parmetis: there's the open source libscotchparmetis (that I currently use) and also the non-free libparmetis

If using the distro's packages, remember to install the -dev packages.

sudo chown peter /opt/
#Change peter to your username
#install acml
cd ~/Install_Files/
mkdir ./acml
cp acml-4-4-0-gfortran-64bit.tgz ./acml
cd /opt/acml
tar xfvz acml-4-4-0-gfortran-64bit.tgz
./install-acml-4-4-0-gfortran-64bit.sh -accept -installdir=/opt/acml4.4.0
echo /opt/acml4.4.0/gfortran64/lib | sudo tee -a /etc/ld.so.conf.d/amd_math.conf
sudo ldconfig

Why not provide the link to the acml page on the wiki? It's already there and well explained.

cd ~/Install_Files
tar xfvz aster-full-src-10.3.0-3.noarch.tar.gz
cd aster-full-src-10.3.0/
#replace PREFER_COMPILER = 'GNU' with PREFER_COMPILER = 'GNU_without_MATH'
#add MATHLIB= '-L/opt/acml4.4.0/gfortran64/lib -lacml'
sed -i "s:PREFER_COMPILER\ =\ 'GNU':PREFER_COMPILER\ =\ 'GNU_without_MATH'\nMATHLIB=\ '-L/opt/acml4.4.0/gfortran64/lib -lacml':g" setup.cfg

I would rather explain the modifications to do in the configuration files instead of using a sed command, that I think is unknown the average user.

Accept Code-Aster installation, Then setup mpich2 using mpd and create and install via checkinstall for easy handeling

See above: if you use the distro's packages, there's no need to compile anything, making life easier for everyone.

diff Makefile.inc /opt/mumps-4.9.2_mpi/Makefile.inc

I think universal diff are usually preferred:

diff -u Makefile.inc /opt/mumps-4.9.2_mpi/Makefile.inc

Anyway, as for the sed line, it's probably clear to include the whole config file.

Bye
Andrea

Offline

#3 2011-02-18 13:41:25

apalazzi
Member
Registered: 2010-05-11
Posts: 283

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

Another couple of observations...

first, it seems to me that your procedure relies on having already built and installed a serial version, isn't it? If so, it should be clearly stated in the beginning.

make
cd /opt/aster
cp -ax STA10.3 PAR10.3

Maybe the install dir for the mpi build should be STA10.3_mpi or something like this, since STA=stable, NEW=new (obviosly :-D ), and has nothing to do with serial, parallel or other building options.

< DEFS           | defined | ?     | LINUX64 _HAVE_MUMPS _USE_MPI _USE_MPI_MUMPS _DISABLE_SCOTCH

Why _DISABLE_SCOTCH ? And I usually enable also the parallel build for the FETI solver: _USE_MPI_FETI

Bye
Andrea

Offline

#4 2011-05-11 17:57:02

kathi
Member
Registered: 2009-09-24
Posts: 186

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

I've compiled CA with this HOWTO, thanks peter. But I used mpich2 of Ubuntu 10.04. So I ran my testcase with an error:

rank 3 in job 31  tom_lin_vm_52980   caused collective abort of all ranks
  exit status of rank 3: killed by signal 9 
rank 2 in job 31  tom_lin_vm_52980   caused collective abort of all ranks
  exit status of rank 2: killed by signal 9 
EXIT_COMMAND_23825_00000014=137
<INFO> Code_Aster run ended, diagnostic : <S>_ERROR

Have you any idea, where I can find more information about this error?

Furthermore: What should aster-mpihosts contain?

Thank you a lot!
Kathi

Last edited by kathi (2011-05-12 09:15:18)

Offline

#5 2011-05-12 09:15:32

kathi
Member
Registered: 2009-09-24
Posts: 186

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

And furtherfurthermore: in asrun:

mpirun_cmd : mpirun -np %(mpi_nbcpu)s --hostfile %(mpi_hostfile)s %(program)s

mpirun (mpiexec) has no option called "--hostfile". And the compile-process complains about that. What should we do instead?

Last edited by kathi (2011-05-12 09:17:35)

Offline

#6 2011-05-12 14:02:54

miko
Member
Registered: 2008-09-21
Posts: 114

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

Try this

mpirun_cmd : mpirun -np %(mpi_nbcpu)s  -hostfile %(mpi_hostfile)s %(program)s

miko

Offline

#7 2011-05-12 15:05:13

kathi
Member
Registered: 2009-09-24
Posts: 186

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

mpirun has no option -hostfile. It seems running well without (my CA with MPI Mumps works now)... but I don't understand, why asrun contains nonexistent options for mpirun?!

Offline

#8 2011-05-12 17:09:40

miko
Member
Registered: 2008-09-21
Posts: 114

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

Ok
Please try again control this by adjusting the path to mpiexec or mpirun and remplece  "-hostfile"  by  "-machinefile"

mpirun_cmd : /opt/mpi/mpich/bin/mpiexec  -machinefile %(mpi_hostfile)s -wdir %(wrkdir)s -n %(mpi_nbcpu)s %(program)s


This command work on my cluster

miko

Last edited by miko (2011-05-12 17:10:45)

Offline

#9 2011-05-12 18:21:19

kathi
Member
Registered: 2009-09-24
Posts: 186

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

hello miko,

how should be the syntax in this hostfile? peter only echoed something I don't know in this file.

thank you and greetings!
kathi

Offline

#10 2011-05-12 18:38:54

miko
Member
Registered: 2008-09-21
Posts: 114

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

hello kathi

Still in "asrun" after having defined "mpirun_cmd" mentioned earlier
set the path to your file "mpi_hostfile"
For example at home is:
"mpi_hostfile: / home / miko / aster.hosts"
and a cat / home / miko / aster.hosts gives

miko @ oscar: ~ $ cat / home / miko / aster.hosts
# Oscar
oscarnode-1
oscarnode-2
oscarnode-3
oscarnode-4
oscarnode-5
# oscarnode-1: 2 binding = user: 0.1
# oscarnode-2: 2 binding = user: 0.1
# oscarnode-3: 2 = user binding: 0.1
# oscarnode-4: 2 = user binding: 0.1
# compute-0-2


Hoping this helps you

miko

Last edited by miko (2011-05-12 18:39:37)

Offline

#11 2011-05-12 22:17:14

kathi
Member
Registered: 2009-09-24
Posts: 186

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

hey miko!

That helps alot. You wrote, you have a cluster. So your hostfile contains the hostnames of all nodes. My file only contains the hostname of my machine. Its less difficult, that i supposed.

Sadly my mpi mumps is slower that mult_front in my case with less than 6*8000 DOF. So perhaps I'll investigate in other possible optimisation methods.

regards and good night,
kathi

Offline

#12 2011-06-01 17:14:17

ppeetteerr
Member
Registered: 2009-06-30
Posts: 234

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

kathi wrote:

hey miko!

Sadly my mpi mumps is slower that mult_front in my case with less than 6*8000 DOF. So perhaps I'll investigate in other possible optimisation methods.

kathi

I've noticed in the past that you deal with contact related problems. Some contact algorithms are capable of using multi-processing while others are not. You may want to look into which algorithims you are using.

Although, I have also found that in several cases (Genetic Algorithms, or exploring the design space) it is much quicker to run multiple instances than one instance with multiple processors.

Offline

#13 2011-10-13 02:43:41

JMB365
Member
Registered: 2008-01-19
Posts: 778

Re: [How to] Mpi Parallel Code Aster, acml, parmetis

ppeetteerr wrote:

After some work I believe I have repeatable steps for installing a Parallel version of Code-Aster. If someone can verify that this works then I'll add it to the wiki.

Hello!  Thanks for the write-up !  In general it works for Ubuntu 11.04 (64bit) and CA_11.0.25.  However, I have yet to run a problem on it to verify.  Yes I agree there are pages in the Wiki that do NOT work.  Some of them initiated by me but never completed because I was not successful in the past.  I will delete them since they are useless, but I kept them there for that someday, when ... ... ...

Regards, JMB

Last edited by JMB365 (2011-10-13 02:48:58)


STA10.3/NEW11 Ubuntu 64 bit on multiple PCs

Offline