$WEB/code/hypre_on_jacquard
From jchen@pppl.gov Mon Nov  6 21:08:48 2006 -0500
Date: Mon, 6 Nov 2006 21:08:48 -0500 (EST)
From: Jin Chen 
To: Tony Drummond 
cc: Linda Sugiyama , "Joshua A. Breslau" , 
    Wonchull Park , "Stephen C. Jardin" , 
    Guo-Yong Fu , Hank Strauss , 
    acts-support@nersc.gov
Subject: Re: Hypre on Jacquard.

Hi Tony,

I have made a little program which shows hypre through petsc/2.3.1-opkgs_O 
gives errors when solving linear system on jacquard. The files are at

/u0/jinchen/tutorials

on Jacquard:

-rwxr-xr-x  1     29027 Nov  6 11:19 makefile
-rwxr-xr-x  1      9500 Nov  6 12:04 ex2.c
-rwxr-xr-x  1   6661531 Nov  6 12:04 ex2_jacquard
-rwxr-xr-x  1      6559 Nov  6 12:04 ex2_gmres_ilu.out_jacquard
-rwxr-xr-x  1     31223 Nov  6 12:08 ex2_cg_hypre.out_jacquard
-rwxr-xr-x  1      5362 Nov  6 12:16 ex2_cg_jacobi.out_jacquard
-rwxr-xr-x  1       520 Nov  6 12:39 bs.job_jacquard
-rwxr-xr-x  1      4741 Nov  6 13:21 ex2_cg_hypre.out_bassi
-rwxr-xr-x  1      4340 Nov  6 13:21 ex2_cg_jacobi.out_bassi
-rwxr-xr-x  1      5537 Nov  6 13:22 ex2_gmres_ilu.out_bassi
-rwxr-xr-x  1   8101960 Nov  6 13:22 ex2_bassi
-rwxr-xr-x  1       713 Nov  6 13:23 bs.job_bassi
-rwxr-xr-x  1      4854 Nov  6 13:37 ex2_cg_hypre.out_bgl
-rwxr-xr-x  1      4340 Nov  6 13:37 ex2_cg_jacobi.out_bgl
-rwxr-xr-x  1      5537 Nov  6 13:37 ex2_gmres_ilu.out_bgl
-rwxr-xr-x  1  15904426 Nov  6 13:38 ex2_bgl
-rwxr-xr-x  1       397 Nov  6 13:38 bs.job_bgl
-rwxr-xr-x  1      7052 Nov  6 14:07 ex2_gmres_ilu.out_seaborg
-rwxr-xr-x  1      6224 Nov  6 14:08 ex2_cg_hypre.out_seaborg
-rwxr-xr-x  1      5801 Nov  6 14:08 ex2_cg_jacobi.out_seaborg
-rwxr-xr-x  1   8020255 Nov  6 14:08 ex2_seaborg
-rwxr-xr-x  1       397 Nov  6 14:09 bs.job_seaborg
-rwxr-xr-x  1      5656 Nov  6 14:15 ex2_gmres_ilu.out_jaguar
-rwxr-xr-x  1      4434 Nov  6 14:15 ex2_cg_jacobi.out_jaguar
-rwxr-xr-x  1      4835 Nov  6 14:15 ex2_cg_hypre.out_jaguar
-rwxr-xr-x  1  24568384 Nov  6 14:19 ex2_jaguar
-rwxr-xr-x  1       698 Nov  6 14:20 bs.job_jaguar

The same code ex2.c has been run on 5 different machines: jacquard, bassi, 
bgl, seaborg, jaguar. Jaguar at ornl uses petsc-2.3.0. All other machines 
use petsc-2.3.1. Three solve options are switched:

1.gmres/ilu
2.cg/jacobi
3.cg/hypre

All solves by gmres/ilu and cg/jacobi generate the same correct results 
(ex2_gmres_ilu.out_*, ex2_cg_jacobi.out*) on all 5 machines. The solve by 
cg/hypre generats the same correct results on the 4 machine: bassi, bgl, 
seaborg, jaguar. But on Jacquard, the code got a runtime error inside 
hypre (ex2_cg_hypre.out_jacquard).

Can you please check?

Thanks,

-Jin-

On Mon, 25 Sep 2006, Tony Drummond wrote:

>
> Linda,
>
> I do recall your email, but I didn't get the details of the problem.
> My tests run fine but these are test linear systems.  Other users of
> the PETSc interface to call Hypre have not reported any problems,
> we have a computational chemistry code that does extensive use
> of it and are sitting next door from me at LBNL.  The other users
> normally contact us if they have problems as well.
>
>  That is why I had requested to either get a copy of your
> code, with instructions on how to compile and run it (I believe we did
> this exercise when you moved to 2.2.1) or if you prefer,  could you turn
> the the petsc profilers (ksp_monitor,
> true_error) in both of your codes (2.3.1 and 2.2.1) and send it to me 
> directly.
>
> The version of PETSc with Hypre enabled on
> Jaquard was only made available Friday, so it was not available
> in August. On jacquard, I had to use the latest
> version of Hypre which was not available long ago either.
>
> Let me know how you would like to proceed.
> -- Tony
>
> On Sep 25, 2006, at 6:03 AM, Linda Sugiyama wrote:
>
>> 
>> Tony,
>> 
>> Hypre is not working on Seaborg with Petsc 2.3.1, at least
>> the last time I checked in August.  Petsc 2.2.1 with hypre works well.
>> 
>> Also, there is a problem with hypre
>> in petsc 2.3.1 (and perhaps 2.3.0) that I have seen on all
>> machines where I tried it (seaborg,bassi,jaguar).  I submitted
>> a bug report to both nersc and ornl.
>> The answers using 2.3.1 differ from those of
>> petsc 2.2.1, and in my cases rapidly cause the code to crash, ie the growth 
>> rate blows up.  The 2.3.1 answers are the same on all machines.
>> 
>> Originally I thought that it was the jacobi preconditioner,
>> because that was the apparent difference in the input,
>> but then I found out that M3D does
>> not set the 'higher' levels of matrix solvers/preconditioners
>> from the 'lower' levels if they aren't specified in the input.
>> Instead, Petsc by default uses gmres/ilu.
>> (The old bug report is still valid and contains complete
>> information, but should be reinterpreted.)
>> 
>> Tony, could you check your test answers for 2.3.1 against petsc 2.2.1 on
>> seaborg?
>> 
>> Linda Sugiyama
>> 
>> On Fri, 22 Sep 2006, Stephen C. Jardin wrote:
>> 
>>> This is very timely.  HYPRE should now be available on Jacquard!!
>>> 
>>> -----Original Message-----
>>> From: Tony Drummond [mailto:LADrummond@lbl.gov]
>>> Sent: Friday, September 22, 2006 3:33 PM
>>> To: Stephen C. Jardin
>>> Cc: Francesca Verdier; support@nersc.gov; Stephane Ethier; Jin Chen; David
>>> Skinner; 'acts-support@nersc.gov'
>>> Subject: Hypre on Jacquard.
>>> 
>>> 
>>> 
>>> Steve,
>>> 
>>> I managed to fix the problem with Hypre on Jacquard.  So now Hypre is
>>> available
>>> on Jacquard (version 1.11.1b) thus PETSc 2.3.1 now links with it.  I
>>> have tested
>>> what i can so far and all looks to be working well (compared results
>>> with other
>>> tests on other platforms and ran all the petsc tests)
>>> 
>>> Please use the modules petsc/2.3.1-opkgs_O (optimized version) or
>>> petsc/2.3.1-opkgs (debugging version)
>>> 
>>> If you encounter problems please let us know (acts-support@nersc.gov)
>>> 
>>> Cheers,
>>> -- Tony
>>> On Sep 22, 2006, at 4:39 AM, Stephen C. Jardin wrote:
>>> 
>>>> Tony,
>>>> 
>>>> At your coaxing, Jin has tried this again on Seaborg and the full
>>>> program now
>>>> compiles and loads without error (using the same makefile)!  She is
>>>> waiting
>>>> in the queue for a test run and we will let you know the outcome.
>>>> 
>>>> What is your understanding regarding Jaquard?  Jin claims that the
>>>> -opgks
>>>> versions of PETSc (which contain Hyper) are not available on Jaquard.
>>>> Is
>>>> this correct?  If so, do you plan to install them?
>>>> 
>>>> Thanks for your efforts,
>>>> 
>>>> -steve
>>> 
>>> 
>>> 
> L. A. Drummond, (Tony)
> Staff Scientist
> Lawrence Berkeley National Laboratory
> One Cyclotron Road
> Berkeley, CA 94720