From - Tue Jul 30 17:18:13 2002 Return-path: Received: from mailserver2.Princeton.EDU (imap2.Princeton.EDU [128.112.128.25]) by imap2.Princeton.EDU (iPlanet Messaging Server 5.2 (built Feb 21 2002)) with SMTP id <0H02005FGI8YZR@imap2.Princeton.EDU> for hammett@ims-ms-daemon (ORCPT hammett@phoenix.princeton.edu); Tue, 30 Jul 2002 11:12:36 -0400 (EDT) Received: from smtpserver2.Princeton.EDU ([128.112.129.148]) by mailserver2.Princeton.EDU (NAVGW 2.5.2.12) with SMTP id M2002073011123527551 for ; Tue, 30 Jul 2002 11:12:35 -0400 Received: from Princeton.EDU (postoffice.Princeton.EDU [128.112.129.120]) by smtpserver2.Princeton.EDU (8.12.2/8.12.2) with ESMTP id g6UFCP2j022937 for ; Tue, 30 Jul 2002 11:12:34 -0400 (EDT) Received: from kendall.umd.edu (kendall.umd.edu [129.2.106.12]) by Princeton.EDU (8.12.3/8.12.3) with ESMTP id g6UFBVt5021514 for ; Tue, 30 Jul 2002 11:11:31 -0400 (EDT) Received: (from bdorland@localhost) by kendall.umd.edu (8.11.6/8.11.6) id g6UErCv00634; Tue, 30 Jul 2002 10:53:12 -0400 Date: Tue, 30 Jul 2002 10:53:12 -0400 From: Bill Dorland Subject: Summer upgrades to GS2 ready To: dmikkelsen@Princeton.EDU, mredi@Princeton.EDU, dwross@mail.utexas.edu, rbudny@Princeton.EDU, dernst@psfc.mit.edu, ghammett@Princeton.EDU, ebelli@Princeton.EDU, clarisse.bourdelle@cea.fr, vetter3@llnl.gov, fsj@ipp.mpg.de, colin.m.roach@ukaea.org.uk, nevins@llnl.gov Reply-to: bdorland@ipr.umd.edu Message-id: <200207301453.g6UErCv00634@kendall.umd.edu> Greetings, I've just finished a round of upgrades to GS2 which should make running at NERSC on the IBM SP2 (seaborg.nersc.gov) easier. This email has some key info for getting started there. First, I recommend getting the latest sources from the website http://kendall.umd.edu/~bdorland/GS2/info.html and recompiling on seaborg. This should be straightforward. Some people have reported unusual compilation errors which are evidently related to some mix of modules that I'm not familiar with. I have the following lines in my .login.ext file (I use tcsh): module load GNU emacs netcdf totalview sptools gnuplot setenv TMPDIR=$SCRATCH The first takes care of some things necessary for compilation as well as some things I use from time to time. You do not need emacs or gnuplot to run the code, for example. The second line is included at the request of the NERSC consultants. It gets in the way during compilation, unfortunately, so I have to type something like % unsetenv TMPDIR % make % setenv TMPDIR $SCRATCH to compile and run without generating comments from the consultants. Compilation should be very infrequent, so it is easiest to set TMPDIR to $SCRATCH at login routinely. You may notice that there is a new executable produced in the src directory: ingen. This is a code which reads an input file, looks for problems, makes recommendations, reports errors, and produces an input file with the defaults made explicit. To run it on an input file called runname.in, do something like this to get set up: % pwd .../gs2/src % cd .. % mkdir runs % cd runs % ln -sf ../src/ingen . and then do this on a routine basis: % pwd .../gs2/runs % ls gs2 runname.in % ./ingen runname % ls runname* runname.error runname.in runname.report [To run on seaborg, you may need to invoke poe, along the lines of: % poe ./ingen runname -tasks_per_node 1 -nodes 1 or possibly % poe ./ingen runname end comment.] The file called runname.report is the one you want. Have a look, and please provide some feedback. I produced it in a hurry, and there may be errors in the report, particularly for choices of inputs that I do not use very often. You should be able to run ingen successfully in any directory that gs2 could run in. That is, any auxiliary files that are needed would be needed by an actual run. It actually requires all auxiliary files to be present, as it checks to see if they are self-consistent. I've paid careful attention to the error reporting and to the estimates of performance, and would especially like to know about problems in those areas. In the short run, I plan to extend ingen to produce related input files from a template, to check for MHD ballooning stability, and to do some other easy tasks up front. If you can think of anything similar that you'd like to see, please let me know. In the long run, I plan to make ingen the principal mechanism for providing documentation for GS2. I've tried to bring the website up to date again as well. The sample runs are slightly different and work with the latest executable. There are new comments on the help.html page, mostly about choosing the number of energy grid points and choosing the number of processors and nodes on seaborg. I've also posted a sample Load Leveler script. Before the summer is out, I'm planning to post a sample nonlinear run, with input, scripts, reports, timings and output. I suppose this will be a Cyclone benchmark case unless someone has a better suggestion. In general, the code is much more flexible now as regards the way the data is laid out for parallel computations. However, I've looked into many different possibilities, and have the following basic recommendations. To get the best performance for a linear run, I recommend choosing the number of nodes equal to the number of species, tasks_per_node = 16, and negrid = 16. You can use more nodes profitably, but for routine linear studies this algorithm should provide enough horsepower. It has the advantage that it is easy to remember and use. [To increase the number of nodes beyond this point, make sure that the number of ky's is an integer multiple of the number of nodes.] To get the best performance for a nonlinear run, I recommend working from the following baseline: number of nodes = number of species * number of energy grid points, and number of tasks per node an integral divisor of the number of ky modes. The number of ky's corresponding to any given value of ny (the actual input) is in the output of ingen, and hasn't changed from the earlier algorithm (for those of your that knew it). A typical case might have ny=48, corresponding to 16 ky's, which all fit neatly on a single seaborg node. This is somewhat larger than the usual runs we've done in the past, I think. Assuming good choices for the numbers of nodes and tasks, the performance of nonlinear runs on seaborg is quite good. The scaling is closer to linear than it was on mcurie. The detailed formula is in ingen, but goes like Time/timestep ~ 1/N_PE**0.95 Perfect scalability would be 1/N_PE. Here, N_PE = number of nodes * number of tasks per node. For reference, I find that a typical run that I would do on approximately 500 processors on seaborg or mcurie runs about 2.5-3 times faster on seaborg. The one thing to watch out for on this machine is that there is still no facility to check how much time remains available to a job running under Load Leveler at NERSC. Because of this, a job that would run longer than the allowed time will be killed without warning. So you should work up to large numbers of timesteps slowly, after you've developed some idea of how long runs take. The time a nonlinear run will take cannot be predicted in advance, because it is a strongly increasing function of the number of times that the timestep changes. (This only happens in nonlinear runs.) In general, a restart costs less than the initial initializations, but at least as much as ntheta*nfields timesteps, where nfields is 1, 2, or 3, depending on how many of Phi, A_par and A_perp are included in the calculation. Later this year, I'm going to try to automate GS2 runs to a high degree. If you are interested in that kind of thing, please let me know. The idea will be to make linear parameter scans simply a matter of providing a template and some simple directions. I'm going to be on vacation for the next week, so if you find problems and don't hear back from me quickly, that's why. I'll be back online around Aug 10. Until then, --Bill