What to do with a TORIC crash in TRANSP ======================================= 0. Background ------------- TORIC [1] is the full wave ICRF solver by the famous RF physicist Marco Brambilla of IPP/Garching, and his team. TORIC is the workhorse code for analysis of ICRF shots of CMOD within TRANSP, and, there is interest in applying it to other analysis and simulation jobs as well. For several years, TRANSP has run TORIC3, an old version, which used a simplified geometry (updown symmetric, midplane radius and shift, elongation, triangularity to characterize flux surfaces). The current development version is TORIC5. It uses a full numerical equilibrium. However, it is fairly new, and (as of Dec. 2005) it still has problems coping with some numerical equilibria provided to it from experimental data. [1] Plasma Phys. Contr. Fusion 41 (1999) 1. 1. Which version of TORIC is in use... -------------------------------------- TRANSP concurrently supports both TORIC3 and TORIC5. The choice between the codes is controlled from the namelist: NICRF=6 ! TORIC3, simplified geometry (for testing only) NICRF=7 ! TORIC5, simplified geometry (for testing only) NICRF=8 ! TORIC5, full numerical equilibrium (**recommended**) NICRF=9 ! TORIC5, full numerical equilibrium, lower resolution allowed. It is useful to know this setting when setting out to debug a TORIC problem. The balance of this document will be concerned only with the newer code, TORIC5. 2. The TORIC5 source code & separate svn repository --------------------------------------------------- (This section modified 10 Dec 2007 -- D. McCune) Note: The TORIC5 source has been moved; the MIT source repository used in 2006 is defunct. There is now an svn repository at IPP/Garching. To get permission for access, please contact Roberto Bilato rbb@ipp.mpg.de and explain your need for access. If it is in connection with TRANSP code development, cc dmccune@pppl.gov (Doug McCune). When permission is granted, you can obtain a password from David Coster, David.Coster@ipp.mpg.de. You will then be able to check out a copy of the TORIC5 "sandbox". This sandbox will have a trunk -- the main line of development -- and branches for those other than the main authors of the code (i.e.the rest of us). Changes we develope are to be committed into the branches, and will be reviewed and integrated into the trunk line by the lead authors: Roberto Bilato, and Marco Brambilla (semi-retired). src -- the toric5 source src_transp -- TRANSP-specific toric5 add-ons and notes The file src_transp/dmc_update_procedure.txt describes the steps involved in updating the TRANSP copy of TORIC5 from the svn repository. This includes such steps as running `fgtok` to check for and/or convert precision of floating point constants. ANY CHANGES MADE TO THE TRANSP SOURCE must be transcribed and committed back into the svn repository! Detailed description of the changes should be provided. An example of such an explanation for a commit of changes in December 2007 can be seen below, in the appendix. 3. Related libraries in TRANSP ------------------------------ These are true TRANSP libraries residing in the TRANSP svn source repository: source/fppmod -- the Fokker Planck calculation, i.e. numerical model of the ICRF minority heated fast ion distribution on an explicit velocity space grid-- bounce averaged, reduced to 1d in real space; gyro averaged and reduced to 2d in velocity space, 3d altogether. This code is slated to be replaced by CQL3D, a more widely used and better validated Fokker Planck solver in the plasma physics community. source/rfxqlo -- a library for interfacing the Fokker Planck code to any ICRF wave solver. All the TRANSP RF solvers (TORIC3, TORIC5, SPRUCE and DSPRUCE) use this interface; however, it is not clear whether/how it will be used with CQL3D; these are issues for the future. source/toric_iface -- a library that interfaces specifically to TORIC5, writing namelists readable by the TORIC5 driver subroutine "t4_torica". 4. TORIC5 input and output file names within a TRANSP run --------------------------------------------------------- As always, executing or crashed runs on a TRANSP server or development platform are characterized by a runid (e.g. "2202" or "37065Z10"), and a tokamak id (e.g. "CMOD" or "TFTR"). A crashed run's data will generally be located in files * in directory / where depends on the machine configuration. But, there is a subdirectory _icrf set aside specifically for data files related to fast ion calculations supporting a TRANSP run. The TORIC5 input and output files will be located in the _icrf subdirectory. For example: if is CMOD and is 2202, the file path is $WORKDIR/CMOD/2202_icrf # on most development machines but of course the production server "pshare" is a bit different; the path will be under a pshr#### directory owned by a particular TRANSP user, in $RESULTDIR/CMOD/2202/2202/2202_icrf (the redundant runid subdirectories being an apparent artifact of the PBS job queueing software currently in use -- as of Dec. 2005). From now on, the leading path up to but not including the _icrf subdirectory will be dropped. TORIC5 runs allow a variable number of toroidal field components or modes to be computed on each antenna. Each mode is characterized by a toroidal mode number Nphi-- a non-zero integer. If a symmetric spectrum is used, then there is assumed to be equal power in each member of each (+Nphi,-Nphi) pair; the field component for +Nphi only has to be computed; the -Nphi component is then inferred from the symmetry assumption. However, the user can also specify a non-symmetric spectrum, in which case each Nphi mode must be computed as a separate wavefield component (a separate TORIC5 call) regardless of sign. Because of the computational cost of TORIC5, most users try hard to get by with a fairly simple spectrum with only a few "representive" modes present per antenna. The total number of TORIC5 calls per timestep is the sum of the number of nodes over all RF antennas. All calls are executed with the same target plasma parameters, temperatures and densities as specified to TORIC5 in the ascii file _icrf/_profnt.dat which is written and updated by the Fokker Planck code (even within a single call to the Fokker Planck code this can be rewritten as the ICRF resonant fast "minority" average energy evolves). The numerical equilibrium is stored in _icrf/_equigs.dat Of course neither of these files vary with the antenna or Nphi mode under consideration. However, for each Antenna/Nphi combination, there is an additional file _icrf/_A__torica.inp which is a namelist that controls TORIC5 for this single mode. The namelist file contains path pointing to the profnt.dat and equigs.dat files. The important point here is that the above 3 files are sufficient to precisely reproduce a single TORIC5 calculation using a standalone code, i.e. independent of the whole of TRANSP. When TORIC5 runs inside TRANSP, it produces 2 output files per Antenna/Nphi combination: _icrf/_A__toric4.msgs and _icrf/_A_.fppdata the former is the message output of TORIC5, often quite informative with errors and warnings clearly reported; the latter is the output data, a wave field description for the specified mode that can be used by the Fokker Planck code (fppmod via rfxqlo) in constructing a quasilinear operator (QLO). Note: if Nphi < 0 the encoding in the filename will be "m" followed by |Nphi|. I.e. Nphi=7 --> "7", Nphi=11 --> "11", Nphi = -8 --> "m8". 5. Reproducing a TRANSP TORIC5 crash in the TORICA standalone driver --------------------------------------------------------------------- This will be given as a recipe: a) Create an empty working directory "toric5" somewhere. b) From TRANSP's _icrf subdirectory copy the files *_profnt.dat, *_equigs.dat, and *_torica.inp into this working directory. c) For each _torica.inp file of interest: execute TORIC5: > toric_main _A__torica.inp In a recent real crash (Dec. 2005) this looked like: > toric_main 2187_A2_11_torica.inp For mpi runs: a) reserve nodes with sufficient mememory and time e.g. qsub -I -X -l walltime=72:00:00 -l nodes=16 -l mem=30000mb -q fielder b) run mpi_toric_main e.g. $MPI_CMD -np 16 $LOCAL/exe/mpi_toric_main _A__torica.inp 6. Packaging a reproducible standalone crash for referral to RF experts ------------------------------------------------------------------------ Continuing the above recipe... d) "cd .." to the parent of the "toric5" working directory. e) "tar -cf toric5.tar toric5" -- tar up the directory. f) "gzip toric5.tar" g) "cp toric5.tar.gz /u/ftp/pub/dmccune/toric5" (of course another filename or anonymous ftp subdirectory could be chosen). h) The .tar.gz file is downloadable from offsite via anonymous ftp at ftp://ftp.pppl.gov:/pub/dmccune/toric5 j) Email the toric5 experts... (probably best to coordinate with D. McCune as he is in contact with all these cats) Marco Brambilla at IPP/Garching -- mab@ipp.mpg.de --------------- --------------- --------------- Appendix: Dec 2007 commit notes (D. McCune email) Irina, Jim, John, Roberto, Based on my testing with the CMod H minority time slice + examination of the source, there are really only just 4 changes to toric5_2 needed to match our results with the earlier version of toric4/5 that has been operating in TRANSP for the past 1.5 years or so. Note that I have worked with the TRANSP "toric5_2" library source committed by Jim. I am working to make this something I can commit into the "McCune" branch of the TORIC5 svn repository, with the minimal possible set of changes for working code. So: I removed some "cosmetic" changes related to version labeling and the like-- would prefer to leave this up to the TORIC5 authors (however Jim could commit such changes into a separate branch of the repository, if he chooses). Also, the issue of the "linpack" name conflict detected by Jim: this is important, especially in the cases where the argument lists are changed. If this is done, the names of the modified routines (and corresponding calls to them) must be changed as well, to avoid a name conflict and subsequent runtime failure when linking the code into a larger integrated model (which likely contains other physics packages which use linpack with unmodified argument lists). Still, the sources affected by this are: t4_mod_ssfpql.F, t4_aamain.F (because it uses t4_mod_ssfpql.F), t4_linpkc.F and t4_linpkr.F. TRANSP does not need any of these, not even t4_aamain because it has its own driver. So, I prefer again to leave the longer term fix of this issue to the TORIC5 authors, and simply omit these sources from the TRANSP installed version of TORIC5 for now. The 4 remaining "irreducible" changes are: New "src_transp" routine: t4_toricmode.F -- very simple routine allows TRANSP driver to set the toric "mode", avoiding the need to move the new namelist "toric_mode" in t4_aamain.F which TRANSP does not use. This has no effect on non-TRANSP uses of TORIC and is outside the main src directory. The 3 remaining changes do affect codes in the main TORIC5 source directory "src": The subroutine t4_mod_profnt.F -- handle TRANSP "Rfxqlo" profnt file as a variant ascii file format. No namelist switch is added; instead this logic is triggered by reading the labeling in the first line of the profnt file. This can be done safely without affecting non-TRANSP uses of TORIC, along as the non-TRANSP users do not write "Rfxqlo_Pro" as the first 10 characters in their profnt data files. This is the method Jim had coded, but I simplified the coding somewhat, borrowing a few lines from our earlier version of TORIC in TRANSP. I note that this part of the code will have to be revisited; extensive I/O changes would be expected if non-Maxwellian targets are to be provided as input. It might be better to use NetCDF than ascii for these inputs in the long run, as maintaining backwards compatibility while using formatted ascii files is very tricky, fragile business (as John noted in our meeting at the APS). The subroutine t4_mod_tofppmod.F -- corrected normalization of TORIC5 output as used by TRANSP, found and carried out by Jim Conboy. I don't think this will affect non-TRANSP uses of TORIC, because TRANSP is the only user of the t4_mod_tofppmod module, I believe. With these changes alone, we get power match to the earlier version TORIC4/5 in our CMod H-minority test, but the driven current is ~25% higher. So, per Jim Conboy's results, this can be fixed by the 4th change to the code, working on the normalization factors in t4_mod_powtwd.F. This recovers the behavior from the earlier version, but I am not sure if the earlier version or the current version is better from a physics standpoint. The change to this routine is the ONLY one that is likely to affect non- TRANSP TORIC5 users, and it needs to be carefully reviewed by TORIC5 authors. I am committing the code as provided by Jim, into the McCune branch. On the TRANSP side, I've svn committed a script $CODESYSDIR/csh/toric5_cmp which generates a compact file-level summary of changes in the TRANSP toric5 library source relative to an svn-checked-out copy of the TORIC5 source code; current output is copied below. My changes are also in the TRANSP svn committed toric5_2 source. Thanks to everyone for their help! I go now to follow John's instructions for placing TORIC5 sources in the McCune svn branch... --------------------Doug PS I attach recent email messages on the TORIC5 normalization issues found by Jim. ---------------- toric5_cmp: comparing TRANSP TORIC5 source (*.for) to TORIC5 svn repository copy (*.F): assert_mod.for ...same blacs_mod.for ...same mod_netcdf_tools.for ...same ptri_module.for ...same stri_module.for ...same t4_bdcond.for ...same t4_dieltn.for ...same t4_elepwr.for ...same t4_fftl.for ...same t4_get_tmpluns.for ...same t4_gloini.for ...same (t4_lunset.F in src_transp) t4_lunset.for ...same t4_master.for ...same t4_mblock.for ...same t4_metric.for ...same t4_mod_diagns.for ...same t4_mod_direl.for ...same t4_mod_equil.for ...same t4_mod_fields.for ...same t4_mod_ibweld.for ...same t4_mod_interp.for ...same t4_mod_kinddef.for ...same t4_mod_master.for ...same t4_mod_matini.for ...same t4_mod_ncdf.for ...same t4_mod_powtwd.for ...DIFFERENT t4_mod_profnt.for ...DIFFERENT t4_mod_public.for ...same t4_mod_qldce.for ...same t4_mod_qlminh.for ...same t4_mod_rescof.for ...same t4_mod_rwdisc.for ...same t4_mod_sumnph.for ...same t4_mod_tofppmod.for ...DIFFERENT t4_mod_toi2mex.for ...same t4_mod_tprof.for ...same t4_mod_zetbes.for ...same t4_operev.for ...same t4_opevac.for ...same t4_outplo.for ...same t4_output.for ...same t4_parallel_mod.for ...same t4_rotdiv.for ...same t4_scalapack_mod.for ...same t4_surfpl.for ...same (t4_tofpp.F in src_transp) t4_tofpp.for ...same t4_torica.for ...same t4_toricmode.for ...NOT FOUND (t4_tprof.F in src_transp) t4_tprof.for ...same toric5_cmp summary: 4 differences, 0 errors.