1) GENRAY Work GENRAY svn is in: /p/nubeam/genray-svn (source in: /p/nubeam/genray-svn/genray/trunk) [ modules required for building with pathscale can be loaded by sourcing pathscale-module_portal_pppl.csh ] GENRAY Preparatory steps: ------------------------- 1) GENRAY SVN is used for doing preliminary work on testing genray on template namelists. In addition to the above mentioned svn, in xshare, the directory /u/xshare/kumar/genray has a copy of genray from around September, 2010. This copy is used to build gfortran executables on sunfire02. There are many make files relevant to pppl. Makefiles listed with "ls -l *makefile*gfortran64*pppl*" and *makefile*pathscale*pppl* are important. To compile, use: gmake -f [ This is for serial code. There were some problems with the mpi version according to Kumar. See the mpi/ subdirectory for more details.] (a) As the authors of genray use gfortran, first compile genray with gfortran in a way similar to the authors (i.e. no floating point exception trapping) and run the regression tests. if anything fails, correspond with the authors. [ The regression tests are located in 00_Genray_Regression_Tests_*.tar.gz. There is a README file which describes the tests and how to run them. You will need to do a 'module load xdraw'. You can also run them individually by just typing xgenray in the test directories. The inputs are a genray.dat or genray.in file (the namelist; dat is the older version before unit conversion to "SI") and an equilibrium file in the geqdsk format whose name is in the namelist. The namelists are well commented.] (b) After (a) enable floating point exception trapping in the gfortran makefile, compile, and run the regression tests. To enable fpe trapping, set: CSPECIAL= -ffpe-trap=invalid,zero,overflow -fno-range-check -fdollar-ok -fno-common in the make file. If the regression tests fail, correspond with authors or fix the relevant code and convince the authors to fix it. [ gmake -f makefile_gfortran64_pppl CSPECIAL="-ffpe-trap=invalid,zero,overflow -fno-range-check -fdollar-ok -fno-common"] (c) Compile genray with pathscale compiler with fpe trapping enabled and run the tests. In my experience, this should usually proceed without difficulty once (a) and (b) have been accomplished. [ using makefile_pathscale_portal_pppl, executable called xgenray-path ] GENRAY in TRANSP: ----------------- GENRAY in transp consists of a library genray_lib and executables genray_transp, prepare_genray_input and process_genray_output. genray_lib/importing_genray.txt contains information on importing genray from the svn. (a) Contents of genray_lib: (i) All the *.i and *.h (include and header files) from the trunk (source) directory in genray svn (ii) All the source files listed in a latest version of makefile under SOURCES. The source files with .f need to be renamed as .for. (Notice that the genray source directory contains more source files than are used for building the genray library and executable. Thus it is necessary to use the makefile to copy the relevant files.) [ a skeleton script for doing this is in genray_lib/import_genray.py ] (iii) Two source files prepare_genray_input.f90 and process_genray_output.f90 from the IPS directory components/rf/genray/src. (IPS directory is a separate svn directory. Please contact Bob Harvey about this.) [ /p/nubeam/genray-svn/my_ips_trunk/components/rf/genray/src ] The files prepare_genray_input.f90, process_genray_output.f90, and genray.f are converted into subroutines prepare_genray_input_sub.for, process_genray_output_sub.for, and genray_transp_sub.for. [ there are a dozen or two differences between the executable and subroutine versions of these files ] (b) The executables genray_transp.f90, prepare_genray_input.f90, and process_genray_output.f90 are dummy routines that call prepare_genray_input_sub.for, process_genray_output_sub.for, and genray_transp_sub.for. (c) The subroutines prepare_genray_input.f90 and process_genray_output.f90 are works in progress inside transp. They are constantly being modified to load data from the plasma state file into the genray namelists. The actual data being loaded depends on the kind of heating/CD (ECH, LH, HHFW, etc.). Currently, it loads data and profiles for ECH. See heatlib/genrayech.f90 for how prepare_genray_input, genray_transp, and process_genray_output are invoked for ECH. [ Also relevant is the call made to trdatusub/copy_gennml() from trdat in order to set up the template namelist files appropriate to the tokamak and the mode of running genray. The template files are located in the $CODESYSDIR/source/nml directory. ] GENRAY and ECH -------------- The basic mechanism for invoking genray is already operational in transp and run successfully for ECH. In terms of the results, there are unresolved differences between GENRAY and TORAY results. Ron Prater of GA has some benchmark studies for ITER. Getting his input for those studies and running those cases with transp namelist is the best way to resolve the differences. [ A request to Ron Prater for the input files used in the comparison was made by Kumar at the 2010 APS. ] GENRAY and LH: -------------- Some preliminary work has been done for invoking genray for LH heating/CD. As of Dec 2010, the goal was to get a simple simulation working with a template namelist for CMOD. To this end, the following changes are made in transp: (a) the switch NLGEN_LHH invokes genray LH. (b) the code genraylh.f90 handles much of the mechanics. It is structured to be similar to genrayech.f90. Examination of genrayech.f90 and genraylh.f90 should clarify the mode of operation of genray in transp. (c) Currently only the profiles are being transferred to the namelist. None of the LH related data from transp namelist is being transferred to genray namelist. This is because such data is not in the plasma state yet. Thus, all the desired LH data need to be first loaded into the plasma state and then prepare_genray_input needs to be modified to transfer this data into genray namelist. (d) When transp is run with genray LH invoked, it fails the first time it tries to run genray with FPE. I have not examined the cause of failure. GENRAY/LH variables in the namelist: The template $CODESYSDIR/source/nml/CMOD_GENRAY_LHH_STANDARD.IN is heavily commented. It has a long list of namelists that are commented. Not everything needs to be changed, but it is worth studying it and corresponding with Bob Harvey and collaborators about what needs to be changed. The following is a minimal list: (i) wave -- frqncy, jwave (=0 is LH) (ii) grill -- many variables are reminiscent of LSC, but not all. 2) CQL3D work: ----------- CQL3D repository is in: /p/nubeam/cql3d-svn/cql3d/trunk [ There is a minor svn conflict in freyasou.f which needs to be fixed in order to build cql3d. ] The PPPL relevant makefiles are listed with: ls -l *makefile*pppl* To build CQL3D, do: gmake -f A copy of CQL3D is in /u/xshare/kumar/cql3d. This was used to build a gcc version of cql3d on sunfire02. CQL3D is incorporated into TRANSP as a library (cql3d_lib) and an executable (cql3d_transp). As of this writing, cql3d is not being invoked in transp. [ The cql3d_lib/import_cql3d.py script is a skeleton python script for moving the code from cql3d svn to cql3d_lib. The executable a_cqlp.f needs to be converted to a_cqlp_sub.for] [ The cql3d_lib library will need to be replaced with a newer version which is still under development which handles up-down asymmetries and multiple ion species. ] 3) TORIC related work ------------------ TORIC svn in: /p/nubeam/pll-toric/toric-new/TORIC (Code and makefiles in: /p/nubeam/pll-toric/toric-new/TORIC/trunk/src) Toric svn build instructions are in: /p/nubeam/pll-toric/toric-new/toric-build-instructions.txt mpi_toric_main is the parallel executable in transp. Recommendation (perhaps long term): Move to a set up where toric is invoked as a separate process similar to genray., i.e. invoke prepare_toric_input. toric_main, and process_toric_output as separate processes. This would make it easier for the MIT folks to troubleshoot any toric failures as well as provide a uniform framwork for third-party software deployment (similar to the IPS setup). 4) MPI broadcasting ---------------- The program pfilecopy (mpi_pfilecopy is the parallel executable) is used to broadcast all the input files (to toric) to all the local directories on participating MPI processes. It is invoked as: mpirun -np $LOCAL/exe/mpi_pfilecopy It uses the portlib/parcopy.f90 and portlib/portlib_parcopy.c to copy any file (irrespective of the nature of contents, i.e. ASCII, binary, netcdf, etc.) to remote participating processes. The C routine portlib_parcopy.c basically copies the data as bytes, thereby making it possible to copy files without regard to the nature of contents. There is documentation at the top of pfilecopy as well as portlib_parcopy.c. 5) FACETS work: ------------ My work over the past year or so has consisted in attending FACETS hands on meetings, sitting on weekly conference calls, preparing monthly and yearly reports, FACETS review presentations, etc. I have also been involved with building FACETS at PPPL. FACETS has a separate project directory (/p/facets) and I have been building FACETS in /p/facets/kindires. After the creation of a PPPL clone at Tech-X and significant amount of work by John Cary, Alex Pletzer, and myself, we were able to compile FACETS. We have not run it yet. FACETS at PPPL: -------------- To build FACETS, you need to generally follow the instructions on the FACETS wiki page: https://www.facetsproject.org/wiki/FacetsBuildInstructions You may need an account (which may be the same as SVN account) to do this. The following are PPPL specific instructions. If you encounter any difficulties, PPPL systems folks, John Cary and Alex Pletzer (at Tech-X) are the people to contact. FACETS is built on a PPPL clone at Tech-X. 1) Login to portalr5 (stix). Load the following modules or source a script that has the following modules: module purge module load gcc4/4.4.1 module load openmpi/1.4.1 module load acml/4.3.0 module load java/1.6.0_16 module load python/2.6.3 module load subversion/1.6.6 setenv F77 $FC setenv MPIF90 mpif90 setenv MPIF77 mpif90 setenv MPICC mpicc setenv MPICXX mpicxx 2) Get SVN aceess to Tech-X FACETS repository. Create appropriate directoris for FACETS. There needs to be a home directory for facets, something like: setenv HOME /p/facets/kindires (example) and also an installation directory: /p/facets/stix-gnu-4.4/ (example) and separate directories facetspkgs and facetsall. 3) cd facetspkgs svn up build facetspkgs with something like: ./mkfcpkgs.sh -a -O -m stix.pppl.gov.gnu -i /p/facets/stix-gnu-4.4/ 4) After the facetspkgs build completes, change to facetsall cd facetsall svn up ./mkfcall.sh -aO -j 2 -m stix.pppl.gov.gnu -i /p/facets/stix-gnu-4.4