Tracing MPI Jobs

1. Find out where jobs are running:
   • qstat -n1 | grep <user>
     shows pbsid and nodes
     -or -
   • qstat <pbsid> -f
2. See what is running:
   ssh <node> ps -efw --forest |grep <user>
3. $WORKDIR
   WORKDIR is /l/<node>/<user>/work/<TOK>
   nubeam files are in: /l/<node>/<user>/work/<TOK>/<runid>_fi/
4. pbs log
   • log into node of rank 0 (serial job)
     ssh -l <user> <node>
   • more /var/spool/torque/spool/<pbsid>.bennu.OU
5. killing a hang job
   and retrieving files
   • ssh <node> ps -ef |grep EXE
     to get pid
   • ssh <node> kill <pid>