====== Checklist slurm upgrade ======

  - ✔ stop slurm accross the cluster (slurmctld, slurmdbd, slurmd)
  - ✔ dump mysql database 'slurmaccounting_cep4' at lcs032 (see lcs032:/usr/local/bin/mysql_backup.sh)
  - ✔ dump slurm accounts and reservations on head01: <file>
scontrol show res -o > /root/slurm/reservations.`date +\%a`.txt
sacctmgr dump lofar_cep3 File=/root/slurm/sacctmgr.`date +\%a`.dump
sacctmgr show qos > /root/slurm/qos.`date +\%a`.dump
</file>
  - ✔ create backup of /data/home/slurm
  - ✔ yum remove  slurm slurm-slurmdbd slurm-sql slurm-plugins
  - ✔ yum install slurm slurm-slurmctld etc...
  - ✔ start slurm on head node
  - ✔ check accounts / users: sacctmgr
  - ✔ check cobalt reservation; scontrol show res
  - ✔ start nodes. Check sinfo -R.



In case accounts need to be recreated:
<file>
sacctmgr create qos inspectionplots Priority=10 Preempt=normal
sacctmgr create qos permanent Priority=20]
sacctmgr load sacctmgr.Thu.dump
</file>