Checklist slurm upgrade

  1. ✔ stop slurm accross the cluster (slurmctld, slurmdbd, slurmd)
  2. ✔ dump mysql database 'slurmaccounting_cep4' at lcs032 (see lcs032:/usr/local/bin/mysql_backup.sh)
  3. ✔ dump slurm accounts and reservations on head01:
    scontrol show res -o > /root/slurm/reservations.`date +\%a`.txt
    sacctmgr dump lofar_cep3 File=/root/slurm/sacctmgr.`date +\%a`.dump
    sacctmgr show qos > /root/slurm/qos.`date +\%a`.dump
  4. ✔ create backup of /data/home/slurm
  5. ✔ yum remove slurm slurm-slurmdbd slurm-sql slurm-plugins
  6. ✔ yum install slurm slurm-slurmctld etc…
  7. ✔ start slurm on head node
  8. ✔ check accounts / users: sacctmgr
  9. ✔ check cobalt reservation; scontrol show res
  10. ✔ start nodes. Check sinfo -R.

In case accounts need to be recreated:

sacctmgr create qos inspectionplots Priority=10 Preempt=normal
sacctmgr create qos permanent Priority=20]
sacctmgr load sacctmgr.Thu.dump
  • Last modified: 2019-02-05 13:59
  • by Reinoud Bokhorst