====== Stop-day activities April 10-11, 2018 ======

\\

^ Coordinator | Teun Grit | roadmin@astron.nl |
^ Software Support | Thomas Juerges + Arno Schaap (2 days)| softwaresupport@astron.nl |
^ Science, Operations and Support | Pietro Zucca | sos@astron.nl |
^ Observer | Jur Sluman | observer@astron.nl |


[[engineering:stop_day_procedures|More stopday details]]

[[https://docs.google.com/spreadsheets/d/18K2QEabv-ipYzjCcJgMfA5hsBX3jH6OycXdXhxGV8Is/edit?ts=5a045853#gid=0|LOFAR Schedule cycle 9]]

===== Systems =====



==== Cobalt ====

  * ✔ Reboots and idrac reboots. (Hopko/Robin)
  * ✔ CBM010 will be present before the stopday
==== CEP3 ====

  * ✔ Block access at 08:00 (Teun)
  * ✔ Mount DAC cables to headnodes to support floating IP (Hopko/Robin)
  * ✔ All nodes: file system check and reboot. (Hopko, Robin)
  * ✔ Was broken: Check/debug persistence of Slurm reservations (Reinoud)  **For the review**: We should have a backup in place!
  * ✔ NFS mounts for cep3, from all the lof-nodes are using control!


==== CEP4 ====

  * ✔ Connect DAC cable (Hopko)

==== LEXARS ====

  *  ✔ lexar003 reboot using XCAT (148 days up) (Was done by Reinoud 9-4-2018)

==== LCU ====

  *  ✔ Reboot of all Dutch LCU's (teun) (ILT stations in local mode) **For the review**: We discovered that all LCU's have a fixed NFS mount on /home. This should be an automount.

==== Central Services ====
 
  * ✔ Update portals to CentOS/KVM
  * ✔ Update and reboot lcs020 .. lcs30
  * ✔ Remove sas001 and sas099 (powered off, disconnect cables)
  * ✔ Update & reboot NFS server lcs115
  * ✔ Almost all nfs mounts on the lcs115 nfs server are over the control network. Only a few correctly use the offline network: only lexar003, lexar004, and lhd002. We should force CEP3 to use off-line!
  * ✔ Mainly MAC/SAS, LCU's and Aartfaac machines use NFS over control VLAN. They don't have a off-line of on-line network connection.
  * Check resolv.conf settings; see https://support.astron.nl/lofar_issuetracker/issues/10448
==== LTA ====

  * ✔ Update and reboot when required (Reinoud)

==== Aartfaac ====

  * ✘ Check for broken disks **Fail**: ais007 has a degraded RAID1  !!
  * **For review**: Add Jasmin & Reinoud to all nodes as admin
==== Core switches ====

  * none (probably June)


==== Communication issues ====

**For review**: At the end of the 1st day software support needs to report status to coordinator
===== Software updates =====

==== MoM and related ====

  * ?

==== MAC/SAS ====

  * ?

==== CEP3 ====

  * ✔ Reboot / fs checks
  * ✔ Make AOFlagger 2.10 the default version (already installed)
  * ✔ Make LOFAR-Release-3_0_14 the default version (linked against AOFlagger 2.10)
  * ✔ Make WSClean 2.5 the default version (already installed)

==== CEP4 ====

  * ?

==== Aartfaac ====

  * none

==== COBALT ====

  * none

==== LTA ====

  * none 

===== In the field =====

  * none


===== External =====

  * ?

==== Next stopday ====

The next stopday is June 5 & 6 (TBD)