ILT-TO meeting April 11, 2011 (ILT Technical Operations)

The ILT Technical Operations meeting is held at the start of a LOFAR observing semester. The purpose of the meeting is to discuss operations procedures for the ILT, covering all aspects from observation scheduling to station maintenance. The audience for this meeting consists of (technical) personnel involved in running and maintaining the station.

This second meeting was held on April 11, 2011, at ASTRON in Dwingeloo.

Meeting presentation (Harm Munk)

All but the Onsala stations have been built. The LBAs at the Jülich station have been installed late in April 2011.

Funding for sixth German station near Bieleveld is being sought. If the funding comes through the station will probably be build in 2012.

eVLBI will require approximately 4 Gb/s bandwidth. That will not leave sufficient bandwidth for two international stations to share the same 10 Gb/s fibre to Groningen. Changing the number of beamlets on the fly will not reduce bandwidth: it merely pads the unused channels with zeroes.

Having sufficient bandwidth to connect all international stations at the same time would enable the study of ionospheric effects.

While installing the LOFAR software at Jülich, in casu NDPPP, it was found that the software was designed for single user clusters: if run on a multi-user cluster the software breaks because it has to compete for computing resources with other jobs.

The LBA's were installed in 2007, the HBA's in 2009. In December 2010 the Rb-clock broke down. After replacing the Rb-clock, no fringes have been observed with DE601. Also, pulsars could not be observed but this might be due to a software problem.

Eight apparently broken RCU boards have been tested at ASTRON and appear to be functioning correctly. The problem is now being investigated in Effelsberg again. It has been decided not to replace the RCU boards in Effelsberg (they lack overvoltage protection on the antenna inputs, which poses a risk especially at low humidity.)

The humidity in the container can become very low during cold winternights because of the heater. A humidifyer will be installed. The airconditionnig unit will not work below 18 degrees centigrade. A humidity below 20% is a potential hazard.

ASTRON will develop an application for monitoring the container humidity.

Several HBAs are still broken. ASTRON will repair these tiles.

Several weeks ago a wrong configuration got loaded which shut the HBA part down.

Often used for long baselines experiments.

A temperature monitoring facility is requested for this station as well.

Network connection up to recently troublesome, but seems ok now.

Some damage due to wild boars. Protection against beech martens (or stone martens) around the LBA wires have been installed. Unknown if that changes the electrical characteristics of the antennae.

A wooden fence has been installed.

Drainage has been improved.

LBAs will be installed in the week of April 18-22.

Cable over-length pit will be filled up soon afterwards.

Instead of trenches, the whole filed has been excavated for the cable

HBAs to placed in 2011, LBAs in 2012.

Connection to CEP in 2012.

A few destructive tests will be performed with the KAIRA station test HBAs. In one test up to ten tons of snow will be piled on the HBA. Also, the effect of low temperatures will be investigated by walking and jumping on the tiles.

Special requests for low temperature test can be send to Derek McKay

There is a need for basic training of station personnel:

• switching the station off and on
• logging in when the station is in ILT mode
• procedure(s) to locally check the proper functioning of the system

Division of time (according to ILT contract): 10% is avialable for maintenance, 90% for observations.

10% of this 90% is available for stand alone mode, making this 9% of the total time.

After a stand alone observation, an authorised person must do the hand over to ILT mode.

A monthly conference call will be held around the 15th of the month to discuss technical, operational and organisational issues.

• The Effelsberg station is visually inspected every day by security personnel, weekly by the Effelsberg operator.

• The Garching/Unterweilenbach station is checked every two weeks. (Station is remote.)

• The Potsdam station is inspected regularly, but there is no schedule for inspections.

• The Chilbolton station is checked nowadays about every two weeks.

• The Jülich station is inspected every other week.

• The Tautenberg station is next to the observatory and therefore checked often. A close inspection is done on average every two months, but there is no schedule.

A remotely controlled pilot transmitter is under construction which will allow for remote testing of the HBAs of the international stations.

At Tautenburg, the LBAs during the winter at some time were leaning over. This was caused by snow and ice buildup on the antenna strings and rubber bands. No permanent effect.

At Effelsberg, several LBA poles are bent. This is probably caused by heat. This effect has not (yet) been seen on other stations.

Note that during high wind and low temperature the resilience of the rubber bands which hold down the covers of the HBA is negatively influenced, which may cause the covers to come loose.

Container maintenance: the firm COMTEST (http://www.comtestnl.com/index.shtml) has prepared a maintenance contract for the containers. This contract will be distributed to the persons responsible for station operations at the international stations. Note that it is the responsibility of the local station operation organisation to manage this contract.

Software upgrades usually change the user interface, even if it is only a command-line tool. When is software upgrade is to be installed, the known changes should be summarised in a release note, which in turn is to be send to all stations. These release notes should be send well in advance to allow for comments and preparation of possible local software.

The Effelsberg station uses first generation RCUs (the other stations use second generation RCUs). Mixing old and new RCUs is undesirable. ASTRON still has sufficient (~10) first generation RCUs to be able to support the Effelsberg station.

On average, the energy bill for a typical international station is between 5 and 7 KEURO. It is suggested that this power consumption is reduced when the station is not into use. This can be achieved by

• switching to sw level 0: switches the antenna's off, but does not affect the clock
• put RSP and TBB boards in low power mode. It is not known if this affects the lifetime of the RSP and TBB boards. There are temperature sensors on all TBB and RSP boards, but these are not monitored
• switch off subracks: the drawback is that the subracks need to reach temperature stability again after switch on, which takes a few hours

Note that the air-conditioning units of the containers can only cool, not heat.

There is only portal access when the station is in ILT mode.

When in stand alone mode, LCU control can be kept intact, but LCU access should be discouraged, or even prevented in this mode. Related to this is a means to monitor the station mode (stand alone or ILT). The state should be available somewhere on the LOFAR Wiki.

Please notify Teun Grit at ASTRON (lofar-network@astron.nl) when suspicious events happen on a LOFAR station.

The backdoor access facility that was available until now will gradually be removed through the use of an access list.

Torque wrenches for the F-connectors: see http://www.lofar.org/operations/doku.php?id=maintenance:maintenance#lofar_usage_of_torque_wrenches