public:lta_tricks

This is an old revision of the document!


Advanced ways to find and retrieve data in the LTA

There are some useful ways to find and retrieve your data in the LTA that might not be immediately obvious. This page explains some of the more advanced options you have.

  • You can use wildcards in your query. This will for example give all observations and pipelines that have a SAS/Observation ID in the 146xxx range:

  • You can put a list of SAS/Observation IDs in the query:

When you are looking at the results of a query you might see something like this:

This means that the observation is known in the LTA, it knows what data was produced, the produced data was not archived, but further processing happened on the raw data and the results of some of those pipelines were archived. If you click on the zero, you will see something like this:

This allows you to navigate from a pipeline back to the original observation, or from the observation to any pipelines that have run on the raw data.

  • You can retrieve data on the Observation and Pipeline level, you don't have to select all files individually.

  • If you have a query with more than 1000 results, you can open the multiple pages each in a separate tab/window.

  • With the small triangle next to a list, you can fold or unfold the list to get a better overview.
Folded entries

Unfolded entries

There is a server that gives the option to run your own queries on the database http://lofar-dbview.target.rug.nl/

A useful query might be this one, that gives you all files for a certain Obs Id (SAS VIC tree ID).

SELECT fo.URI, dp."dataProductType", dp."dataProductIdentifier",
 dp."processIdentifier"
FROM AWOPER."DataProduct+" dp,
     AWOPER.FileObject fo,
     AWOPER."Process+" pr
WHERE dp."processIdentifier" = pr."processIdentifier"
  AND pr."observationId" = '123456'
  AND fo.data_object = dp."object_id"
  AND dp."isValid" > 0

In this '123456' should be replaced with the Obs Id of an Observation/Pipeline you're looking for.

There is also a python interface to the LTA. With this you can also script some advanced queries. This for example gives all observations on a certain patch of the sky:

# python code
from pprint import pprint
from common.database.Context import context
from lofar.main.Pointing import Pointing
result = {}
for project in sorted(context.get_projects()) :
  print "Project %(project)s" % vars()
  ok = context.set_project(project)
  # do your query
  obs_ids = set()
  query = (Pointing.rightAscension > 15 * 8)  & \
      (Pointing.declination    < 15 * 17) & \
      (Pointing.declination > 0)          & \
      (Pointing.declination < 60)
  for pointing in query :
    query_subarr = SubArrayPointing.pointing == pointing
    for subarr in query_subarr:
      query_obs = Observation.subArrayPointings.contains(subarr)
      for obs in query_obs :
        obs_ids.add(obs.observationId)
  result[project] = sorted(list(obs_ids))
  print result[project]

pprint(result)
  • Last modified: 2014-07-01 18:02
  • by Andreas Horneffer