public:lta_tricks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Next revisionBoth sides next revision
public:lta_tricks [2020-04-10 10:01] – [AstroWise Python Interface] Reinoud Bokhorstpublic:lta_tricks [2020-07-21 13:53] – Updated link for Python stager API releases Thomas Jürges
Line 51: Line 51:
 There is a server that gives the option to run your own queries on the database [[https://lta-dbview.lofar.eu/|https://lta-dbview.lofar.eu/]] There is a server that gives the option to run your own queries on the database [[https://lta-dbview.lofar.eu/|https://lta-dbview.lofar.eu/]]
  
-A useful query might be this one, that gives you all files for a certain Obs Id (SAS VIC tree ID).<code>+A useful query might be this one, that gives you all files for a certain Obs Id (SAS VIC tree ID). 
 +<code> 
 SELECT fo.URI, dp."dataProductType", dp."dataProductIdentifier", SELECT fo.URI, dp."dataProductType", dp."dataProductIdentifier",
  dp."processIdentifier"  dp."processIdentifier"
Line 61: Line 63:
   AND fo.data_object = dp."object_id"   AND fo.data_object = dp."object_id"
   AND dp."isValid"> 0   AND dp."isValid"> 0
 +
 </code> </code>
  
Line 77: Line 80:
   AND fo.data_object = dp."object_id"   AND fo.data_object = dp."object_id"
   AND dp."isValid"> 0   AND dp."isValid"> 0
 +
 </code> </code>
  
Line 85: Line 89:
 Once you have installed the client, set up your user name and password. These are the same as for MoM. Remember that this is just a different interface to the LTA catalogue: you will need the same credentials as for the web interface. Once you have installed the client, set up your user name and password. These are the same as for MoM. Remember that this is just a different interface to the LTA catalogue: you will need the same credentials as for the web interface.
  
-After installing the LTA client, the file .awe/Environment.cfg will appear in your home directory (if not, then create one). Make sure the file at least contains the following lines:<file>+After installing the LTA client, the file .awe/Environment.cfg will appear in your home directory (if not, then create one). Make sure the file at least contains the following lines: 
 +<file> 
 [global] [global]
 database_user       : <your username> database_user       : <your username>
 database_password   : <your password> database_password   : <your password>
 +
 </file> </file>
  
Line 120: Line 127:
  
 pprint(result) pprint(result)
 +
 </code> </code>
  
Line 134: Line 142:
 Pointing found RA 95.561458 DEC 24.584056 Pointing found RA 95.561458 DEC 24.584056
 ..etc.. ..etc..
 +
 </code> </code>
  
Line 151: Line 160:
     Observation     Observation
 from awlofar.toolbox.LtaStager import LtaStager, LtaStagerError from awlofar.toolbox.LtaStager import LtaStager, LtaStagerError
 +
 </code> </code>
  
 The lines above must be added to each of the scripts below for these to work. The lines above must be added to each of the scripts below for these to work.
  
-This simple script will allow you to find all data within a single project, for example LC2_035. Please change the project name to the code of a project of yours. If you also want to stage the data you found, just set the do_stage variable to True. Be careful with how many files you stage and what size they have: the same limits as for the web interface apply here.+=== Ex: get staging URI's === 
 + 
 +This script will allow you to find all data within a single project, for example LC2_035. Please change the project name to the code of a project of yours. If you also want to stage the data you found, just set the do_stage variable to True. Be careful with how many files you stage and what size they have: the same limits as for the web interface apply here.
  
 <code python> <code python>
Line 195: Line 207:
     stager = LtaStager()     stager = LtaStager()
     stager.stage_uris(uris)     stager.stage_uris(uris)
 +
  
 </code> </code>
 +
 +=== Ex: filter on subbands ===
  
 The following script will find subbands 301 and 302 for all targets within two different projects. The following script will find subbands 301 and 302 for all targets within two different projects.
Line 251: Line 266:
     for project in (project1, project2) :     for project in (project1, project2) :
         stager.stage_uris(uris[project])         stager.stage_uris(uris[project])
 +
 </code> </code>
 +
 +=== Ex: filter on frequency and observation date ===
  
 Here, we find data between freq1 and freq2 taken within one project between day1 and day2 Here, we find data between freq1 and freq2 taken within one project between day1 and day2
Line 299: Line 317:
     stager = LtaStager()     stager = LtaStager()
     stager.stage_uris(uris)     stager.stage_uris(uris)
 +
  
 </code> </code>
  
-== Example; query public data ==+=== Ex: query public data ===
  
 Querying public data in projects you are not member of. First set project ALL, then construct a query and optionally limit the query to a certain project : Querying public data in projects you are not member of. First set project ALL, then construct a query and optionally limit the query to a certain project :
Line 312: Line 331:
 print(len(query)) print(len(query))
 # 1800 # 1800
 +
 +</code>
 +
 +=== Ex: get release dates ===
 +
 +<code python>
 +from awlofar.main.aweimports import Observation, PipelineRun, DataProduct
 +from common.database.Context import context
 +
 +project = 'LC2_035'
 +
 +# Query for private data of the project, you must be member of the project
 +private_data = True
 +
 +# To see private data of this project, you must be member of this project
 +if private_data :
 +    context.set_project(project)
 +    if project != context.get_current_project().name:
 +        raise Exception("You are not member of project %s" % project)
 +
 +# Observations
 +query_observations = Observation.select_all().project_only(project)
 +for observation in query_observations :
 +    print("Querying ObservationID %s, %s" % (observation.observationId, observation.releaseDate))
 +
 +# Pipelines
 +query_pipelines = PipelineRun.select_all().project_only(project)
 +for pipeline in query_pipelines :
 +    print("Pipeline: %s, %s, %s" % (type(pipeline).__name__, pipeline.pipelineName, pipeline.releaseDate))
 +
 +# Data products
 +query_products = DataProduct.select_all().project_only(project)
 +query_products &= DataProduct.isValid == 1
 +for product in query_products :
 +    print("Product: %s, %s, %s, %s" % (product.dataProductIdentifier, product.dataProductIdentifierSource, product.dataProductType, product.releaseDate))
 +
 </code> </code>
  
 ===== Python Module for Staging ===== ===== Python Module for Staging =====
  
-The python interaction with the LTA catalog can be complemented with the use of a specific module developed to give users more control over their staging requests.  The module is made available **[[https://git.astron.nl/ro/lofar_stager_api/-/tags|here]]** and its functions are mostly self-explanatory.+The python interaction with the LTA catalog can be complemented with the use of a specific module developed to give users more control over their staging requests. The module is made available **[[https://git.astron.nl/ro/lofar_stager_api/-/releases|here]]** and its functions are mostly self-explanatory. 
 + 
 +**Alternatively to the .awe/Environment.cfg described above, user credentials can also be provided via a file ~/.stagingrc with credentials of your Lofar account**, similar to ./wgetrc: 
 +<code>
  
-**Alternatively to the .awe/Environment.cfg described above, user credentials can also be provided via a file ~/.stagingrc with credentials of your Lofar account**, similar to ./wgetrc:<code> 
   user=XXX   user=XXX
   password=YYY   password=YYY
 +
 </code> </code>
  
Line 357: Line 415:
   - Status      ->     on hold   - Status      ->     on hold
   - User id      ->     1919   - User id      ->     1919
 +
 </code> </code>
  
  
  • Last modified: 2023-07-17 08:51
  • by Robbie Luijben