public:lta_tricks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
public:lta_tricks [2019-09-20 07:25] – Updated link to LOFAR stager API Python module to point to the ASTRON gitlab repo Thomas Jürgespublic:lta_tricks [2023-07-17 08:51] (current) – Robbie Luijben
Line 51: Line 51:
 There is a server that gives the option to run your own queries on the database [[https://lta-dbview.lofar.eu/|https://lta-dbview.lofar.eu/]] There is a server that gives the option to run your own queries on the database [[https://lta-dbview.lofar.eu/|https://lta-dbview.lofar.eu/]]
  
-A useful query might be this one, that gives you all files for a certain Obs Id (SAS VIC tree ID).<code>+A useful query might be this one, that gives you all files for a certain Obs Id (SAS VIC tree ID). 
 +<code> 
 SELECT fo.URI, dp."dataProductType", dp."dataProductIdentifier", SELECT fo.URI, dp."dataProductType", dp."dataProductIdentifier",
  dp."processIdentifier"  dp."processIdentifier"
Line 61: Line 63:
   AND fo.data_object = dp."object_id"   AND fo.data_object = dp."object_id"
   AND dp."isValid"> 0   AND dp."isValid"> 0
 +
 </code> </code>
  
Line 77: Line 80:
   AND fo.data_object = dp."object_id"   AND fo.data_object = dp."object_id"
   AND dp."isValid"> 0   AND dp."isValid"> 0
 +
 </code> </code>
  
Line 85: Line 89:
 Once you have installed the client, set up your user name and password. These are the same as for MoM. Remember that this is just a different interface to the LTA catalogue: you will need the same credentials as for the web interface. Once you have installed the client, set up your user name and password. These are the same as for MoM. Remember that this is just a different interface to the LTA catalogue: you will need the same credentials as for the web interface.
  
-After installing the LTA client, the file .awe/Environment.cfg will appear in your home directory (if not, then create one). Make sure the file at least contains the following lines:<file>+After installing the LTA client, the file .awe/Environment.cfg will appear in your home directory (if not, then create one). Make sure the file at least contains the following lines: 
 +<file> 
 [global] [global]
 database_user       : <your username> database_user       : <your username>
 database_password   : <your password> database_password   : <your password>
 +
 </file> </file>
  
Line 120: Line 127:
  
 pprint(result) pprint(result)
 +
 </code> </code>
  
Line 134: Line 142:
 Pointing found RA 95.561458 DEC 24.584056 Pointing found RA 95.561458 DEC 24.584056
 ..etc.. ..etc..
 +
 </code> </code>
  
Line 140: Line 149:
 In case of errors, there may be the need to open some port on the firewall at your institution. Specifically, port 1521 should be open. Also make sure that the LTA client library can be found in your PYTHONPATH (see [[:lta:client_installation|LTA Client installation]] for more details). In case of trouble, get in contact with Science Operations and Support. In case of errors, there may be the need to open some port on the firewall at your institution. Specifically, port 1521 should be open. Also make sure that the LTA client library can be found in your PYTHONPATH (see [[:lta:client_installation|LTA Client installation]] for more details). In case of trouble, get in contact with Science Operations and Support.
  
-== Examples ==+==== Examples ====
  
 Once you have tested that your connection to the catalogue is working, you are ready to browse the archive and stage the data you need. Here we will list a few examples of python scripts that can be used to access the LTA. All of them will need to import some modules: Once you have tested that your connection to the catalogue is working, you are ready to browse the archive and stage the data you need. Here we will list a few examples of python scripts that can be used to access the LTA. All of them will need to import some modules:
Line 151: Line 160:
     Observation     Observation
 from awlofar.toolbox.LtaStager import LtaStager, LtaStagerError from awlofar.toolbox.LtaStager import LtaStager, LtaStagerError
 +
 +
 </code> </code>
  
 The lines above must be added to each of the scripts below for these to work. The lines above must be added to each of the scripts below for these to work.
  
-This simple script will allow you to find all data within a single project, for example LC2_035. Please change the project name to the code of a project of yours. If you also want to stage the data you found, just set the do_stage variable to True. Be careful with how many files you stage and what size they have: the same limits as for the web interface apply here.+=== Ex: get staging URI's === 
 + 
 +This script will allow you to find all data within a single project, for example LC2_035. Please change the project name to the code of a project of yours. If you also want to stage the data you found, just set the do_stage variable to True. Be careful with how many files you stage and what size they have: the same limits as for the web interface apply here.
  
 <code python> <code python>
Line 195: Line 208:
     stager = LtaStager()     stager = LtaStager()
     stager.stage_uris(uris)     stager.stage_uris(uris)
 +
  
 </code> </code>
 +
 +=== Ex: filter on subbands ===
  
 The following script will find subbands 301 and 302 for all targets within two different projects. The following script will find subbands 301 and 302 for all targets within two different projects.
Line 251: Line 267:
     for project in (project1, project2) :     for project in (project1, project2) :
         stager.stage_uris(uris[project])         stager.stage_uris(uris[project])
 +
 +
 </code> </code>
 +
 +=== Ex: filter on frequency and observation date ===
  
 Here, we find data between freq1 and freq2 taken within one project between day1 and day2 Here, we find data between freq1 and freq2 taken within one project between day1 and day2
Line 299: Line 319:
     stager = LtaStager()     stager = LtaStager()
     stager.stage_uris(uris)     stager.stage_uris(uris)
 +
  
 </code> </code>
  
-== Example; query public data ==+=== Ex: query public data ===
  
 Querying public data in projects you are not member of. First set project ALL, then construct a query and optionally limit the query to a certain project : Querying public data in projects you are not member of. First set project ALL, then construct a query and optionally limit the query to a certain project :
Line 312: Line 333:
 print(len(query)) print(len(query))
 # 1800 # 1800
 +
 +
 </code> </code>
  
-===== Python Module for Staging =====+=== Ex: get release dates === 
 + 
 +<code python> 
 +from awlofar.main.aweimports import Observation, PipelineRun, DataProduct 
 +from common.database.Context import context 
 + 
 +project = 'LC2_035' 
 + 
 +# Query for private data of the project, you must be member of the project 
 +private_data = True 
 + 
 +# To see private data of this project, you must be member of this project 
 +if private_data : 
 +    context.set_project(project) 
 +    if project != context.get_current_project().name: 
 +        raise Exception("You are not member of project %s" % project) 
 + 
 +# Observations 
 +query_observations = Observation.select_all().project_only(project) 
 +for observation in query_observations : 
 +    print("Querying ObservationID %s, %s" % (observation.observationId, observation.releaseDate)) 
 + 
 +# Pipelines 
 +query_pipelines = PipelineRun.select_all().project_only(project) 
 +for pipeline in query_pipelines : 
 +    print("Pipeline: %s, %s, %s" % (type(pipeline).__name__, pipeline.pipelineName, pipeline.releaseDate)) 
 + 
 +# Data products 
 +query_products = DataProduct.select_all().project_only(project) 
 +query_products &= DataProduct.isValid == 1 
 +for product in query_products : 
 +    print("Product: %s, %s, %s, %s" % (product.dataProductIdentifier, product.dataProductIdentifierSource, product.dataProductType, product.releaseDate))
  
-The python interaction with the LTA catalog can be complemented with the use of a specific module developed to give users more control over their staging requests.  The module is made available **[[https://git.astron.nl/ro/lofar_stager_api/-/tags|here]]** and its functions are mostly self-explanatory. 
  
-**Alternatively to the .awe/Environment.cfg described above, user credentials can also be provided via a file ~/.stagingrc with credentials of your Lofar account**, similar to ./wgetrc:<code> 
-  user=XXX 
-  password=YYY 
 </code> </code>
 +
 +===== Python Module for Staging =====
 +
 +The python interaction with the LTA catalog can be complemented with the use of a specific module developed to give users more control over their staging requests.
 +
 +Current released version 2.0 ([[http://git.astron.nl/astron-sdc/lofar_stager_api/-/releases/2.0|tagged on master branch]]) is to be used with the new LTA stager (stageit), it is not backwards compatible with the old LTA stager. Older versions of this script (i.e., 1.7 and older) have become obsolete. Please see **"Version 2.0 usage notes"**  listed below for documentation (or check the README file in the repository linked above).
 +
 +  * User documentation for __//stageit //__  can be found at: [[https://support.astron.nl/confluence/display/SDCP/User+documentation|https://support.astron.nl/confluence/display/SDCP/User+documentation]]
 +  * Version 2.0 release can be found at: [[https://git.astron.nl/astron-sdc/lofar_stager_api/-/releases/2.0|https://git.astron.nl/astron-sdc/lofar_stager_api/-/releases/2.0]]
 +
 +==== Version 2.0 usage notes ====
 +
 +The module is made available [[http://git.astron.nl/astron-sdc/lofar_stager_api/-/releases/2.0|here]] (tagged on master branch). Simply checkout the tagged commit and use the script. Please see the last note in the list below with regards to required dependencies.
 +
 +__Notes:__
 +
 +  * You need an access token to the stageit api. Please refer to the user guide linked above to sign up and login to stageit. After logging in, a token can be obtained in one of two ways:
 +      * Visit [[https://sdc.astron.nl/stageit/api/staging/get-token|https://sdc.astron.nl/stageit/api/staging/get-token]]
 +      * From anywhere in the application, click on your account name in the top right to access your profile. From your profile page, click the "Request token" button to receive a token.
 +  * The token is valid indefinitely. Requesting a token multiple times will yield the same token.
 +  * Make sure the token is available in your ~/.stagingrc file:
 +      * api_token=YOUR_TOKEN_HERE
 +      * remove the old username and password from the ''.stagingrc''  file
 +  * The script is Python2 compatible, there is a Dockerfile available for Python2 testing in ''./tests/docker''
 +  * The requests library is a required dependency. If you care about Python2 compatability, you can use at most version 2.22.0 of requests. Otherwise, you can install any version (note: you can also ''pip install -r 'requirements.txt''', which will install version 2.22.0)
 +
 +Also note that some functions are not supported in the new LTA stager. The states that a request can be in have been simplified. As such, there is no need for these functions anymore. Upon use, they will display an error stating that the function is deprecated. Please look at the ''stager_access.py''  file for more information.
 +
 +==== Functionality ====
  
 For a description of what the user can do, we list here the functions that are available. For a description of what the user can do, we list here the functions that are available.
Line 340: Line 419:
  
 Below is an example of how to use this: Below is an example of how to use this:
- +<code> 
-<code>> python+> python
 Python 2.7.10 (default, Oct 23 2015, 19:19:21) Python 2.7.10 (default, Oct 23 2015, 19:19:21)
 [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Line 350: Line 429:
  
 + 12227 + 12227
-  - File count      ->     100+  - File count   ->     100
   - Files done      ->     40   - Files done      ->     40
   - Flagged abort      ->     false   - Flagged abort      ->     false
Line 357: Line 436:
   - Status      ->     on hold   - Status      ->     on hold
   - User id      ->     1919   - User id      ->     1919
 +
 </code> </code>
  
  
  • Last modified: 2019-09-20 07:25
  • by Thomas Jürges