Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
public:lta_tricks [2020-04-10 10:01] – [AstroWise Python Interface] Reinoud Bokhorst | public:lta_tricks [2023-04-14 07:44] – Updated link to moved GitLab project Robbie Luijben | ||
---|---|---|---|
Line 51: | Line 51: | ||
There is a server that gives the option to run your own queries on the database [[https:// | There is a server that gives the option to run your own queries on the database [[https:// | ||
- | A useful query might be this one, that gives you all files for a certain Obs Id (SAS VIC tree ID).< | + | A useful query might be this one, that gives you all files for a certain Obs Id (SAS VIC tree ID). |
+ | < | ||
SELECT fo.URI, dp." | SELECT fo.URI, dp." | ||
| | ||
Line 61: | Line 63: | ||
AND fo.data_object = dp." | AND fo.data_object = dp." | ||
AND dp." | AND dp." | ||
+ | |||
</ | </ | ||
Line 77: | Line 80: | ||
AND fo.data_object = dp." | AND fo.data_object = dp." | ||
AND dp." | AND dp." | ||
+ | |||
</ | </ | ||
Line 85: | Line 89: | ||
Once you have installed the client, set up your user name and password. These are the same as for MoM. Remember that this is just a different interface to the LTA catalogue: you will need the same credentials as for the web interface. | Once you have installed the client, set up your user name and password. These are the same as for MoM. Remember that this is just a different interface to the LTA catalogue: you will need the same credentials as for the web interface. | ||
- | After installing the LTA client, the file .awe/ | + | After installing the LTA client, the file .awe/ |
+ | < | ||
[global] | [global] | ||
database_user | database_user | ||
database_password | database_password | ||
+ | |||
</ | </ | ||
Line 120: | Line 127: | ||
pprint(result) | pprint(result) | ||
+ | |||
</ | </ | ||
Line 134: | Line 142: | ||
Pointing found RA 95.561458 DEC 24.584056 | Pointing found RA 95.561458 DEC 24.584056 | ||
..etc.. | ..etc.. | ||
+ | |||
</ | </ | ||
Line 151: | Line 160: | ||
Observation | Observation | ||
from awlofar.toolbox.LtaStager import LtaStager, LtaStagerError | from awlofar.toolbox.LtaStager import LtaStager, LtaStagerError | ||
+ | |||
+ | |||
</ | </ | ||
The lines above must be added to each of the scripts below for these to work. | The lines above must be added to each of the scripts below for these to work. | ||
- | This simple | + | === Ex: get staging URI's === |
+ | |||
+ | This script will allow you to find all data within a single project, for example LC2_035. Please change the project name to the code of a project of yours. If you also want to stage the data you found, just set the do_stage variable to True. Be careful with how many files you stage and what size they have: the same limits as for the web interface apply here. | ||
<code python> | <code python> | ||
Line 195: | Line 208: | ||
stager = LtaStager() | stager = LtaStager() | ||
stager.stage_uris(uris) | stager.stage_uris(uris) | ||
+ | |||
</ | </ | ||
+ | |||
+ | === Ex: filter on subbands === | ||
The following script will find subbands 301 and 302 for all targets within two different projects. | The following script will find subbands 301 and 302 for all targets within two different projects. | ||
Line 251: | Line 267: | ||
for project in (project1, project2) : | for project in (project1, project2) : | ||
stager.stage_uris(uris[project]) | stager.stage_uris(uris[project]) | ||
+ | |||
+ | |||
</ | </ | ||
+ | |||
+ | === Ex: filter on frequency and observation date === | ||
Here, we find data between freq1 and freq2 taken within one project between day1 and day2 | Here, we find data between freq1 and freq2 taken within one project between day1 and day2 | ||
Line 299: | Line 319: | ||
stager = LtaStager() | stager = LtaStager() | ||
stager.stage_uris(uris) | stager.stage_uris(uris) | ||
+ | |||
</ | </ | ||
- | == Example; | + | === Ex: query public data === |
Querying public data in projects you are not member of. First set project ALL, then construct a query and optionally limit the query to a certain project : | Querying public data in projects you are not member of. First set project ALL, then construct a query and optionally limit the query to a certain project : | ||
Line 312: | Line 333: | ||
print(len(query)) | print(len(query)) | ||
# 1800 | # 1800 | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | === Ex: get release dates === | ||
+ | |||
+ | <code python> | ||
+ | from awlofar.main.aweimports import Observation, | ||
+ | from common.database.Context import context | ||
+ | |||
+ | project = ' | ||
+ | |||
+ | # Query for private data of the project, you must be member of the project | ||
+ | private_data = True | ||
+ | |||
+ | # To see private data of this project, you must be member of this project | ||
+ | if private_data : | ||
+ | context.set_project(project) | ||
+ | if project != context.get_current_project().name: | ||
+ | raise Exception(" | ||
+ | |||
+ | # Observations | ||
+ | query_observations = Observation.select_all().project_only(project) | ||
+ | for observation in query_observations : | ||
+ | print(" | ||
+ | |||
+ | # Pipelines | ||
+ | query_pipelines = PipelineRun.select_all().project_only(project) | ||
+ | for pipeline in query_pipelines : | ||
+ | print(" | ||
+ | |||
+ | # Data products | ||
+ | query_products = DataProduct.select_all().project_only(project) | ||
+ | query_products &= DataProduct.isValid == 1 | ||
+ | for product in query_products : | ||
+ | print(" | ||
+ | |||
+ | |||
</ | </ | ||
===== Python Module for Staging ===== | ===== Python Module for Staging ===== | ||
- | The python interaction with the LTA catalog can be complemented with the use of a specific module developed to give users more control over their staging requests. | + | The python interaction with the LTA catalog can be complemented with the use of a specific module developed to give users more control over their staging requests. The module is made available **[[https:// |
+ | |||
+ | **Alternatively to the .awe/ | ||
+ | < | ||
- | **Alternatively to the .awe/ | ||
user=XXX | user=XXX | ||
password=YYY | password=YYY | ||
+ | |||
</ | </ | ||
Line 357: | Line 419: | ||
- Status | - Status | ||
- User id -> 1919 | - User id -> 1919 | ||
+ | |||
</ | </ | ||