WHY JUPYTER NOTEBOOKS?

 

Jupyter Notebooks is a great way to explore data and process it on WEkEO.

You can work with Jupyter Notebooks directly from the WEkEO portal instead of using a virtual machine (VM). The Harmonised Data Access (HDA) API can be used from the Jupyter Notebooks enabling you to query and access data for your processing needs.

 

The Jupyter Notebooks Workspace

 

First, login to WEkEO and go to the top right side of the screen. Select your username and open the drop-down menu to find the "JupyterHub" option.

 

JupyterHub option under username on the mainpage

 

Once you are in the JupyterHub homepage, click on the green button to select "My Server". The server is started so that you can access your Jupyter Notebooks workspace.

Select My Server button

 

In the JupyterHub there are some default folders and files available for all the users. You can also create new files and folders to organize your workspace.

 

  • Default files and folders

Bear in mind that the default folders and files cannot be deleted as you will need them to download datasets. Read the section "Harmonised Data Access (HDA) API in Jupyter Notebooks" to learn more about the procedure to download the datasets.

 

Jupyter Notebooks workspace

 

- samples: this folder contains the file "How_To_Guide-Harmonized_Data_Access-vX.Y.Z.ipynb" which is required to query and access datasets using the Harmonised Data Access (HDA) API.
- products: all the downloaded datasets using Jupyter Notebooks will be stored in this folder.
 

 

  • New files

You can create new folders to store the downloaded datasets and other Jupyter Notebook files (*.ipynb), -that you might need if you want to work online with the data. These files and folders will only be visible for you and the other WEkEO users will not have access to them.

 

 

Harmonised Data Access (HDA) API in Jupyter Notebooks

 

This section provides insights into the "How_To_Guide-Harmonized_Data_Access-vX.Y.Z.ipynb" notebook that demonstrates the use of HDA API for querying and accessing datasets.

In order to download the data, you can run each of the cells (sections) separately or you can run all the steps of the notebook using the button "Run" on the upper menu. There are some parts of the Jupyter Notebooks that need to be modified according to the data that will be downloaded. These changes are only needed in step 1 and step 6.

Steps 1 to 7 are required to download the data. Additionally, depending on how you want to download the data, you might continue with step 8 or with steps 9 and 10.

 

  • To download data locally, go to Step 8: Get Results Download Link
  • To download data in streaming mode, go to Step 9: Download to your JupyterNotebooks workspace. This means that your data will be downloaded in the JupyterHub server, but not in your computer. If you want to plot these data in the same Jupyter Notebooks you can go to Step 10: Access and plot your data.

 

These options are not exclusive, you might like to do both, and you can do it by running steps 8, 9 and 10.

 

Step 1: Initialization - WEkEO Endpoints and API Key

In this section, the WEkEO HDA endpoints are set. An example dataset is given in the dataset ID and it should be changed by the user according to the dataset of interest (selected in blue).

step1

 

Step 2: Find your API Key

To find your API Key go to the top right part of the screen, on the main page of WEkEO. Click your username, open the drop-down menu and select API Subscriptions.

 

main page API Subscriptions

 

Once you are on the API Subscriptions page, click the button "Show hidden keys". Go to the section "User credentials" and copy your encoded key. You can find this encoded key that it is replaced by the 'Base64(consumer-key:consumer-secret). 

API subscription keys

 

Go back to the Jupyter sample notebook. In section 2, paste the encoded API key in the field #api_key = "<API encoded key>".

api key empty

 

Remember to uncomment the line by deleting the #.

api key
 

Step 3: Get Access Token

Run this cell to get the token and proceed to the next step. The access token is valid for one hour.

step3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Step 4: Query Metadata

Once the access token is available, you can query the dataset of interest. Run this cell and proceed to step 5.

step4
 

Step 5: Accept Terms and Conditions

Run this cell to accept the Copernicus terms and conditions. You can then proceed to the following step.

step5

 
 
 
 
 
 
 

 

 

Step 6: Query your Dataset products

In this section, the user has to change the data inside the blue square in the screenshot bellow.

step6

These data can be obtained from the HDA option in the Dataset Navigator. There, select the parameters of your interest and open the tab at the bottom, "Request Payload". You can copy the data of the payload to the area selected in blue in the Jupyter Notebooks screenshot (above).

dn_jn_1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

dn_jn_2

 

dn_jn_3

Step 7: Check job status

After running this cell, you should obtain a successfully response about a completed job. Sometimes, depending on the data to be downloaded, it takes longer for the job to finish and some 'False' messages might appear instead of a 'True'. This is not a problem as long as a final message appears as 'True'. When the job finishes, the number of products available and the number of results that will be downloaded is shown.

step7

Step 8: Get Results List

The query results are paginated. Parameters for page number and the number of results per page can be used to fetch only the necessary results. The pages are numbered from 0 (i.e. the number of the first page is 0). In the example below, each page contains 5 results (size='5') and we are going to show the results from the 1st page (zero based numbering).

After running this cell, a list with the results that can be downloaded is shown. As explained before, you can either go to step 9 or to step 10 to download the data.

step8

 

Step 9: Get Results Download Link

If all the previous steps have worked correctly, you will see a list with all the files to download after running this cell. You can click on these links in order to download the files locally, in your computer.

step9

Step 10: Download to your Jupyter Notebooks Workspace

In this step, run the two following cells to download the data and store it in Jupyter Notebooks itself.

step10-1

step10-2

step10-3

step10-4

 

The downloaded data will be available in the folder named products, once the download of files is finished.

step10-5

 

Step 11: Access and Plot your Data

After step 10, you might want to plot the downloaded data directly in Jupyter Notebooks. You can modify the Python code on the following cells for this purpose.

step11-1

step11-2

For the example CMEMS dataset, a plot is shown for the ice concentration on the Southern hemisphere. 

step11-3

 

 

 

Harmonised Data Access (HDA) API in Jupyter Notebook video demo

This section provides insights into "How_To_Guide-Harmonized_Data_Access-vX.Y.Z.ipynb" notebook that demonstrates the use of HDA API for querying and accessing datasets.