DEDL - HDA Tutorial - Queryables

DEDL - HDA Tutorial - Queryables#

Author: EUMETSAT
Copyright: 2024 EUMETSAT
Licence: MIT

How to use the queryables API

The queryables API returns a list of variable terms that can be used for filtering the specified collection.

The queryables API presents the appropriate filters for the selected dataset, determined by the chosen values. If the user selects a certain variable, the choice is narrowed down for other variables. This notebook demonstrates how to filter data in a specific collection using the list of variable terms returned by the queryables API.

The notebook is dedicated to collections from C3S and DestinE digital twins. C3S and DestinE digital twins datasets offer a wide range of possibilities in terms of information and constraints and the queryables API is a valuable tool for exploring these datasets.

Throughout this notebook, you will learn:

Authenticate: How to authenticate for searching and access DEDL collections.
Queryables: How to exploit the STAC API filter extension features. The “queryables” API helps users to determine the property names and types available for filtering data.
Search data: How to search DEDL data using filters obtained by the “queryables” API.
Download data: How to download DEDL data through HDA.

The detailed HDA API and definition of each endpoint and parameters is available in the HDA Swagger UI at: STAC API - Filter Extension

Prequisites:

For queryables API: none

For filtering data inside collections : DestinE user account

Authenticate#

Define some constants for the API URLs#

In this section, we define the relevant constants, holding the URL strings for the different endpoints.

# Collection https://hda.data.destination-earth.eu/ui/dataset/EO.ECMWF.DAT.CAMS_EUROPE_AIR_QUALITY_FORECASTS as default
COLLECTION_ID = "EO.ECMWF.DAT.CAMS_EUROPE_AIR_QUALITY_FORECASTS"

# Core API
HDA_API_URL = "https://hda.data.destination-earth.eu"

# STAC API
## Core
STAC_API_URL = f"{HDA_API_URL}/stac"

## Item Search
SEARCH_URL = f"{STAC_API_URL}/search"

## Collections
COLLECTIONS_URL = f"{STAC_API_URL}/collections"

## Queryables
QUERYABLES_URL = f"{STAC_API_URL}/queryables"
QUERYABLES_BY_COLLECTION_ID = f"{COLLECTIONS_URL}/{COLLECTION_ID}/queryables"
HDA_FILTERS =''

## HTTP Success
HTTP_SUCCESS_CODE = 200

Import the relevant modules and define some functions#

We start off by importing the relevant modules for DestnE authentication, HTTP requests, json handling, widgets and some utility.

pip install --quiet --upgrade destinelab

Note: you may need to restart the kernel to use updated packages.

import destinelab as deauth

pip install DEDLUtils

Collecting DEDLUtils
  Downloading dedlutils-0.0.2-py3-none-any.whl.metadata (703 bytes)
Requirement already satisfied: ipython in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from DEDLUtils) (8.27.0)
Requirement already satisfied: ipywidgets in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from DEDLUtils) (8.1.5)
Requirement already satisfied: json5 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from DEDLUtils) (0.9.25)
Requirement already satisfied: requests in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from DEDLUtils) (2.32.3)
Requirement already satisfied: rich in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from DEDLUtils) (13.8.1)
Requirement already satisfied: urllib3 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from DEDLUtils) (1.26.19)
Requirement already satisfied: decorator in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (0.19.1)
Requirement already satisfied: matplotlib-inline in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (0.1.7)
Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (3.0.47)
Requirement already satisfied: pygments>=2.4.0 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (2.18.0)
Requirement already satisfied: stack-data in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (0.6.2)
Requirement already satisfied: traitlets>=5.13.0 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (5.14.3)
Requirement already satisfied: typing-extensions>=4.6 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (4.12.2)
Requirement already satisfied: pexpect>4.3 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipython->DEDLUtils) (4.9.0)
Requirement already satisfied: comm>=0.1.3 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipywidgets->DEDLUtils) (0.2.2)
Requirement already satisfied: widgetsnbextension~=4.0.12 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipywidgets->DEDLUtils) (4.0.13)
Requirement already satisfied: jupyterlab-widgets~=3.0.12 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from ipywidgets->DEDLUtils) (3.0.13)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from requests->DEDLUtils) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from requests->DEDLUtils) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from requests->DEDLUtils) (2024.8.30)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from rich->DEDLUtils) (3.0.0)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from jedi>=0.16->ipython->DEDLUtils) (0.8.4)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich->DEDLUtils) (0.1.2)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from pexpect>4.3->ipython->DEDLUtils) (0.7.0)
Requirement already satisfied: wcwidth in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython->DEDLUtils) (0.2.13)
Requirement already satisfied: executing>=1.2.0 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from stack-data->ipython->DEDLUtils) (2.1.0)
Requirement already satisfied: asttokens>=2.1.0 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from stack-data->ipython->DEDLUtils) (2.4.1)
Requirement already satisfied: pure-eval in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from stack-data->ipython->DEDLUtils) (0.2.3)
Requirement already satisfied: six>=1.12.0 in /opt/conda/envs/python_dedl/lib/python3.11/site-packages (from asttokens>=2.1.0->stack-data->ipython->DEDLUtils) (1.16.0)
Downloading dedlutils-0.0.2-py3-none-any.whl (4.6 kB)
Installing collected packages: DEDLUtils
Successfully installed DEDLUtils-0.0.2
Note: you may need to restart the kernel to use updated packages.

from DEDLUtils import dedl_utilities
#import dedl_utilities

import requests
import json
from getpass import getpass

import ipywidgets as widgets
from IPython.display import display, clear_output, HTML
from ipywidgets import Layout, Box
import datetime

from rich.console import Console
import rich.table

from IPython.display import JSON

Obtain Authentication Token#

To perform a query on HDA we need to be authenticated.

DESP_USERNAME = input("Please input your DESP username or email: ")
DESP_PASSWORD = getpass("Please input your DESP password: ")

auth = deauth.AuthHandler(DESP_USERNAME, DESP_PASSWORD)
access_token = auth.get_token()
if access_token is not None:
    print("DEDL/DESP Access Token Obtained Successfully")
else:
    print("Failed to Obtain DEDL/DESP Access Token")

auth_headers = {"Authorization": f"Bearer {access_token}"}

Response code: 200
DEDL/DESP Access Token Obtained Successfully

Queryables#

The “queryables” API helps users to determine the property names and types available for filtering data inside a specific collection.

Below a dropdown menu to choose the collection. We can choose the collection of which we want to inspect the filters.

prefix = "EO.ECMWF"

dedlUtils=dedl_utilities.DEDLUtilities(COLLECTION_ID )
dedlUtils.create_collections_dropdown(prefix)

# Layout
# Define the layout for the dropdown
dropdown_layout = Layout(display='space-between', justify_content='center', width='90%')
# Create a box to hold the dropdown with the specified layout
box = Box([dedlUtils.dropdown, dedlUtils.outputArea], layout=dropdown_layout)
display( box)  

The QUERYABLES ENDPOINT (above) returns the applicable filters under the section named ‘properties’.

The ‘properties’ section contains

the name of the filter, description,
the filter type,
the possible filter values, enum (conditioned by the values selected for the other filters)
and the the default (or chosen) value applied

We can print the’properties’ section for the selected collection in the table below. The table shows the filters and the values applied by default when we perform a search for the chosen dataset without specifying any filter.

filters_resp=requests.get(dedlUtils.queryablesByCollectionId)

filters = filters_resp.json()["properties"]
dedlQueryablesUtils=dedl_utilities.DEDLQueryablesUtilities(dedlUtils.collectionId)
console = Console()
console.print(dedlQueryablesUtils.create_queryables_table(filters))

                                                Applicable filters                                                 
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃           Description ┃     Type ┃                               enum ┃                                   value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│                  BBox │  "array" │                                    │                                         │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│                format │ "string" │                                    │                                   "zip" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│      api_product_type │ "string" │          climate_impact_indicators │             "climate_impact_indicators" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│              variable │ "string" │                 2m_air_temperature │                    "2m_air_temperature" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│         variable_type │ "string" │ absolute_change_from_reference_pe… │ "absolute_change_from_reference_period" │
│                       │          │                  , absolute_values │                                         │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│       processing_type │ "string" │          bias_corrected , original │                              "original" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│      time_aggregation │ "string" │         annual_mean , monthly_mean │                          "monthly_mean" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│ horizontal_resolution │ "string" │                0_11_degrees , 5_km │                                  "5_km" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│            experiment │ "string" │                    degree_scenario │                       "degree_scenario" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│                   rcm │ "string" │       cclm4_8_17 , racmo22e , rca4 │                            "cclm4_8_17" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│                   gcm │ "string" │                           ec_earth │                              "ec_earth" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│       ensemble_member │ "string" │                            r12i1p1 │                               "r12i1p1" │
├───────────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────────┤
│                period │ "string" │              1_5_c , 2_0_c , 3_0_c │                                 "1_5_c" │
└───────────────────────┴──────────┴────────────────────────────────────┴─────────────────────────────────────────┘

Calling the queryables API specifying filters, that means using as parameters the values chosen for filtering the selected dataset, the API replies with the applicable filters, conditioned by the chosen values. Then if the user selects a certain value for a parameter then the choice is narrowed down for other variables.

The queryables API, in this way, helps user to build a correct search request for the given dataset.

Below an interactive example, to see that once you select a value for a property the choice is narrowed down for other variables.

# Event listeners
dedlQueryablesUtils.update_queryables_dropdowns()
display(dedlQueryablesUtils.dropdownContainer, dedlQueryablesUtils.outputArea)

Filtering a collection with the list returned by the queryable API#

This section wil show how to use the list of variable terms returned by the queryables API for filtering a specific dataset.

If you choose a digital twins collection, check if the access is granted#

If DT access is not granted, you will not be able to search and access DT data.

auth.is_DTaccess_allowed(access_token)

DT Output access allowed

True

Build the query from the selected values#

The parameters chosen in the previous steps can be used to build the corresponding HDA queries.

# The JSON objects containing the generic query parameters:
jsonGenericQuery = '{"collections": ["'+dedlUtils.collectionId+'"], "datetime": "2024-04-01T00:00:00Z/2024-04-19T00:00:00Z"}'
# Convert JSON strings to Python dictionaries
dictQuery = json.loads(jsonGenericQuery)

# Include the filters selected in the previous steps inside the JSON containing the generic query parameters:
dictQuery['query'] = dedlQueryablesUtils.hdaFilters
# Convert the merged dictionary back to a JSON string
queryJson = json.dumps(dictQuery, indent=4)

print(queryJson)

{
    "collections": [
        "EO.ECMWF.DAT.SIS_HYDROLOGY_METEOROLOGY_DERIVED_PROJECTIONS"
    ],
    "datetime": "2024-04-01T00:00:00Z/2024-04-19T00:00:00Z",
    "query": {
        "api_product_type": {
            "eq": "climate_impact_indicators"
        },
        "variable": {
            "eq": "2m_air_temperature"
        },
        "variable_type": {
            "eq": "absolute_change_from_reference_period"
        },
        "processing_type": {
            "eq": "bias_corrected"
        },
        "time_aggregation": {
            "eq": "annual_mean"
        },
        "horizontal_resolution": {
            "eq": "5_km"
        },
        "experiment": {
            "eq": "degree_scenario"
        },
        "rcm": {
            "eq": "cclm4_8_17"
        },
        "gcm": {
            "eq": "ec_earth"
        },
        "ensemble_member": {
            "eq": "r12i1p1"
        },
        "period": {
            "eq": "1_5_c"
        }
    }
}

Search#

response = requests.post("https://hda.data.destination-earth.eu/stac/search", headers=auth_headers, json= json.loads(queryJson) )
#print(response)
# Requests to ADS data always return a single item containing all the requested data
product = response.json()["features"][0]
JSON(product, expanded= False)

<IPython.core.display.JSON object>

Download#

Once we have found the product we can download it:

download_url = product["assets"]["downloadLink"]["href"]
print(download_url )

https://hda.data.destination-earth.eu/stac/collections/EO.ECMWF.DAT.SIS_HYDROLOGY_METEOROLOGY_DERIVED_PROJECTIONS/items/SIS_HYDRO_MET_PROJ_20240401_20240418_2971c17e5a4a72b9efcc38f5652c950808ddea93/download?provider=copernicus_climate_data_store&_dc_qs=%257B%2522data_format%2522%253A%2B%2522zip%2522%252C%2B%2522ensemble_member%2522%253A%2B%2522r12i1p1%2522%252C%2B%2522experiment%2522%253A%2B%2522degree_scenario%2522%252C%2B%2522gcm%2522%253A%2B%2522ec_earth%2522%252C%2B%2522horizontal_resolution%2522%253A%2B%25225_km%2522%252C%2B%2522period%2522%253A%2B%25221_5_c%2522%252C%2B%2522processing_type%2522%253A%2B%2522bias_corrected%2522%252C%2B%2522product_type%2522%253A%2B%2522climate_impact_indicators%2522%252C%2B%2522rcm%2522%253A%2B%2522cclm4_8_17%2522%252C%2B%2522time_aggregation%2522%253A%2B%2522annual_mean%2522%252C%2B%2522variable%2522%253A%2B%25222m_air_temperature%2522%252C%2B%2522variable_type%2522%253A%2B%2522absolute_change_from_reference_period%2522%257D

from tqdm import tqdm
import time
import re
direct_download_url=''

response = requests.get(download_url, headers=auth_headers)
if (response.status_code == HTTP_SUCCESS_CODE):
    direct_download_url = product['assets']['downloadLink']['href']
else:
    JSON(response.json(), expanded=True)


# we poll as long as the data is not ready
if direct_download_url=='':
    while url := response.headers.get("Location"):
        print(f"order status: {response.json()['status']}")
        response = requests.get(url, headers=auth_headers, stream=True)
        response.raise_for_status()

filename = re.findall('filename=\"?(.+)\"?', response.headers["Content-Disposition"])[0]
total_size = int(response.headers.get("content-length", 0))

print(f"downloading {filename}")

with tqdm(total=total_size, unit="B", unit_scale=True) as progress_bar:
    with open(filename, 'wb') as f:
        for data in response.iter_content(1024):
            progress_bar.update(len(data))
            f.write(data)

order status: accepted
order status: accepted
order status: accepted
order status: accepted
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
order status: running
downloading 3c6eeca1bf32e5034981734bc56f8655.zip

100%|██████████| 11.4M/11.4M [00:00<00:00, 33.9MB/s]