Climate Change Adaptation Digital Twin Series

🚀 Launch in JupyterHub

Credit: Earthkit and HDA Polytope used in this context are both packages provided by the European Centre for Medium-Range Weather Forecasts (ECMWF).

DEDL Harmonised Data Access is used in this example.

Documentation DestinE Data Lake HDA

Climate DT data catalogue

Obtain Authentication Token¶

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import json
import os
from getpass import getpass
import destinelab as deauth

First, we get an access token for the API

DESP_USERNAME = input("Please input your DESP username or email: ")
DESP_PASSWORD = getpass("Please input your DESP password: ")

auth = deauth.AuthHandler(DESP_USERNAME, DESP_PASSWORD)
access_token = auth.get_token()
if access_token is not None:
    print("DEDL/DESP Access Token Obtained Successfully")
else:
    print("Failed to Obtain DEDL/DESP Access Token")

auth_headers = {"Authorization": f"Bearer {access_token}"}

Please input your DESP username or email:  eum-dedl-user
Please input your DESP password:  ········

Response code: 200
DEDL/DESP Access Token Obtained Successfully

Check if DT access is granted¶

If DT access is not granted, you will not be able to execute the rest of the notebook.

import importlib
installed_version = importlib.metadata.version("destinelab")
version_number = installed_version.split('.')[1]
if((int(version_number) >= 8 and float(installed_version) < 1) or float(installed_version) >= 1):
    auth.is_DTaccess_allowed(access_token)

DT Output access allowed

Query using the DEDL HDA API¶

Filter¶

We have to setup up a filter and define which data to obtain.

Choose a valid combination of => Activity + Experiment + Model (based on the year of interest)

Following activities/experiment/model & dates are possible:

ScenarioMIP/ssp3-7.0/ICON: Start date 20200101, 40 years
ScenarioMIP/ssp3-7.0/IFS-NEMO: Start date 20200101, 40 years

datechoice = "2028-06-10T00:00:00Z"
filters = {
    key: {"eq": value}
    for key, value in {
        "class": "d1",             # fixed 
        "dataset": "climate-dt",   # fixed climate-dt access
        "activity": "ScenarioMIP", # activity + experiment + model (go together)
        "experiment": "SSP3-7.0",  # activity + experiment + model (go together)
        "model": "IFS-NEMO",       # activity + experiment + model (go together)
        "generation": "1",         # fixed Specifies the generation of the dataset, which can be incremented as required (latest is 1)
        "realization": "1",        # fixed Specifies the climate realization. Default 1. Based on perturbations of initial conditions
        "resolution": "high",      # standard/ high 
        "expver": "0001",          # fixed experiment version 
        "stream": "clte",          # fixed climate
        "time": "0000",            # choose the hourly slot(s)
        "type": "fc",              # fixed forecasted fields
        "levtype": "sfc",          # Surface fields (levtype=sfc), Height level fields (levtype=hl), Pressure level fields (levtype=pl), Model Level (Levtype=ml)
#        "levelist": "1/2/3/...",  # for ml/pl/sol type data
        "param": "167"             # 2m Temperature parameter
    }.items()
}

Make Data Request¶

We request data, it is not available at the moment. We can see that our request is in queuedstatus.
We will poll the API until the data is ready and then download it.

Please note that the basic HDA quota allows a maximum of 4 requests per second. The following code limits polling to this quota.

pip install ratelimit --quiet

Note: you may need to restart the kernel to use updated packages.

from tqdm import tqdm
import time
import re
from datetime import datetime
from IPython.display import JSON
from ratelimit import limits, sleep_and_retry


#Sometimes requests to polytope get timeouts, it is then convenient define a retry strategy
###
retry_strategy = Retry(
    total=5,  # Total number of retries
    status_forcelist=[500, 502, 503, 504],  # List of 5xx status codes to retry on
    allowed_methods=["GET",'POST'],  # Methods to retry
    backoff_factor=1  # Wait time between retries (exponential backoff)
)

# Create an adapter with the retry strategy
adapter = HTTPAdapter(max_retries=retry_strategy)

# Create a session and mount the adapter
session = requests.Session()
session.mount("https://", adapter)


# Set limit: max 4 calls per 1 seconds
###
CALLS = 4
PERIOD = 1  # seconds
@sleep_and_retry
@limits(calls=CALLS, period=PERIOD)
def call_api(url,auth_headers):
    response = session.get(url, headers=auth_headers, stream=True)
    return response

# Define date choice and filters if needed
datechoice = "2024-07-01"

# Initialize a list to store filenames
filenames = []

# Define start and end years
start_year = 2024
start_month = 7
end_year = 2028

# Loop 
for year in range(start_year, end_year + 1):
    # Create a datetime object 
    obsdate = datetime(year, start_month, 1)
    datechoice = obsdate.strftime("%Y-%m-%dT12:00:00Z")
    response = session.post("https://hda.data.destination-earth.eu/stac/search", headers=auth_headers, json={
        "collections": ["EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION"],
        "datetime": datechoice,
        "query": filters
    })

    # Requests to EO.ECMWF.DAT.DT_CLIMATE always return a single item containing all the requested data
    # print(response.json())
    product = response.json()["features"][0]

    # DownloadLink is an asset representing the whole product
    download_url = product["assets"]["downloadLink"]["href"]
    print(download_url)
    HTTP_SUCCESS_CODE = 200
    HTTP_ACCEPTED_CODE = 202

    direct_download_url = ''

    response =session.get(download_url, headers=auth_headers)
    response.raise_for_status()
    if (response.status_code == HTTP_SUCCESS_CODE):
        direct_download_url = product['assets']['downloadLink']['href']
    elif (response.status_code != HTTP_ACCEPTED_CODE):
        JSON(response.json(), expanded=True)

    # we poll as long as the data is not ready
    if direct_download_url=='':
        while url := response.headers.get("Location"):
            print(f"order status: {response.json()['status']}")
            response = call_api(url,auth_headers)  
            response.raise_for_status()
            
            
    # Check if Content-Disposition header is present
    if "Content-Disposition" not in response.headers:
        print(response)
        print(response.json())
        raise Exception("Headers: \n"+str(response.headers)+"\nContent-Disposition header not found in response. Must be something wrong.")
    filename = re.findall('filename=\"?(.+)\"?', response.headers["Content-Disposition"])[0]
    total_size = int(response.headers.get("content-length", 0))

    print(f"downloading {filename}")

    with tqdm(total=total_size, unit="B", unit_scale=True) as progress_bar:
        with open(filename, 'wb') as f:
            for data in response.iter_content(1024):
                progress_bar.update(len(data))
                f.write(data)
    
    # Add the filename to the list
    filenames.append(filename)

https://hda.data.destination-earth.eu/stac/collections/EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION/items/DT_CLIMATE_ADAPTATION_20240701_20240701_ec9a796494eb745d6c6ddd684e5438778e13eb02/download?provider=dedt_lumi&_dc_qs=%257B%2522activity%2522%253A%2B%2522ScenarioMIP%2522%252C%2B%2522class%2522%253A%2B%2522d1%2522%252C%2B%2522dataset%2522%253A%2B%2522climate-dt%2522%252C%2B%2522date%2522%253A%2B%252220240701%252Fto%252F20240701%2522%252C%2B%2522experiment%2522%253A%2B%2522SSP3-7.0%2522%252C%2B%2522expver%2522%253A%2B%25220001%2522%252C%2B%2522generation%2522%253A%2B%25221%2522%252C%2B%2522levtype%2522%253A%2B%2522sfc%2522%252C%2B%2522model%2522%253A%2B%2522IFS-NEMO%2522%252C%2B%2522param%2522%253A%2B%2522167%2522%252C%2B%2522realization%2522%253A%2B%25221%2522%252C%2B%2522resolution%2522%253A%2B%2522high%2522%252C%2B%2522stream%2522%253A%2B%2522clte%2522%252C%2B%2522time%2522%253A%2B%25220000%2522%252C%2B%2522type%2522%253A%2B%2522fc%2522%257D
order status: queued
order status: queued
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
downloading 8bf16f4e-a6e8-49ce-bc19-817daece6981.grib

100%|██████████| 26.2M/26.2M [00:00<00:00, 49.8MB/s]

https://hda.data.destination-earth.eu/stac/collections/EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION/items/DT_CLIMATE_ADAPTATION_20250701_20250701_a02882cb88731e129d9b867641a1530950b93f22/download?provider=dedt_lumi&_dc_qs=%257B%2522activity%2522%253A%2B%2522ScenarioMIP%2522%252C%2B%2522class%2522%253A%2B%2522d1%2522%252C%2B%2522dataset%2522%253A%2B%2522climate-dt%2522%252C%2B%2522date%2522%253A%2B%252220250701%252Fto%252F20250701%2522%252C%2B%2522experiment%2522%253A%2B%2522SSP3-7.0%2522%252C%2B%2522expver%2522%253A%2B%25220001%2522%252C%2B%2522generation%2522%253A%2B%25221%2522%252C%2B%2522levtype%2522%253A%2B%2522sfc%2522%252C%2B%2522model%2522%253A%2B%2522IFS-NEMO%2522%252C%2B%2522param%2522%253A%2B%2522167%2522%252C%2B%2522realization%2522%253A%2B%25221%2522%252C%2B%2522resolution%2522%253A%2B%2522high%2522%252C%2B%2522stream%2522%253A%2B%2522clte%2522%252C%2B%2522time%2522%253A%2B%25220000%2522%252C%2B%2522type%2522%253A%2B%2522fc%2522%257D
order status: queued
order status: queued
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
downloading 50e3b5aa-4889-4236-a55e-5a7f5e6a9137.grib

100%|██████████| 24.6M/24.6M [00:00<00:00, 53.1MB/s]

https://hda.data.destination-earth.eu/stac/collections/EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION/items/DT_CLIMATE_ADAPTATION_20260701_20260701_e999192b403bd78d714acd0fa0ed83cd6392d6a4/download?provider=dedt_lumi&_dc_qs=%257B%2522activity%2522%253A%2B%2522ScenarioMIP%2522%252C%2B%2522class%2522%253A%2B%2522d1%2522%252C%2B%2522dataset%2522%253A%2B%2522climate-dt%2522%252C%2B%2522date%2522%253A%2B%252220260701%252Fto%252F20260701%2522%252C%2B%2522experiment%2522%253A%2B%2522SSP3-7.0%2522%252C%2B%2522expver%2522%253A%2B%25220001%2522%252C%2B%2522generation%2522%253A%2B%25221%2522%252C%2B%2522levtype%2522%253A%2B%2522sfc%2522%252C%2B%2522model%2522%253A%2B%2522IFS-NEMO%2522%252C%2B%2522param%2522%253A%2B%2522167%2522%252C%2B%2522realization%2522%253A%2B%25221%2522%252C%2B%2522resolution%2522%253A%2B%2522high%2522%252C%2B%2522stream%2522%253A%2B%2522clte%2522%252C%2B%2522time%2522%253A%2B%25220000%2522%252C%2B%2522type%2522%253A%2B%2522fc%2522%257D
order status: queued
order status: queued
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
downloading 1c596b44-eb38-4ff9-8ed2-fa34753935b3.grib

100%|██████████| 26.2M/26.2M [00:00<00:00, 55.1MB/s]

https://hda.data.destination-earth.eu/stac/collections/EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION/items/DT_CLIMATE_ADAPTATION_20270701_20270701_949418df1ab80ec427f06fa759a8940dde2d6ee7/download?provider=dedt_lumi&_dc_qs=%257B%2522activity%2522%253A%2B%2522ScenarioMIP%2522%252C%2B%2522class%2522%253A%2B%2522d1%2522%252C%2B%2522dataset%2522%253A%2B%2522climate-dt%2522%252C%2B%2522date%2522%253A%2B%252220270701%252Fto%252F20270701%2522%252C%2B%2522experiment%2522%253A%2B%2522SSP3-7.0%2522%252C%2B%2522expver%2522%253A%2B%25220001%2522%252C%2B%2522generation%2522%253A%2B%25221%2522%252C%2B%2522levtype%2522%253A%2B%2522sfc%2522%252C%2B%2522model%2522%253A%2B%2522IFS-NEMO%2522%252C%2B%2522param%2522%253A%2B%2522167%2522%252C%2B%2522realization%2522%253A%2B%25221%2522%252C%2B%2522resolution%2522%253A%2B%2522high%2522%252C%2B%2522stream%2522%253A%2B%2522clte%2522%252C%2B%2522time%2522%253A%2B%25220000%2522%252C%2B%2522type%2522%253A%2B%2522fc%2522%257D
order status: queued
order status: queued
order status: queued
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
downloading 4254a2e5-b002-42f5-8b13-5ad1da886412.grib

100%|██████████| 26.2M/26.2M [00:00<00:00, 47.5MB/s]

https://hda.data.destination-earth.eu/stac/collections/EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION/items/DT_CLIMATE_ADAPTATION_20280701_20280701_686ac060140c939aed16cb1c87cd6e8ae6280b9d/download?provider=dedt_lumi&_dc_qs=%257B%2522activity%2522%253A%2B%2522ScenarioMIP%2522%252C%2B%2522class%2522%253A%2B%2522d1%2522%252C%2B%2522dataset%2522%253A%2B%2522climate-dt%2522%252C%2B%2522date%2522%253A%2B%252220280701%252Fto%252F20280701%2522%252C%2B%2522experiment%2522%253A%2B%2522SSP3-7.0%2522%252C%2B%2522expver%2522%253A%2B%25220001%2522%252C%2B%2522generation%2522%253A%2B%25221%2522%252C%2B%2522levtype%2522%253A%2B%2522sfc%2522%252C%2B%2522model%2522%253A%2B%2522IFS-NEMO%2522%252C%2B%2522param%2522%253A%2B%2522167%2522%252C%2B%2522realization%2522%253A%2B%25221%2522%252C%2B%2522resolution%2522%253A%2B%2522high%2522%252C%2B%2522stream%2522%253A%2B%2522clte%2522%252C%2B%2522time%2522%253A%2B%25220000%2522%252C%2B%2522type%2522%253A%2B%2522fc%2522%257D
order status: queued
order status: queued
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
order status: processing
downloading 2c7595df-a665-46ac-967a-391be6100490.grib

100%|██████████| 26.3M/26.3M [00:00<00:00, 48.3MB/s]

EarthKit¶

Lets plot the result file [EarthKit Documentation] https://earthkit-data.readthedocs.io/en/latest/index.html

This section requires that you have ecCodes >= 2.35 installed on your system.
You can follow the installation procedure at https://confluence.ecmwf.int/display/ECC/ecCodes+installation

import earthkit.data
import earthkit.maps
import earthkit.regrid

# Iterate over filenames
for filename in filenames:
    print(filename)  # For example, print each filename
    data = earthkit.data.from_source("file", filename)
    data.ls
    earthkit.maps.quickplot(data,#style=style
                       )

8bf16f4e-a6e8-49ce-bc19-817daece6981.grib

50e3b5aa-4889-4236-a55e-5a7f5e6a9137.grib

1c596b44-eb38-4ff9-8ed2-fa34753935b3.grib

4254a2e5-b002-42f5-8b13-5ad1da886412.grib

2c7595df-a665-46ac-967a-391be6100490.grib