DEDL - HDA Tutorial - quick start

DEDL - HDA Tutorial - quick start#

Author: EUMETSAT
Copyright: 2024 EUMETSAT
Licence: MIT

First steps using Harmonised Data access API

Discover Data of DestinE Data Portfolio

Access Data of DestinE Data Portfolio

This notebook demonstrates how to use the HDA (Harmonized Data Access) API by sending a few HTTP requests to the API, using Python code.

Throughout this quickstart notebook, you will learn:

Discover: How to discover DEDL collections and services through HDA.
Authenticate: How to authenticate fro searching and access DEDL collections.
Search data: How to search DEDL data through HDA.
Download data: How to download DEDL data through HDA.

The detailed API and definition of each endpoint and parameters is available in the HDA Swagger UI at:

https://hda.data.destination-earth.eu/docs/

Prerequisites:

For Data discovery: none

For Data access : DestinE user account

Discover#

Import the relevant modules#

pip install --quiet --upgrade destinelab

Note: you may need to restart the kernel to use updated packages.

import requests
import json
from getpass import getpass
from tqdm import tqdm
import time
import re

import destinelab as deauth

from IPython.display import JSON

Define some constants for the API URL#

# Define the collection to be used
COLLECTION_ID = "EO.EUM.DAT.SENTINEL-3.OL_2_WRR___"

# Core API
HDA_API_URL = "https://hda.data.destination-earth.eu"

# STAC API
## Core
STAC_API_URL = f"{HDA_API_URL}/stac"

## Collections
COLLECTIONS_URL = f"{STAC_API_URL}/collections"
COLLECTION_BY_ID_URL = f"{COLLECTIONS_URL}/{COLLECTION_ID}"

## Items
COLLECTION_ITEMS_URL = f"{COLLECTIONS_URL}/{COLLECTION_ID}/items"

## Item Search
SEARCH_URL = f"{STAC_API_URL}/search"

## HTTP Success
HTTP_SUCCESS_CODE = 200

Discover data#

Below an example for discovering collections concerning Chlorophyll-a Concentration and algal pigment.

response = requests.get(COLLECTIONS_URL,params = {"q": "Chlorophyll-a Concentration,algal pigment"})

JSON(response.json(), expanded=False)

<IPython.core.display.JSON object>

Authenticate#

DESP_USERNAME = input("Please input your DESP username or email: ")
DESP_PASSWORD = getpass("Please input your DESP password: ")

auth = deauth.AuthHandler(DESP_USERNAME, DESP_PASSWORD)
access_token = auth.get_token()
if access_token is not None:
    print("DEDL/DESP Access Token Obtained Successfully")
else:
    print("Failed to Obtain DEDL/DESP Access Token")

auth_headers = {"Authorization": f"Bearer {access_token}"}

Response code: 200
DEDL/DESP Access Token Obtained Successfully

Search#

Once selected a collection it is possible to search for items that match the specified input filters and order the results.

The collection used for this tutorial is OLCI Level 2 Ocean Colour Reduced Resolution - Sentinel-3

BODY = {
    "collections": [
        COLLECTION_ID,
    ],
    "datetime" : "2024-09-08T00:00:00Z/2024-09-09T23:59:59Z",
    "bbox": [-11,35,
              50,72 ],
    "sortby": [{"field": "datetime","direction": "desc"}
              ],
    "limit": 3,
}

r=requests.post(SEARCH_URL, json=BODY, headers=auth_headers)
if(r.status_code!= 200):
    (print(r.text))
r.raise_for_status()
JSON(r.json(), expanded=False)   

<IPython.core.display.JSON object>

Download#

Once obtained the search results we can download the returned data.

#select the first item in the result to download
product = r.json()["features"][0]

# DownloadLink is an asset representing the whole product
download_url = product["assets"]["downloadLink"]["href"]
ITEM_ID = product["id"]

response = requests.get(download_url,stream=True,headers=auth_headers)

# If the request was successful, download the file
if (response.status_code == HTTP_SUCCESS_CODE):
        print("Downloading ...")
        filename = ITEM_ID + ".zip"
        with open(filename, 'wb') as f:
            for chunk in r.iter_content(chunk_size=1024): 
                if chunk:
                    f.write(chunk)
                    f.flush()
        print("The dataset has been downloaded to: {}".format(filename))
else: print("Request Unsuccessful! Error-Code: {}".format(response.status_code))

Downloading ...
The dataset has been downloaded to: S3A_OL_2_WRR____20240909T130929_20240909T135342_20240910T163350_2653_116_352______MAR_O_NT_003.zip