HDA Tutorial - Quick start - DEDL Notebook Gallery

Prerequisites:

To search and access DEDL data a DestinE user account is needed

References:

DestinE Data Lake (DEDL) Harmonized Data Access (HDA) documentation

This notebook demonstrates how to use the HDA (Harmonized Data Access) API by sending a few HTTP requests to the API, using Python code. The tutorial shows the first steps in using Harmonised Data to discover and access the data offered by the DestinE Data Portfolio

hroughout this quickstart notebook, you will learn:

Discover: How to discover DEDL collections and services through HDA.
Authenticate: How to authenticate for searching and access DEDL collections.
Search data: How to search DEDL data through HDA.
Download data: How to download DEDL data through HDA.

The detailed API and definition of each endpoint and parameters is available in the HDA Swagger UI at: https://hda.data.destination-earth.eu/docs/

Discover¶

Import the relevant modules¶

import requests
import json
from getpass import getpass
from tqdm import tqdm
import time
import re

import destinelab as deauth

from IPython.display import JSON

Define some constants for the API URL¶

# Define the collection to be used
COLLECTION_ID = "EO.EUM.DAT.SENTINEL-3.OL_2_WRR___"

# Core API
HDA_API_URL = "https://hda.data.destination-earth.eu"

# STAC API
## Core
STAC_API_URL = f"{HDA_API_URL}/stac/v2"

## Collections
COLLECTIONS_URL = f"{STAC_API_URL}/collections"
COLLECTION_BY_ID_URL = f"{COLLECTIONS_URL}/{COLLECTION_ID}"

## Items
COLLECTION_ITEMS_URL = f"{COLLECTIONS_URL}/{COLLECTION_ID}/items"

## Item Search
SEARCH_URL = f"{STAC_API_URL}/search"

## HTTP Success
HTTP_SUCCESS_CODE = 200

Discover data¶

Below an example for discovering collections concerning Chlorophyll-a Concentration and algal pigment.

response = requests.get(COLLECTIONS_URL,params = {"q": "Chlorophyll-a Concentration,algal pigment"})

JSON(response.json(), expanded=False)

Authenticate¶

DESP_USERNAME = input("Please input your DESP username or email: ")
DESP_PASSWORD = getpass("Please input your DESP password: ")

auth = deauth.AuthHandler(DESP_USERNAME, DESP_PASSWORD)
access_token = auth.get_token()
if access_token is not None:
    print("DEDL/DESP Access Token Obtained Successfully")
else:
    print("Failed to Obtain DEDL/DESP Access Token")

auth_headers = {"Authorization": f"Bearer {access_token}"}

Search¶

Once selected a collection it is possible to search for items that match the specified input filters and order the results.

The collection used for this tutorial is OLCI Level 2 Ocean Colour Reduced Resolution - Sentinel-3

BODY = {
    "collections": [
        COLLECTION_ID,
    ],
    "datetime" : "2024-09-08T00:00:00Z/2024-09-09T23:59:59Z",
    "bbox": [-11,35,
              50,72 ],
    "sortby": [{"field": "datetime","direction": "desc"}
              ],
    "limit": 3,
}

r=requests.post(SEARCH_URL, json=BODY, headers=auth_headers)
if(r.status_code!= 200):
    (print(r.text))
r.raise_for_status()
JSON(r.json(), expanded=False)

Download¶

Once obtained the search results we can download the returned data. The code below is to download the first data returned by our search.

#select the first item in the result to download
product = r.json()["features"][0]

# DownloadLink is an asset representing the whole product
download_url = product["assets"]["downloadLink"]["href"]
ITEM_ID = product["id"]

response = requests.get(download_url,stream=True,headers=auth_headers)

# If the request was successful, download the file
if (response.status_code == HTTP_SUCCESS_CODE):
        print("Downloading ...")
        filename = ITEM_ID + ".zip"
        with open(filename, 'wb') as f:
            for chunk in response.iter_content(chunk_size=1024): 
                if chunk:
                    f.write(chunk)
                    f.flush()
        print("The dataset has been downloaded to: {}".format(filename))
else: print("Request Unsuccessful! Error-Code: {}".format(response.status_code))