EODAG - DestinE Data Lake Provider - DEDL Notebook Gallery

Prerequisites:

To search and access DEDL data a DestinE user account is needed

References:

Credit:

Code used in this context takes inspiration from the Python API User Guide produced by CS Group.

EODAG is a command line tool and a Python package for searching and downloading earth observation data via a unified API.

This tutorial demonstrates how to use EODAG to search and access DEDL data. The notebook uses the DEDL provider in EODAG to access data via Python code.

Setup: EODAG configuration to use the provider DEDL .
Search: search DEDL data, we search for Sentinel-3 data.
Filter: filter DEDL data.
Download: download DEDL data.

The complete guide on how to use EODAG Python API is available via https://eodag.readthedocs.io/en/stable/api_user_guide.html.

Note:

Please note that the two factor authentication (2FA) is still not implemented in EODAG. The users who have enabled 2FA on DESP will not be able to run this notebook.

Setup¶

In this section, we set:

The output_dir, the directory where to store downloaded products.
The DEDL credentials, you’ll be asked to enter your DEDL credentials.
The search timeout, it is of 60 seconds to avoid any unexpected errors because of long running search queries.

pip install --user --quiet --upgrade eodag

Note: you may need to restart the kernel to use updated packages.

import os
from getpass import getpass

workspace = 'eodag_workspace'
if not os.path.isdir(workspace):
    os.mkdir(workspace)
    
os.environ["EODAG__DEDL__DOWNLOAD__OUTPUT_DIR"] = os.path.abspath(workspace)
#os.environ["EODAG__DEDL__DOWNLOAD__OUTPUTS_PREFIX"] = os.path.abspath(workspace)

os.environ["EODAG__DEDL__PRIORITY"]="10"
os.environ["EODAG__DEDL__SEARCH__TIMEOUT"]="60"


DESP_USERNAME = input("Please input your DESP username or email: ")
DESP_PASSWORD = getpass("Please input your DESP password: ")

os.environ["EODAG__DEDL__AUTH__CREDENTIALS__USERNAME"]=DESP_USERNAME
os.environ["EODAG__DEDL__AUTH__CREDENTIALS__PASSWORD"]=DESP_PASSWORD

Please input your DESP username or email:  eum-dedl-user
Please input your DESP password:  ········

Import EODAG and list available products on DEDL¶

We now need to import the EODataAccessGateway class. The class is going to take care of all the following operations.

We can start listing the products available using dedl as provider.


from eodag import EODataAccessGateway, setup_logging
setup_logging(0)
dag = EODataAccessGateway()

options=[item.id for item in dag.list_collections("dedl")]

for pt in dag.list_collections("dedl"):
    print(f'{pt.id} - {pt.title}')

Fetching long content....

Search¶

To search we use the search method passing the ID of our dataset of interest and a geo-time filter.

The search method returns a SearchResult object that stores the products obtained from a given page (default: page=1) and a given maximum number of items per page (default: items_per_page=20).

In the following cell, we change the default value of items_per_page and define the search criteria to retrieve Sentinel-2 MSI Level-2 images over Sicily, first days of July 2024. Our goal is to check whether any effects of Mount Etna’s eruptions during that period are visible in the Sentinel-2 imagery.

search_criteria = {
    "collection": "EO.ESA.DAT.SENTINEL-2.MSI.L2A",
    "datetime": "2024-07-04T07:00:00.00Z/2024-07-08T07:00:00.00Z",
    "limit":50,
    "bbox": [12,37,16,39],
    "count": True
}

products_first_page = dag.search(**search_criteria)

Results are stored in a ‘SearchResult’ object that contains the details on the single search result.

products_first_page

It is possible to list the metadata associated with a certain product, we choose the first one returned [0], and look into it.

one_product = products_first_page[0]
one_product.properties.keys()

dict_keys(['constellation', 'datetime', 'end_datetime', 'id', 'instruments', 'platform', 'published', 'start_datetime', 'title', 'updated', 'dedl:providers', 'dedl:type', 'dedl:uid', 'eo:cloud_cover', 'eodag:default_geometry', 'eodag:download_link', 'external_fdp:sourceProduct', 'external_fdp:sourceProductOriginDate', 'file:checksum', 'file:size', 'grid:code', 'order:status', 'processing:datetime', 'processing:level', 'processing:version', 'product:type', 's2:datastrip_id', 's2:datatake_id', 's2:datatake_type', 's2:tile_id', 'sat:absolute_orbit', 'sat:relative_orbit'])

one_product.properties['eo:cloud_cover']

0.000265

Filter¶

EODAG can filter the search result. We can then refine our initial search without asking the provider again. Products can be filtered according to their properties or also with finer geometry filters.

The following example shows how to filter products to keep only those whose cloud coverage is less than 20%. And then restrict the results to products containing a smaller area over the mount Etna.

Let’s define now a smaller area around the mount Etna and a function to see the area on a map together with the results

from eodag.crunch import FilterProperty
from eodag.crunch import FilterOverlap
import shapely
import folium
from shapely.geometry import Polygon

small_geom = Polygon([[15.1, 37.7], [15.5, 37.7], [15.1, 37.75], [15.1, 37.75], [15.1, 37.7]])

smaller_area = {"lonmin": 15.1, "latmin": 37.7, "lonmax": 15.5, "latmax": 37.75}

search_geometry = shapely.geometry.box(
    smaller_area["lonmin"],
    smaller_area["latmin"],
    smaller_area["lonmax"],
    smaller_area["latmax"],
)


def create_search_result_map(search_results, extent):
    """Small utility to create an interactive map with folium
    that displays an extent in red and EO Producs in blue"""
    fmap = folium.Map([38, 14], zoom_start=7,
            width=1000,
            height=500)
    folium.GeoJson(
        extent,
        style_function=lambda x: dict(color="red")
    ).add_to(fmap)
    folium.GeoJson(
        search_results
    ).add_to(fmap)
    return fmap

# Crunch the results
filtered_results = products_first_page.crunch(FilterProperty({"eo:cloud_cover": 20, "operator" : "lt"}))

print(f"Got now {len(filtered_results)} products after filtering by cloudCover.")

Got now 22 products after filtering by cloudCover.

filtered_products = filtered_results.crunch(
    FilterOverlap(dict(contains=True)),
    geometry=small_geom
)
print(f"Got now {len(filtered_products)} products after filtering by geometry.")

Got now 1 products after filtering by geometry.

Let’s use the function defined to see the area defined on a map (red) together with the initial results (blue) filtered by cloud coverage and geometry (green).

fmap = create_search_result_map(products_first_page, search_geometry)
# Create a layer that represents the filtered products in green
folium.GeoJson(
    filtered_products,
    style_function=lambda x: dict(color="green")
).add_to(fmap)
fmap

Download¶

EOProducts can be downloaded individually. The last one is going to be downloaded.

product_to_download = filtered_products[-1]
product_path = dag.download(product_to_download)
product_path

'/home/jovyan/dev-branch/DestinE-DataLake-Lab/HDA/EODAG/eodag_workspace/S2A_MSIL2A_20240707T094041_N0510_R036_T33SWB_20240707T131659'

The quicklook shows effectively the ash plume caused by the eruptions.

import matplotlib.pyplot as plt
import matplotlib.image as mpimg

fig = plt.figure(figsize=(20, 40))
quicklook_path=product_path+"/"+os.path.basename(product_to_download.location)+"-ql.jpg" 
quicklook_path

'/home/jovyan/dev-branch/DestinE-DataLake-Lab/HDA/EODAG/eodag_workspace/S2A_MSIL2A_20240707T094041_N0510_R036_T33SWB_20240707T131659/S2A_MSIL2A_20240707T094041_N0510_R036_T33SWB_20240707T131659-ql.jpg'

<Figure size 2000x4000 with 0 Axes>


ig, ax = plt.subplots(figsize=(8, 8))

img = mpimg.imread(quicklook_path)

ax.imshow(img)
ax.set_title(
    f"{filtered_products[-1].properties['datetime']}\nTile: {filtered_products[-1].properties['s2:tile_id']}"
)
ax.axis("off")

plt.show()