ERA5 hourly data on single levels from 1940 to present
This notebook shows how to authenticate with the DestinE API, queries and downloads ERA5 single-level reanalysis data using the DEDL HDA service, and visualizes the result with EarthKit.
Prerequisites:
To search and access DEDL data a DestinE user account is needed
Earthkit used in this context is a package provided by the European Centre for Medium-Range Weather Forecasts (ECMWF).
This notebook demonstrates how to use the HDA (Harmonized Data Access) API to access ERA5 hourly data on single levels, and how to visualize the data using the Earthkit package provided by ECMWF.
The method used to access this dataset can be applied to all the datasets hosted by the climate data store provided through HDA.
The complete list of the climate data store federated datasets provided by HDA: https://
Below the main steps covered by this tutorial.
Authenticate: How to authenticate for searching and access DEDL collections.
Order data: How to order ERA5 hourly data on single levels data through HDA.
Download data: How to download hourly data on single levels data through HDA.
Visualize: How to visualize hourly data on single levels data through Earthkit.
Authenticate¶
First we import the required packages
import requests
import json
import os
import json
from getpass import getpass
from IPython.display import JSON
import destinelab as deauth
from tqdm import tqdm
import time
from urllib.parse import unquote
from time import sleepWe get an access token for the API
DESP_USERNAME = input("Please input your DESP username or email: ")
DESP_PASSWORD = getpass("Please input your DESP password: ")
auth = deauth.AuthHandler(DESP_USERNAME, DESP_PASSWORD)
access_token = auth.get_token()
if access_token is not None:
print("DEDL/DESP Access Token Obtained Successfully")
else:
print("Failed to Obtain DEDL/DESP Access Token")
auth_headers = {"Authorization": f"Bearer {access_token}"}Please input your DESP username or email: eum-dedl-user
Please input your DESP password: ········
DEDL/DESP Access Token Obtained Successfully
Response code: 200
DEDL/DESP Access Token Obtained Successfully
Order¶
Climate data store datasets need to be ordered. Below the steps to order the data of our interest.
HDA endpoint¶
HDA API is based on the Spatio Temporal Asset Catalog specification (STAC), it is convenient define a costant with its endpoint.
HDA_STAC_ENDPOINT="https://hda.data.destination-earth.eu/stac/v2"Collection discovery¶
Before querying a collection, we need its unique HDA collection ID. We can discover this ID using the HDA Discovery API by filtering on the federated provider and searching for the keywords “ERA5”, “single levels”, and “hourly data”.
response = requests.get(f"{HDA_STAC_ENDPOINT}/collections",params = {"query": '{"federation:backends": {"eq": "cop_cds"}}',"q": 'ERA5 AND "hourly data on single levels"'})
print(json.dumps(response.json(),indent=1)) {
"collections": [
{
"type": "Collection",
"title": "ERA5 hourly data on single levels from 1940 to present",
"id": "EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS",
"description": "ERA5 is the fifth generation ECMWF reanalysis for the global climate and weather for the past 8 decades. Data is available from 1940 onwards. ERA5 replaces the ERA-Interim reanalysis.\n\nReanalysis combines model data with observations from across the world into a globally complete and consistent dataset using the laws of physics. This principle, called data assimilation, is based on the method used by numerical weather prediction centres, where every so many hours (12 hours at ECMWF) a previous forecast is combined with newly available observations in an optimal way to produce a new best estimate of the state of the atmosphere, called analysis, from which an updated, improved forecast is issued. Reanalysis works in the same way, but at reduced resolution to allow for the provision of a dataset spanning back several decades. Reanalysis does not have the constraint of issuing timely forecasts, so there is more time to collect observations, and when going further back in time, to allow for the ingestion of improved versions of the original observations, which all benefit the quality of the reanalysis product.\n\nERA5 provides hourly estimates for a large number of atmospheric, ocean-wave and land-surface quantities. An uncertainty estimate is sampled by an underlying 10-member ensemble at three-hourly intervals. Ensemble mean and spread have been pre-computed for convenience. Such uncertainty estimates are closely related to the information content of the available observing system which has evolved considerably over time. They also indicate flow-dependent sensitive areas. To facilitate many climate applications, monthly-mean averages have been pre-calculated too, though monthly means are not available for the ensemble mean and spread.\n\nERA5 is updated daily with a latency of about 5 days. In case that serious flaws are detected in this early release (called ERA5T), this data could be different from the final release 2 to 3 months later. In case that this occurs users are notified.\n\nThe data set presented here is a regridded subset of the full ERA5 data set on native resolution. It is online on spinning disk, which should ensure fast and easy access. It should satisfy the requirements for most common applications.\n\nData has been regridded to a regular lat-lon grid of 0.25 degrees for the reanalysis and 0.5 degrees for the uncertainty estimate (0.5 and 1 degree respectively for ocean waves). There are four main sub sets: hourly and monthly products, both on pressure levels (upper air fields) and single levels (atmospheric, ocean-wave and land surface quantities).",
"links": [
{
"rel": "retrieve",
"type": "application/geo+json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS/order",
"title": "Retrieve",
"method": "POST"
},
{
"rel": "http://www.opengis.net/def/rel/ogc/1.0/queryables",
"type": "application/schema+json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS/queryables",
"title": "Queryables"
},
{
"rel": "items",
"type": "application/geo+json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS/items",
"title": "Items"
},
{
"rel": "self",
"type": "application/json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS",
"title": "ERA5 hourly data on single levels from 1940 to present"
},
{
"rel": "license",
"type": "application/pdf",
"href": "https://object-store.os-api.cci2.ecmwf.int/cci2-prod-catalogue/licences/licence-to-use-copernicus-products/licence-to-use-copernicus-products_b4b9451f54cffa16ecef5c912c9cebd6979925a956e3fa677976e0cf198c2c18.pdf",
"title": "Licence to Use Copernicus Products"
},
{
"rel": "cite-as",
"type": "text/html",
"href": "https://doi.org/10.24381/cds.adbb2d47",
"title": "ERA5 hourly data on single levels from 1940 to present"
},
{
"rel": "cite-as",
"type": "text/html",
"href": "https://doi.org/10.1002/qj.3803",
"title": "The ERA5 global reanalysis"
},
{
"rel": "cite-as",
"type": "text/html",
"href": "https://doi.org/10.1002/qj.4174",
"title": "The ERA5 global reanalysis: Preliminary extension to 1950"
},
{
"rel": "example",
"type": "application/x-ipynb+json",
"href": "https://raw.githubusercontent.com/destination-earth/DestinE-DataLake-Lab/refs/heads/main/HDA/CDS_data/DEDL-HDA-EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS.ipynb",
"title": "Destination Earth - ERA5 hourly data on single levels from 1940 to present - Data Access using DEDL HDA",
"application:type": "jupyter-notebook",
"application:embedded": true,
"application:language": "Python"
}
],
"assets": {
"thumbnail": {
"href": "https://object-store.os-api.cci2.ecmwf.int/cci2-prod-catalogue/resources/reanalysis-era5-single-levels/overview_c37c9fd3b18a36a2c656bb4541d37c3bb8a08d2d9ef6708227b87cb47e90a873.png",
"roles": [
"thumbnail"
],
"title": "overview",
"type": "image/png"
}
},
"extent": {
"spatial": {
"bbox": [
[
-180,
-90,
180,
90
]
]
},
"temporal": {
"interval": [
[
"1940-01-01T00:00:00Z",
null
]
]
}
},
"license": "other",
"keywords": [
"Atmospheric conditions",
"Atmosphere (surface)",
"Atmosphere (upper air)",
"Past",
"Global",
"Reanalysis",
"Copernicus C3S"
],
"summaries": {
"federation:backends": [
"cop_cds",
"wekeo_ecmwf"
]
},
"stac_version": "1.1.0",
"stac_extensions": [
"https://stac-extensions.github.io/scientific/v1.0.0/schema.json",
"https://stac-extensions.github.io/timestamps/v1.1.0/schema.json",
"https://stac-extensions.github.io/application/v0.1.0/schema.json"
],
"providers": [
{
"name": "ECMWF",
"roles": [
"licensor",
"producer",
"processor"
],
"url": "https://www.ecmwf.int/"
},
{
"name": "Copernicus Climate Change Service (C3S)",
"roles": [
"host"
],
"url": "https://climate.copernicus.eu/"
}
],
"created": "2023-08-28T15:11:18Z",
"updated": "2026-04-24T10:24:59Z",
"published": "2023-08-28T15:11:18Z",
"sci:doi": "10.24381/cds.adbb2d47",
"sci:citation": "Hersbach, H., Bell, B., Berrisford, P., Biavati, G., Hor\u00e1nyi, A., Mu\u00f1oz Sabater, J., Nicolas, J., Peubey, C., Radu, R., Rozum, I., Schepers, D., Simmons, A., Soci, C., Dee, D., Th\u00e9paut, J-N. (2023): ERA5 hourly data on single levels from 1940 to present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS), DOI: 10.24381/cds.adbb2d47 (Accessed on DD-MMM-YYYY)",
"sci:publications": [
{
"doi": "10.1002/qj.3803",
"citation": "Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Hor\u00e1nyi, A., Mu\u00f1oz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., De Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R. J., H\u00f3lm, E., Janiskov\u00e1, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., de Rosnay, P., Rozum, I., Vamborg, F., Villaume, S., and Th\u00e9paut, J.-N.: The ERA5 Global Reanalysis, Q. J. Roy. Meteorol. Soc., 146, 1999-2049, https://doi.org/10.1002/qj.3803, 2020."
},
{
"doi": "10.1002/qj.4174",
"citation": "Bell, B., Hersbach, H., Simmons, A., Berrisford, P., Dahlgren, P., Hor\u00e1nyi, A., Mu\u00f1oz-Sabater, J., Nicolas, J., Radu, R., Schepers, D., Soci, C., Villaume, Sebastien., Bidlot, J.-R., Haimberger, L., Woollen, J., Buontempo, C. and Th\u00e9paut, J.-N.: The ERA5 global reanalysis: Preliminary extension to 1950, Q. J. Roy. Meteorol. Soc., 147, 4186-4227, https://doi.org/10.1002/qj.4174, 2021."
}
],
"dedl:short_description": "This dataset contains hourly ERA5 reanalysed data from 1940-present on various spatial resolutions, combining model outputs with historical observations through data assimilation methods, providing multiple variables including uncertainties and monthly-means."
}
],
"links": [
{
"rel": "root",
"type": "application/json",
"href": "https://hda.data.destination-earth.eu/stac/v2/",
"title": "DEDL HDA STAC API"
},
{
"rel": "self",
"type": "application/json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections?query=%7B%22federation%3Abackends%22%3A+%7B%22eq%22%3A+%22cop_cds%22%7D%7D&q=ERA5+AND+%22hourly+data+on+single+levels%22",
"title": "Current Page"
}
]
}
From the discovery results displayed above, we can see that the collection “ERA5 hourly data on single levels from 1940 to present” has the ID EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS. We will use this identifier in all subsequent operations.
COLLECTION_ID = "EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS"Filtering¶
The “ERA5 hourly data on single levels from 1940 to present” dataset, as well as the others datasets provided by the CDS (Climate Data Store), can be subset requesting only the data of interest.
The set of parameters to subset the data are available through the HDA queryables endpoint. For this specific collection: Filter Options To understand how the HDA queryables API works you can also have a look at the queryables notebook
Using the information provided by the queryables endpoint, we can download the data we are interested in. In this example we will download the 2m temperature and sea surface temperature data for the hottest day in 2024, July 22nd.
Search into asynchronous datasets, as the CDS datasets are, always return a single item:
filters = {
key: {"eq": value}
for key, value in {
"ecmwf:data_format": "grib",
"ecmwf:variable": ["sea_surface_temperature","mean_sea_level_pressure"],
"ecmwf:time": ["12:00"],
"ecmwf:day": ["22"],
"ecmwf:month": ["07"],
"ecmwf:year": ["2024"],
"ecmwf:product_type": ["reanalysis"],
"ecmwf:download_format": "unarchived"
}.items()
}
response = requests.post(f"{HDA_STAC_ENDPOINT}/search", headers=auth_headers, json={
"collections": ["EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS"],
"query": filters
})
if(response.status_code!= 200):
(print(response.text))
response.raise_for_status()
product = response.json()["features"][0]
print(json.dumps(product,indent=4)){
"type": "Feature",
"assets": {},
"id": "ERA5_SL_ORDERABLE_1cdf4756534a928ec8848372d685d693ab9e7487",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
180.0,
-90.0
],
[
180.0,
90.0
],
[
-180.0,
90.0
],
[
-180.0,
-90.0
],
[
180.0,
-90.0
]
]
]
},
"bbox": [
-180.0,
-90.0,
180.0,
90.0
],
"collection": "EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS",
"stac_version": "1.1.0",
"properties": {
"datetime": "2024-07-22T12:00:00.000Z",
"end_datetime": "2024-07-22T12:00:00.000Z",
"start_datetime": "2024-07-22T12:00:00.000Z",
"title": "ERA5_SL_ORDERABLE_1cdf4756534a928ec8848372d685d693ab9e7487",
"ecmwf:data_format": "grib",
"ecmwf:dataset": "reanalysis-era5-single-levels",
"ecmwf:day": [
"22"
],
"ecmwf:download_format": "unarchived",
"ecmwf:month": [
"07"
],
"ecmwf:product_type": [
"reanalysis"
],
"ecmwf:time": [
"12:00"
],
"ecmwf:variable": [
"mean_sea_level_pressure",
"sea_surface_temperature"
],
"ecmwf:year": [
"2024"
],
"order:status": "orderable",
"federation:backends": [
"cop_cds"
],
"providers": [
{
"name": "cop_cds",
"roles": [
"host"
]
}
]
},
"stac_extensions": [
"https://stac-extensions.github.io/order/v1.1.0/schema.json"
],
"links": [
{
"rel": "retrieve",
"type": "application/geo+json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS/order",
"method": "POST",
"title": "Retrieve",
"body": {
"data_format": "grib",
"day": [
"22"
],
"download_format": "unarchived",
"month": [
"07"
],
"product_type": [
"reanalysis"
],
"time": [
"12:00"
],
"variable": [
"mean_sea_level_pressure",
"sea_surface_temperature"
],
"year": [
"2024"
]
}
},
{
"rel": "collection",
"type": "application/json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS",
"title": "ERA5 hourly data on single levels from 1940 to present"
},
{
"rel": "self",
"type": "application/geo+json",
"href": "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS/items/ERA5_SL_ORDERABLE_1cdf4756534a928ec8848372d685d693ab9e7487",
"title": "Original item link"
}
]
}
The single item returned (above) contains:
The product id: “ERA5_SL_ORDERABLE_...”, that is a placeholder, its name contains the term “ORDERABLE”.
The order:status that indicates that the product is “orderable”
Request params used for the order extracted from the search result:
ecmwf:variables: “2m_temperature”, “sea_surface_temperature”
ecmwf:day:“22”
ecmwf:dataset:“reanalysis-era5-single-levels”
ecmwf:month:“7”
ecmwf:data_format:“grib”
ecmwf:time:“12:00”
ecmwf:year:“2024”
ecmwf:product_type:“reanalysis”
link = next((l for l in product.get('links', []) if l.get("rel") == "retrieve"), None)
if link:
href = link.get("href")
body = link.get("body")
print("order endpoint:", href)
print("order body:")
print(json.dumps(body, indent=4))
else:
print(f"No link with rel='{target_rel}' found")
order endpoint: https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS/order
order body:
{
"data_format": "grib",
"day": [
"22"
],
"download_format": "unarchived",
"month": [
"07"
],
"product_type": [
"reanalysis"
],
"time": [
"12:00"
],
"variable": [
"mean_sea_level_pressure",
"sea_surface_temperature"
],
"year": [
"2024"
]
}
Order data¶
We have now all the information to order the data.
From the search results we know that the product is orderable and offline, we then need to order the product we searched for.
response = requests.post(href, json=body, headers=auth_headers)
if response.status_code != 200:
print(response.content)
response.raise_for_status()
ordered_item = response.json()
product_id = ordered_item["id"]
storage_tier = ordered_item["properties"].get("storage:tier", "online")
order_status = ordered_item["properties"].get("order:status", "unknown")
federation_backend = ordered_item["properties"].get("federation:backends", [None])[0]
print(f"Product ordered: {product_id}")
print(f"Provider: {federation_backend}")
print(f"Order status: {order_status}")Product ordered: 971adc47-6275-4a12-b851-641ee5985bb3
Provider: cop_cds
Order status: ordered
Poll the API until product is ready¶
We request the product itself to get an update of its status.
#timeout and step for polling (sec)
TIMEOUT = 300
STEP = 1
ORDER_STATUS = "succeeded"
self_url = f"{HDA_STAC_ENDPOINT}/collections/{COLLECTION_ID}/items/{product_id}"
item = {}
for i in range(0, TIMEOUT, STEP):
print(f"Polling {i + 1}/{TIMEOUT // STEP}")
response = requests.get(self_url, headers=auth_headers)
response.raise_for_status()
item = response.json()
print(item["properties"].get("order:status"))
status = item["properties"].get("order:status")
if status == ORDER_STATUS:
download_url = item["assets"]["downloadLink"]["href"]
print("Product is ready to be downloaded.")
print(f"Asset URL: {download_url}")
break
sleep(STEP)
else:
order_status = item["properties"].get("order:status", "unknown")
print(f"We could not download the product after {TIMEOUT // STEP} tries. Current order status is {order_status}")
Polling 1/300
ordered
Polling 2/300
ordered
Polling 3/300
ordered
Polling 4/300
ordered
Polling 5/300
ordered
Polling 6/300
ordered
Polling 7/300
ordered
Polling 8/300
ordered
Polling 9/300
ordered
Polling 10/300
ordered
Polling 11/300
ordered
Polling 12/300
succeeded
Product is ready to be downloaded.
Asset URL: https://hda-download.marenostrum.data.destination-earth.eu/data/cop_cds/EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS/971adc47-6275-4a12-b851-641ee5985bb3/downloadLink
Download¶
response = requests.get(download_url, stream=True, headers=auth_headers)
response.raise_for_status()
content_disposition = response.headers.get('Content-Disposition')
total_size = int(response.headers.get("content-length", 0))
if content_disposition:
filename = content_disposition.split('filename=')[1].strip('"')
filename = unquote(filename)
else:
filename = os.path.basename(url)
# Open a local file in binary write mode and write the content
print(f"downloading {filename}")
with tqdm(total=total_size, unit="B", unit_scale=True) as progress_bar:
with open(filename, 'wb') as f:
for data in response.iter_content(1024):
progress_bar.update(len(data))
f.write(data)downloading 780103d4a10110a65650e3cc99d7f783.grib
3.58MB [00:00, 5.62MB/s]
import earthkit.data
import earthkit.plots
import earthkit.regrid
data = earthkit.data.from_source("file", filename)
earthkit.plots.quickplot(data)<earthkit.plots.components.figures.Figure at 0x7f81914d8850>