Filtering datasets with the STAC API Filter extension (CQL2)

Contents¶

Objective: Learn to write CQL2 filters (STAC API Filter extension) to discover datasets in the HDA catalog, going step by step from an exact ID lookup to a combined, multi-condition search.
Data sources: The DEDL Harmonised Data Access (HDA) STAC API v2: https://hda.data.destination-earth.eu/stac/v2.
Methods: Each section introduces one CQL2 operator: =, LIKE, CASEI(), IN, AND/OR/NOT, fields beyond id/title, and the JSON encoding (cql2-json).
Prerequisites: Only the pystac-client package. Dataset discovery does not require authentication or a DESP account.
Expected output: Printed lists of collection IDs and titles matching each filter, ending with a single collection found by combining operators.

What is the Filter extension?¶

The STAC API Filter extension adds a filter parameter to STAC search requests. The idea is familiar from SQL: the filter is a condition, like a WHERE clause, that every returned record must satisfy.

The condition is written in CQL2 (Common Query Language), a small standard language with two equivalent encodings:

cql2-text : a readable string, e.g. id = 'EO.EUM.DAT.METOP.GLB-SST-NC'
cql2-json : the same condition as a JSON document, convenient when the filter is built by code

In HDA the filter parameter is used here for collection search: the catalog has 300+ collections and CQL2 narrows them down precisely. pystac-client handles the encoding: pass the filter as a string and it is sent as cql2-text automatically.

Note: the Filter extension also applies to item search (/search). In HDA that support is currently partial, so this notebook focuses on collection search. Item search is covered in HDA-PyStac-Client.ipynb.

The goal of discovery is always the same: find the collection ID. With CQL2 there are several ways to get it, depending on what you already know. Each cell below shows one strategy, from the most precise to the most open; the following sections explain each operator in detail.

Imports¶

pip install --user --quiet --upgrade destinelab

Note: you may need to restart the kernel to use updated packages.

import pystac_client

Connect to the HDA catalog¶

No token or credentials are needed for discovery.

HDA_STAC_URL = "https://hda.data.destination-earth.eu/stac/v2"

catalog = pystac_client.Client.open(HDA_STAC_URL)
print(f"Connected to: {catalog.title}")

Connected to: DEDL HDA STAC API

Every example ends the same way: run the search and list the matching IDs and titles. A small helper avoids repeating those lines in each cell. It prints at most limit collections.

def show(results, limit=10):
    print(f"{len(results)} collections found")
    for collection in results[:limit]:
        print(f"  {collection.id}  —  {collection.title}")

1. Searching the ID for part of the name using : `LIKE`¶

Most of the time you do not know the full ID or just remember part of it. In that case you can use the operator LIKE that compares a field against a pattern, where % matches any sequence of characters. Where you place the % matters:

'EO.EUM.DAT.MSG%' : starts with : everything from one satellite dataset and provider
'%Temperature%' : contains: the word appears anywhere
'AVHRR%' : starts with : titles beginning with an instrument name

HDA collection IDs follow the convention EO.<PROVIDER>.DAT.<DATASET>, so an anchored prefix is the natural way to select a provider or a satellite family. The first example selects the whole Meteosat Second Generation (MSG) family.

results = catalog.collection_search(filter="id LIKE 'EO.EUM.DAT.MSG%'").collection_list()
show(results, limit=10)

15 collections found
  EO.EUM.DAT.MSG.CLM  —  Cloud Mask - MSG - 0 degree
  EO.EUM.DAT.MSG.CLM-IODC  —  Cloud Mask - MSG - Indian Ocean
  EO.EUM.DAT.MSG.CTH  —  Cloud Top Height - MSG - 0 degree
  EO.EUM.DAT.MSG.CTH-IODC  —  Cloud Top Height - MSG - Indian Ocean
  EO.EUM.DAT.MSG.HRSEVIRI  —  High Rate SEVIRI Level 1.5 Image Data - MSG - 0 degree
  EO.EUM.DAT.MSG.HRSEVIRI-IODC  —  High Rate SEVIRI Level 1.5 Image Data - MSG - Indian Ocean
  EO.EUM.DAT.MSG.LSA-FRM  —  Fire Risk Map - Released Energy Based - MSG
  EO.EUM.DAT.MSG.LSA-LST-CDR  —  Land Surface Temperature Climate Data Record - MSG
  EO.EUM.DAT.MSG.LSA-LSTDE  —  Land Surface Temperature with Directional Effects - MSG
  EO.EUM.DAT.MSG.MSG15-RSS  —  Rapid Scan High Rate SEVIRI Level 1.5 Image Data - MSG

# Filtering for a key word in the title containing
results = catalog.collection_search(filter="title LIKE 'AVHRR%'").collection_list()
show(results)

5 collections found
  EO.EUM.DAT.METOP.AVHRDUAL0100  —  AVHRR Global (dual) Atmospheric Motion Vectors Climate Data Record Release 1 - Metop-A/B and -B/A
  EO.EUM.DAT.METOP.AVHRPWE0200  —  AVHRR Polar Atmospheric Motion Vectors Climate Data Record Release 2 - Metop-A and -B
  EO.EUM.DAT.METOP.AVHRRGACR02  —  AVHRR GAC Atmospheric Motion Vectors Climate Data Record Release 2 - Multimission - Polar
  EO.EUM.DAT.METOP.AVHRRL1  —  AVHRR Level 1B - Metop - Global
  EO.EUM.DAT.MULT.AVHGAC1C0100  —  AVHRR Fundamental Data Record - Release 1 - Multimission

# Filtering for a key word in the title that starst with
results = catalog.collection_search(filter="title LIKE '%sea surface temperature%'").collection_list()
show(results)

1 collections found
  EO.MO.DAT.SST_GLO_SST_L4_REP_OBSERVATIONS_010_024  —  ESA SST CCI and C3S reprocessed sea surface temperature analyses

# You remember a fragment of the ID: substring search
results = catalog.collection_search(filter="id LIKE '%SST%'").collection_list()
show(results, limit=5)

5 collections found
  EO.EUM.DAT.METOP.GLB-SST-NC  —  Global L3C AVHRR Sea Surface Temperature (GHRSST) - Metop
  EO.MO.DAT.SST_GLO_SST_L3S_NRT_OBSERVATIONS_010_010  —  ODYSSEA Global Ocean - Sea Surface Temperature Multi-sensor L3 Observations
  EO.MO.DAT.SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001  —  Global Ocean OSTIA Sea Surface Temperature and Sea Ice Analysis
  EO.MO.DAT.SST_GLO_SST_L4_REP_OBSERVATIONS_010_011  —  Global Ocean OSTIA Sea Surface Temperature and Sea Ice Reprocessed
  EO.MO.DAT.SST_GLO_SST_L4_REP_OBSERVATIONS_010_024  —  ESA SST CCI and C3S reprocessed sea surface temperature analyses

2. Using strings case-sensitive: `CASEI()`¶

CQL2 string comparisons are case-sensitive, and HDA enforces this strictly: searching for '%metop%' in lowercase finds nothing, because the titles say Metop. CASEI() (case-insensitive) solves this: it normalizes the casing of a value before comparing. Wrap both sides (the property and the pattern) and the exact capitalization no longer matters.

#Filtering by using case-sensitive CASEI()
results = catalog.collection_search(filter="CASEI(title) LIKE CASEI('%metop%')").collection_list()
show(results, limit=10)

34 collections found
  EO.EUM.CM.METOP.ASCSZFR02  —  ASCAT Level 1 SZF Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZOR02  —  ASCAT Level 1 SZO Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZRR02  —  ASCAT Level 1 SZR Climate Data Record Release 2 - Metop
  EO.EUM.DAT.METOP.AMSUL1  —  AMSU-A Level 1B - Metop - Global
  EO.EUM.DAT.METOP.ASCSZF1B  —  ASCAT Level 1 Sigma0 Full Resolution - Metop - Global
  EO.EUM.DAT.METOP.ASCSZO1B  —  ASCAT Level 1 Sigma0 resampled at 25 km Swath Grid - Metop - Global
  EO.EUM.DAT.METOP.ASCSZR1B  —  ASCAT Level 1 Sigma0 resampled at 12.5 km Swath Grid - Metop - Global
  EO.EUM.DAT.METOP.AVHRDUAL0100  —  AVHRR Global (dual) Atmospheric Motion Vectors Climate Data Record Release 1 - Metop-A/B and -B/A
  EO.EUM.DAT.METOP.AVHRPWE0200  —  AVHRR Polar Atmospheric Motion Vectors Climate Data Record Release 2 - Metop-A and -B
  EO.EUM.DAT.METOP.AVHRRL1  —  AVHRR Level 1B - Metop - Global

# You only know the topic: search the title, ignoring the casing
results = catalog.collection_search(
    filter="CASEI(title) LIKE CASEI('%sea surface temperature%')"
).collection_list()
show(results, limit=10)

7 collections found
  EO.EUM.DAT.METOP.GLB-SST-NC  —  Global L3C AVHRR Sea Surface Temperature (GHRSST) - Metop
  EO.EUM.DAT.SENTINEL-3.SL_2_WSTBC003  —  SLSTR Level 2 Sea Surface Temperature (SST) (version BC003) - Sentinel-3 - Reprocessed
  EO.EUM.DAT.SENTINEL-3.SL_2_WST___  —  SLSTR Level 2 Sea Surface Temperature (SST) - Sentinel-3
  EO.MO.DAT.SST_GLO_SST_L3S_NRT_OBSERVATIONS_010_010  —  ODYSSEA Global Ocean - Sea Surface Temperature Multi-sensor L3 Observations
  EO.MO.DAT.SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001  —  Global Ocean OSTIA Sea Surface Temperature and Sea Ice Analysis
  EO.MO.DAT.SST_GLO_SST_L4_REP_OBSERVATIONS_010_011  —  Global Ocean OSTIA Sea Surface Temperature and Sea Ice Reprocessed
  EO.MO.DAT.SST_GLO_SST_L4_REP_OBSERVATIONS_010_024  —  ESA SST CCI and C3S reprocessed sea surface temperature analyses

3. Using a short list of candidates: `IN`¶

When a few known IDs are of interest, IN retrieves them in a single request instead of one equality search per ID. Here: sea surface temperature and land surface temperature, both from Metop.

results = catalog.collection_search(
    filter="id IN ('EO.EUM.DAT.METOP.GLB-SST-NC', 'EO.EUM.DAT.METOP.LSA-002')"
).collection_list()
show(results)

2 collections found
  EO.EUM.DAT.METOP.GLB-SST-NC  —  Global L3C AVHRR Sea Surface Temperature (GHRSST) - Metop
  EO.EUM.DAT.METOP.LSA-002  —  Daily Land Surface Temperature - Metop

4. Combining conditions: `AND`, `OR`, `NOT`¶

Real searches usually mix criteria. The three logical operators behave as expected:

AND narrows: both conditions must hold
OR widens: either condition may hold
NOT excludes: the condition must not hold

⚠️ Known limitation: mixing AND and OR: the HDA server currently loses the grouping when a filter combines AND with a parenthesized OR group. The conditions are re-grouped following operator precedence (AND before OR), so part of the filter is silently not applied and unexpected collections appear in the results. This happens with both encodings, cql2-text and cql2-json.
Until this is fixed, keep each filter as a pure AND chain (with NOT and CASEI(), which work fine), or run one query per OR branch and merge the results.

Filtering EUMETSAT collections whose title mentions climate (AND):

# using AND
results = catalog.collection_search(
    filter="id LIKE '%EUM%' AND title LIKE '%Climate%'"
).collection_list()
show(results, limit=10)

27 collections found
  EO.EUM.CM.METOP.ASCSZFR02  —  ASCAT Level 1 SZF Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZOR02  —  ASCAT Level 1 SZO Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZRR02  —  ASCAT Level 1 SZR Climate Data Record Release 2 - Metop
  EO.EUM.DAT.AMVR02  —  Atmospheric Motion Vectors Climate Data Record Release 2 - MFG and MSG - 0 degree
  EO.EUM.DAT.GSAL2R02  —  GSA Level 2 Climate Data Record Release 2 - MFG and MSG - 0 degree
  EO.EUM.DAT.METOP.AVHRDUAL0100  —  AVHRR Global (dual) Atmospheric Motion Vectors Climate Data Record Release 1 - Metop-A/B and -B/A
  EO.EUM.DAT.METOP.AVHRPWE0200  —  AVHRR Polar Atmospheric Motion Vectors Climate Data Record Release 2 - Metop-A and -B
  EO.EUM.DAT.METOP.AVHRRGACR02  —  AVHRR GAC Atmospheric Motion Vectors Climate Data Record Release 2 - Multimission - Polar
  EO.EUM.DAT.METOP.GOMPMA020100  —  Polar Multi-Sensor Aerosol Optical Properties Climate Data Record Release 1 - Metop-A and -B
  EO.EUM.DAT.METOP.IASSO22X0100  —  IASI SO2 Climate Data Record Release 1 - Metop-A and -B

Meteosat data from either the first (MFG) or the second generation (MSG) (OR):

# using OR
results = catalog.collection_search(
    filter="id LIKE 'EO.EUM.DAT.MFG%' OR id LIKE 'EO.EUM.DAT.MSG%'"
).collection_list()
show(results, limit=10)

21 collections found
  EO.EUM.DAT.MFG.GSA-57  —  GSA Level 2 Climate Data Record Release 2 - MFG - 57 degree
  EO.EUM.DAT.MFG.GSA-63  —  GSA Level 2 Climate Data Record Release 2 - MFG - 63 degree
  EO.EUM.DAT.MFG.MFGAMV570IODC1  —  Atmospheric Motion Vectors Climate Data Record Release 1 - MFG - Indian Ocean 57 degrees E
  EO.EUM.DAT.MFG.MFGAMV630IODC1  —  Atmospheric Motion Vectors Climate Data Record Release 1 - MFG - Indian Ocean 63 degrees E
  EO.EUM.DAT.MFG.MTP15EASY0200_0DEG  —  MVIRI Level 1.5 Climate Data Record Release 2 - MFG - 0 degree
  EO.EUM.DAT.MFG.MTP15EASY0200_57DEG  —  MVIRI Level 1.5 Climate Data Record Release 2 - MFG - 57 degree
  EO.EUM.DAT.MSG.CLM  —  Cloud Mask - MSG - 0 degree
  EO.EUM.DAT.MSG.CLM-IODC  —  Cloud Mask - MSG - Indian Ocean
  EO.EUM.DAT.MSG.CTH  —  Cloud Top Height - MSG - 0 degree
  EO.EUM.DAT.MSG.CTH-IODC  —  Cloud Top Height - MSG - Indian Ocean

EUMETSAT collections that are NOT climate data records (NOT):

#Using AND, NOT
results = catalog.collection_search(
    filter="id LIKE 'EO.EUM.DAT.MSG%' AND NOT title LIKE '%Climate Data Record%'"
).collection_list()
show(results, limit=10)

11 collections found
  EO.EUM.DAT.MSG.CLM  —  Cloud Mask - MSG - 0 degree
  EO.EUM.DAT.MSG.CLM-IODC  —  Cloud Mask - MSG - Indian Ocean
  EO.EUM.DAT.MSG.CTH  —  Cloud Top Height - MSG - 0 degree
  EO.EUM.DAT.MSG.CTH-IODC  —  Cloud Top Height - MSG - Indian Ocean
  EO.EUM.DAT.MSG.HRSEVIRI  —  High Rate SEVIRI Level 1.5 Image Data - MSG - 0 degree
  EO.EUM.DAT.MSG.HRSEVIRI-IODC  —  High Rate SEVIRI Level 1.5 Image Data - MSG - Indian Ocean
  EO.EUM.DAT.MSG.LSA-FRM  —  Fire Risk Map - Released Energy Based - MSG
  EO.EUM.DAT.MSG.LSA-LSTDE  —  Land Surface Temperature with Directional Effects - MSG
  EO.EUM.DAT.MSG.MSG15-RSS  —  Rapid Scan High Rate SEVIRI Level 1.5 Image Data - MSG
  EO.EUM.DAT.MSG.RSS-CLM  —  Rapid Scan Cloud Mask - MSG

5. Beyond `id` and `title`¶

Filters are not limited to the ID and the title. Two more fields are useful for discovery:

description : the full descriptive text of the collection, good for technical terms that never appear in the title
keywords : a list of thematic tags; LIKE matches if any keyword contains the pattern

Note how the first example reuses the AND from section 3 to keep only EUMETSAT results. Other providers also mention scatterometers in their descriptions.

#Using a description over the id
results = catalog.collection_search(
    filter="id LIKE 'EO.EUM%' AND description LIKE '%scatterometer%'"
).collection_list()
show(results)

4 collections found
  EO.EUM.DAT.METOP.OSI-150-A  —  ASCAT L2 25 km Winds Data Record Release 1 - Metop
  EO.EUM.DAT.METOP.OSI-150-B  —  ASCAT L2 12.5 km Winds Data Record Release 1 - Metop
  EO.EUM.DAT.METOP.SOMO12  —  ASCAT Soil Moisture at 12.5 km Swath Grid in NRT - Metop
  EO.EUM.DAT.METOP.SOMO25  —  ASCAT Soil Moisture at 25 km Swath Grid in NRT - Metop

#Using a keyword over the topic
results = catalog.collection_search(
    filter="keywords LIKE '%Fundamental Climate Data Record%'"
).collection_list()
show(results)

6 collections found
  EO.EUM.CM.METOP.ASCSZFR02  —  ASCAT Level 1 SZF Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZOR02  —  ASCAT Level 1 SZO Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZRR02  —  ASCAT Level 1 SZR Climate Data Record Release 2 - Metop
  EO.EUM.DAT.METOP.MHSMWHS0200  —  MHS Microwave Humidity Sounder Climate Data Record Release 2 - Metop-A and -B
  EO.EUM.DAT.MSG.SEVIRI-RSS_HR_IMG-L1_5-V1  —  SEVIRI Rapid Scan High Rate Level 1.5 Image Data Climate Data Record Release 1 - MSG
  EO.EUM.DAT.MULT.MHSMWHS0100  —  MHS Microwave Humidity Sounder Climate Data Record Release 1 - Metop and NOAA

6. The same filter as JSON: `cql2-json`¶

Everything so far used the text encoding. The JSON encoding expresses the same conditions as a plain Python dict, convenient when the filter is built programmatically ( to avoid going through a list of conditions), because there is no string quoting to get wrong.

Pass the dict to filter and set filter_lang="cql2-json" explicitly (without it, pystac-client tries to convert the dict to text and asks for an extra package).

The filter below is identical to the AND example of section 6, so it returns the same collections:

# Using a dictionary
filter_json = {
    "op": "and",
    "args": [
        {"op": "like", "args": [{"property": "id"}, "EO.EUM%"]},
        {"op": "like", "args": [{"property": "title"}, "%Climate%"]},
    ],
}

results = catalog.collection_search(filter=filter_json, filter_lang="cql2-json").collection_list()
show(results, limit=5)

27 collections found
  EO.EUM.CM.METOP.ASCSZFR02  —  ASCAT Level 1 SZF Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZOR02  —  ASCAT Level 1 SZO Climate Data Record Release 2 - Metop
  EO.EUM.CM.METOP.ASCSZRR02  —  ASCAT Level 1 SZR Climate Data Record Release 2 - Metop
  EO.EUM.DAT.AMVR02  —  Atmospheric Motion Vectors Climate Data Record Release 2 - MFG and MSG - 0 degree
  EO.EUM.DAT.GSAL2R02  —  GSA Level 2 Climate Data Record Release 2 - MFG and MSG - 0 degree

7. Putting it all together¶

A realistic search combining what we learned. Say we want Atmospheric Motion Vectors derived from Meteosat Second Generation:

we know the satellite family, so we anchor the ID with LIKE 'EO.EUM.DAT.MSG%'
we are not sure how the product is capitalized in the title, so we use CASEI()
both must hold, so we join them with AND

The filter lands on exactly one collection, the ID we need for the next step, searching items.

#Combining multiple operators (Like, AND, CASEI())
results = catalog.collection_search(
    filter="id LIKE 'EO.EUM.DAT.MSG%' AND CASEI(title) LIKE CASEI('%atmospheric motion vector%')"
).collection_list()
show(results)

1 collections found
  EO.EUM.DAT.MSG.SEVIRI-RSS_AMV-CDR-V1  —  SEVIRI Rapid Scan Atmospheric Motion Vectors Climate Data Record Release 1 - MSG

#
f1 = (
    "id LIKE 'EO.EUM%' "                                # LIKE: EUM provider
    "AND title LIKE '%Level 2%' "                       # LIKE: Processing level
    "AND CASEI(title) LIKE CASEI('%global%') "          # CASEI: 'global'coverage
    "AND NOT title LIKE '%Climate Data Record%'"        # NOT: exclude climate data records
)
show(catalog.collection_search(filter=f1).collection_list(), limit=5)

2 collections found
  EO.EUM.DAT.SENTINEL-3.SR_2_WATBC005  —  SRAL Level 2 Altimetry Global (version BC005) - Sentinel-3 - Reprocessed
  EO.EUM.DAT.SENTINEL-3.SR_2_WAT___  —  SRAL Level 2 Altimetry Global - Sentinel-3

# Same filter in cql2-json: explicit grouping, no parentheses involved.
# Returns the same wrong results, so the bug is in the query evaluation,
# not in the cql2-text parser.
filter_json = {
    "op": "and",
    "args": [
        {"op": "like", "args": [{"property": "title"}, "%Global%"]},
        {"op": "or", "args": [
            {"op": "like", "args": [{"property": "title"}, "%Level 2%"]},
            {"op": "like", "args": [{"property": "title"}, "%L2%"]},
        ]},
    ],
}
results = catalog.collection_search(filter=filter_json, filter_lang="cql2-json").collection_list()
show(results, limit=30)

4 collections found
  EO.EUM.DAT.METOP.OSI-150-A  —  ASCAT L2 25 km Winds Data Record Release 1 - Metop
  EO.EUM.DAT.METOP.OSI-150-B  —  ASCAT L2 12.5 km Winds Data Record Release 1 - Metop
  EO.EUM.DAT.SENTINEL-3.SR_2_WATBC005  —  SRAL Level 2 Altimetry Global (version BC005) - Sentinel-3 - Reprocessed
  EO.EUM.DAT.SENTINEL-3.SR_2_WAT___  —  SRAL Level 2 Altimetry Global - Sentinel-3

Summary¶

The Filter extension turns collection search into a precise query: pick the operator based on what you already know.

You know...	Use
The exact collection ID	`id = '...'`
Part of the ID or title	`LIKE` with `%` wildcards
The words, but not the casing	`CASEI(field) LIKE CASEI('...')`
A short list of IDs	`id IN (...)`
Several criteria at once	`AND`, `OR`, `NOT`
A technical term or a thematic tag	`description LIKE` / `keywords LIKE`
The filter is built by code	a dict + `filter_lang="cql2-json"`

In every case the outcome is a collection ID the key for the next step, searching and downloading items inside the collection. The title is used for thematic search (descriptive keywords), and the id is used to filter by the catalog structure (provider, product family).

Resources and references¶

STAC API Filter extension
OGC Common Query Language (CQL2) standard
pystac-client documentation
DEDL documentation — Data Discovery
DestinE Data Portfolio (web UI)
HDA-Dataset-Discovery.ipynb for the full discovery workflow (free text, providers, space and time)
HDA-PyStac-Client.ipynb for item search and access