Create a Reference Model with virtualizarr

In this tutorial, we illustrate how to assemble a reference model from .nc files located in a remote S3 bucket. We will use the library virtualizarr, which allows us to conveniently concatenate and save the final dataset in the parquet format without loading in memory the data (with a function called openvirtual_dataset()).

Here a summary of the main steps for building a reference model:

1. Identify where the raw data of your reference is stored.

2. Copy the data to the (S3) bucket.

3. Unify the data to a single dataset and store it locally to a parq/ folder (i.e., parquet format).

4. Optionally, upload the dataset on the remote bucket to share the model.

We will illustrate this guide with an example using the Ocean Physics Analysis and Forecast model for the European North West Shelf.

We will assemble the following data from 2022 to 2023:

• Sea surface height above geoid SSH (variable zos).

• Sea Water Potential Temperature (variable thetao).

• Bathymetry (sea bottom depth, variable deptho).

• Sea-land mask (variable mask).

1.a. Find the raw data

First, we need to find where the raw data of the targeted reference model is stored.

Once you have identified the model you want to use, use this web page to find the url of the data. Enter/Search for the name of your model in the input ‘’Collections’’. Once you’ve submitted your search, different data will be listed, along with their respective time intervals. In our case, we look for the hourly mean fields (and the additional static data bathymetry and land-sea mask), which are located here:

• zos.

• thetao.

• deptho and mask.

We suggest using the native data, whose url can be copy-pasted under the panel ‘’Native dataset’’ in the ‘’Assets’’ section (see screenshot below).

Note: if you don’t work remotely, feel free to download the files locally! For our illustration though, we assume otherwise.

1.b. Fetch the data

We fetch the data with rclone. Ensure that you have the adequate credentials (check the file ~/.config/rclone/rclone.conf with cat)*.

*For GFTS users, it should be done automatically during your onboarding.

Adapt and execute the following command:

rclone copy cmarine:SRC_URL gfts:DEST_URL

Where SRC_URL is the url copy-pasted earlier and DEST_URL the path where you want to store the data on the GFTS bucket, for each url.

In our example, it corresponds to:

rclone copy cmarine:mdl-native-13/native/NWSHELF_ANALYSISFORECAST_PHY_004_013/cmems_mod_nws_phy_anfc_0.027deg-2D_PT1H-m_202411 gfts:gfts-reference-data/demo/NWSHELF_ANALYSISFORECAST_PHY_004_013/
rclone copy cmarine:mdl-native-13/native/NWSHELF_ANALYSISFORECAST_PHY_004_013/cmems_mod_nws_phy_anfc_0.027deg-3D_PT1H-m_202309 gfts:gfts-reference-data/demo/NWSHELF_ANALYSISFORECAST_PHY_004_013/
rclone copy cmarine:mdl-native-13/native/NWSHELF_ANALYSISFORECAST_PHY_004_013/cmems_mod_nws_phy_anfc_0.027deg-3D_static_202411/NWS-MFC_004_013_mask_bathy.nc gfts:gfts-reference-data/demo/NWSHELF_ANALYSISFORECAST_PHY_004_013/

2. Assembling The Model

Preliminaries

The idea is simple, and consists in those steps for each path/data source:

1. Open virtually (i.e., without loading data in memory) all the .nc files.

2. Depending on your application, post-process the datasets (e.g., dropping or renaming variables or coordinates).

3. Concatenate them along time.

Then, we merge all the concatenated data and store the resulting dataset with kerchunk in the parquet format (usually, under a *.parq/ folder).

Below, we first present how to setup the notebook (installing the package, defining different variables etc.). Next, we show how to generally perform the steps above. Finally, we implement them to our example.

Setup

As mentioned earlier, we rely on the virtualizarr library and its function open_virtual_dataset for opening the remote datasets.

First, install virtualizarr with pip and import the function open_virtual_dataset:

!pip install virtualizarr

Requirement already satisfied: virtualizarr in /srv/conda/envs/notebook/lib/python3.12/site-packages (1.2.0)
Requirement already satisfied: xarray>=2024.10.0 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from virtualizarr) (2024.11.0)
Requirement already satisfied: numpy>=2.0.0 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from virtualizarr) (2.1.3)
Requirement already satisfied: packaging in /srv/conda/envs/notebook/lib/python3.12/site-packages (from virtualizarr) (24.1)
Requirement already satisfied: universal-pathlib in /srv/conda/envs/notebook/lib/python3.12/site-packages (from virtualizarr) (0.2.5)
Requirement already satisfied: numcodecs in /srv/conda/envs/notebook/lib/python3.12/site-packages (from virtualizarr) (0.12.1)
Requirement already satisfied: ujson in /srv/conda/envs/notebook/lib/python3.12/site-packages (from virtualizarr) (5.10.0)
Requirement already satisfied: pandas>=2.1 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from xarray>=2024.10.0->virtualizarr) (2.2.2)
Requirement already satisfied: fsspec!=2024.3.1,>=2022.1.0 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from universal-pathlib->virtualizarr) (2024.6.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from pandas>=2.1->xarray>=2024.10.0->virtualizarr) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from pandas>=2.1->xarray>=2024.10.0->virtualizarr) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from pandas>=2.1->xarray>=2024.10.0->virtualizarr) (2024.1)
Requirement already satisfied: six>=1.5 in /srv/conda/envs/notebook/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas>=2.1->xarray>=2024.10.0->virtualizarr) (1.16.0)

Import the open_virtual_dataset function and configure the remote access to the S3 bucket.

from virtualizarr import open_virtual_dataset
import s3fs
import xarray as xr

s3 = s3fs.S3FileSystem(
    anon=False,
    profile="gfts",
    client_kwargs={
        "endpoint_url": "https://s3.gra.perf.cloud.ovh.net",
        "region_name": "gra",
    },
)
storage_options = {
    "anon": False,
    "client_kwargs": {
        "endpoint_url": "https://s3.gra.perf.cloud.ovh.net/",
        "region_name": "gra",
    },
}  # for opening remote virtual dataset with virtualizarr

How-to General Procedure

Open virtually the .nc files given a remote root folder

root = "path_of_your_data_on_s3/" # update w.r.t your use case
s3path = f"{root}*.nc"            # update w.r.t your use case

remote_files = sorted(s3.glob(s3path)) # remind to sort (time-wise) the files!

virtual_datasets = [
    open_virtual_dataset(
        "s3://" + filepath, 
        reader_options={"storage_options": storage_options},
        indexes={}, # we will concatenate the datasets along time later 
        decode_times=True,
        # indicate here the variables/coords you want to load in memory
        # by doing so they will be **saved** in the final `.parq/` folder
        loadable_variables=["depth", "longitude", "latitude"]
    )
    for filepath in remote_files
]

Concatenating the vds along time

vds = xr.concat(virtual_datasets, dim="time", compat="override", coords="minimal", data_vars="minimal", combine_attrs="drop_conflicts")

Repeat the procedure for each root path (or data source). Now, assuming you end up with, for instance, vds1 and vds2, here how to merge them before saving the result:

import xarray as xr
merged_vds = xr.merge([vds1, vds2])
merged_vds.virtualize.to_kerchunk("output.parq", format="parquet") # update w.r.t your application

NB: if you only have a source (i.e., a single vds), skip the mergging step and directly run vds.virtualize.to_kerchunk("output.parq", format="parquet").

Speeding up the computation with dask

The opening of the remote files can be speeded up with dask as follows:

Creates a local cluster

from distributed import LocalCluster
import dask
cluster = LocalCluster()
client = cluster.get_client()
client

Then, embed the calls to open_virtual_dataset with dask.delayed(...) and lunch the computation with dask.compute().

...
virtual_datasets = [
    dask.delayed(open_virtual_dataset)(
        "s3://" + filepath, 
        ...
    )
    for filepath in remote_files
]
vds = dask.compute(*virtual_datasets)
...

Not only it speeds up the computation, but it also lets you follow the progression (something very helpful if there are a lot of .nc files to read).

Application: Example

We finish this how-to guide by implementing the procedure above to our example.

Let’s first define variables to store the root paths of each data source:

# data sources
root = "gfts-reference-data/demo/NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013/"
static_root = root + "cmems_mod_nws_phy_anfc_0.027deg-3D_static_202411/"
ssh_root = root + "cmems_mod_nws_phy_anfc_0.027deg-2D_PT1H-m_202411/"
temp_root = root + "cmems_mod_nws_phy_anfc_0.027deg-3D_PT1H-m_202309/"

# creates a cluster to speed up the computation
from distributed import LocalCluster
import dask

cluster = LocalCluster()
client = cluster.get_client()
client

/srv/conda/envs/notebook/lib/python3.12/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 35019 instead
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.12/site-packages/distributed/client.py:1617: VersionMismatchWarning: Mismatched versions found

+---------+--------+-----------+---------+
| Package | Client | Scheduler | Workers |
+---------+--------+-----------+---------+
| numpy   | 2.1.3  | 2.1.3     | 2.0.2   |
+---------+--------+-----------+---------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))

Client

Client-279661ab-b3dd-11ef-8176-b2230698acfe

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:35019/status

Launch dashboard in JupyterLab

Cluster Info

LocalCluster

598810e8

Dashboard: http://127.0.0.1:35019/status	Workers: 3
Total threads: 6	Total memory: 24.00 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-6ffe4645-5234-4037-9c54-67cec1ff25a4

Comm: tcp://127.0.0.1:36717	Workers: 3
Dashboard: http://127.0.0.1:35019/status	Total threads: 6
Started: Just now	Total memory: 24.00 GiB

Workers

Worker: 0

Comm: tcp://127.0.0.1:44673	Total threads: 2
Dashboard: http://127.0.0.1:44217/status	Memory: 8.00 GiB
Nanny: tcp://127.0.0.1:37825
Local directory: /tmp/dask-scratch-space/worker-5tqqw777

Worker: 1

Comm: tcp://127.0.0.1:45655	Total threads: 2
Dashboard: http://127.0.0.1:45185/status	Memory: 8.00 GiB
Nanny: tcp://127.0.0.1:43869
Local directory: /tmp/dask-scratch-space/worker-fptob1pn

Worker: 2

Comm: tcp://127.0.0.1:37051	Total threads: 2
Dashboard: http://127.0.0.1:43085/status	Memory: 8.00 GiB
Nanny: tcp://127.0.0.1:35887
Local directory: /tmp/dask-scratch-space/worker-p4ogs8b5

Static data (mask and deptho)

# locates the `.nc` file
s3path = f"{static_root}*_mask_bathy.nc"
remote_files = s3.glob(s3path)
assert len(remote_files) == 1
print(f"found {len(remote_files)} .nc files.")
print(remote_files)

found 1 .nc files.
['gfts-reference-data/NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013/MetO-NWS-PHY-001-013-STATIC_202112/NWS-MFC_004_013_mask_bathy.nc']

# opens the virtual dataset while dropping unused variables
vds = (
    open_virtual_dataset(
        "s3://" + remote_files[0],
        reader_options={"storage_options": storage_options},
        indexes={},
        decode_times=True,
        loadable_variables=["depth", "longitude", "latitude"],
    )
    .drop_vars(["lat", "lon", "deptho_lev_interp"])
    .rename_dims({"latitude": "lat", "longitude": "lon"})
)
vds

<xarray.Dataset> Size: 162MB
Dimensions:    (lat: 1240, lon: 958, depth: 33)
Coordinates:
  * depth      (depth) float32 132B 0.0 3.0 5.0 10.0 ... 2e+03 3e+03 4e+03 5e+03
  * latitude   (lat) float32 5kB 46.0 46.01 46.03 46.04 ... 62.72 62.73 62.74
  * longitude  (lon) float32 4kB -16.0 -15.97 -15.94 -15.91 ... 12.94 12.97 13.0
Dimensions without coordinates: lat, lon
Data variables:
    deptho     (lat, lon) float32 5MB ManifestArray<shape=(1240, 958), dtype=...
    mask       (depth, lat, lon) float32 157MB ManifestArray<shape=(33, 1240,...
Attributes:
    Conventions:          CF-1.7
    contact:              servicedesk.cmems@mercator-ocean.eu
    creation_date:        2021-10-26T06:41:01Z
    credit:               E.U. Copernicus Marine Service Information (CMEMS)
    forcing_data_source:  ECMWF Global Atmospheric Model (HRES); UKMO NATL12;...
    history:              See source and creation_date attributes
    institution:          UK Met Office
    licence:              http://marine.copernicus.eu/services-portfolio/serv...
    netcdf-version-id:    netCDF-4
    product:              NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013
    references:           http://marine.copernicus.eu/
    source:               PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWAT...

xarray.Dataset

• Dimensions:
  • lat: 1240
  • lon: 958
  • depth: 33
• Coordinates: (3)
  • depth
    (depth)
    float32
    0.0 3.0 5.0 ... 3e+03 4e+03 5e+03

    units :

    m

    positive :

    down

    standard_name :

    depth

    long_name :

    depth

    axis :

    Z

    array([0.00e+00, 3.00e+00, 5.00e+00, 1.00e+01, 1.50e+01, 2.00e+01, 2.50e+01,
           3.00e+01, 4.00e+01, 5.00e+01, 6.00e+01, 7.50e+01, 1.00e+02, 1.25e+02,
           1.50e+02, 1.75e+02, 2.00e+02, 2.25e+02, 2.50e+02, 3.00e+02, 3.50e+02,
           4.00e+02, 4.50e+02, 5.00e+02, 5.50e+02, 6.00e+02, 7.50e+02, 1.00e+03,
           1.50e+03, 2.00e+03, 3.00e+03, 4.00e+03, 5.00e+03], dtype=float32)

  • latitude
    (lat)
    float32
    46.0 46.01 46.03 ... 62.73 62.74

    nav_model :

    grid_T

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    axis :

    Y

    array([46.     , 46.01351, 46.02703, ..., 62.71622, 62.72973, 62.74324],
          dtype=float32)

  • longitude
    (lon)
    float32
    -16.0 -15.97 -15.94 ... 12.97 13.0

    nav_model :

    grid_T

    long_name :

    longitude

    standard_name :

    longitude

    units :

    degrees_east

    axis :

    X

    array([-16.     , -15.9697 , -15.93939, ...,  12.93939,  12.9697 ,  13.     ],
          dtype=float32)

• Data variables: (2)
  • deptho
    (lat, lon)
    float32
    ManifestArray<shape=(1240, 958),...

    long_name :

    Bathymetry

    standard_name :

    sea_floor_depth_below_geoid

    units :

    m

    ManifestArray<shape=(1240, 958), dtype=float32, chunks=(620, 479)>

  • mask
    (depth, lat, lon)
    float32
    ManifestArray<shape=(33, 1240, 9...

    long_name :

    Land-sea mask: sea = 1 ; land = 0

    standard_name :

    sea_binary_mask

    units :

    1

    ManifestArray<shape=(33, 1240, 958), dtype=float32, chunks=(33, 1240, 958)>

• Indexes: (3)
  • depth
    PandasIndex

    PandasIndex(Index([   0.0,    3.0,    5.0,   10.0,   15.0,   20.0,   25.0,   30.0,   40.0,
             50.0,   60.0,   75.0,  100.0,  125.0,  150.0,  175.0,  200.0,  225.0,
            250.0,  300.0,  350.0,  400.0,  450.0,  500.0,  550.0,  600.0,  750.0,
           1000.0, 1500.0, 2000.0, 3000.0, 4000.0, 5000.0],
          dtype='float32', name='depth'))

  • latitude
    PandasIndex

    PandasIndex(Index([              46.0, 46.013511657714844,  46.02703094482422,
             46.0405387878418,  46.05405044555664, 46.067569732666016,
            46.08108139038086,  46.09458923339844,  46.10810852050781,
           46.121620178222656,
           ...
           62.621620178222656,  62.63513946533203, 62.648651123046875,
            62.66215896606445,   62.6756706237793,  62.68918991088867,
           62.702701568603516,  62.71622085571289,  62.72972869873047,
            62.74324035644531],
          dtype='float32', name='latitude', length=1240))

  • longitude
    PandasIndex

    PandasIndex(Index([              -16.0,  -15.96969985961914, -15.939390182495117,
           -15.909090042114258, -15.878789901733398, -15.848480224609375,
           -15.818180084228516, -15.787879943847656, -15.757579803466797,
           -15.727270126342773,
           ...
            12.727270126342773,  12.757579803466797,  12.787879943847656,
            12.818180084228516,  12.848489761352539,  12.878789901733398,
            12.909090042114258,  12.939390182495117,   12.96969985961914,
                          13.0],
          dtype='float32', name='longitude', length=958))

• Attributes: (12)

  Conventions :

  CF-1.7

  contact :

  servicedesk.cmems@mercator-ocean.eu

  creation_date :

  2021-10-26T06:41:01Z

  credit :

  E.U. Copernicus Marine Service Information (CMEMS)

  forcing_data_source :

  ECMWF Global Atmospheric Model (HRES); UKMO NATL12; UKMO GLOWave; CMEMS-BALTIC; climatological rivers

  history :

  See source and creation_date attributes

  institution :

  UK Met Office

  licence :

  http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/

  netcdf-version-id :

  netCDF-4

  product :

  NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013

  references :

  http://marine.copernicus.eu/

  source :

  PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWATCH-III v4.18_NEMOVAR v6

Explanations:

• We want to work with lat and lon coordinates. However, no all the datasets use the same names (longitude vs lon). So, along with deptho_lev_interp, we remove lat and lon (if any) and then rename latitude and longitude to lat and lon, respectively.

static_vds = vds.rename({"latitude": "lat", "longitude": "lon"})
static_vds

/tmp/ipykernel_374/3887474104.py:1: UserWarning: rename 'latitude' to 'lat' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
  static_vds = vds.rename({"latitude": "lat", "longitude": "lon"})
/tmp/ipykernel_374/3887474104.py:1: UserWarning: rename 'longitude' to 'lon' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
  static_vds = vds.rename({"latitude": "lat", "longitude": "lon"})

<xarray.Dataset> Size: 162MB
Dimensions:  (lat: 1240, lon: 958, depth: 33)
Coordinates:
  * depth    (depth) float32 132B 0.0 3.0 5.0 10.0 ... 2e+03 3e+03 4e+03 5e+03
  * lat      (lat) float32 5kB 46.0 46.01 46.03 46.04 ... 62.7 62.72 62.73 62.74
  * lon      (lon) float32 4kB -16.0 -15.97 -15.94 -15.91 ... 12.94 12.97 13.0
Data variables:
    deptho   (lat, lon) float32 5MB ManifestArray<shape=(1240, 958), dtype=fl...
    mask     (depth, lat, lon) float32 157MB ManifestArray<shape=(33, 1240, 9...
Attributes:
    Conventions:          CF-1.7
    contact:              servicedesk.cmems@mercator-ocean.eu
    creation_date:        2021-10-26T06:41:01Z
    credit:               E.U. Copernicus Marine Service Information (CMEMS)
    forcing_data_source:  ECMWF Global Atmospheric Model (HRES); UKMO NATL12;...
    history:              See source and creation_date attributes
    institution:          UK Met Office
    licence:              http://marine.copernicus.eu/services-portfolio/serv...
    netcdf-version-id:    netCDF-4
    product:              NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013
    references:           http://marine.copernicus.eu/
    source:               PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWAT...

xarray.Dataset

• Dimensions:
  • lat: 1240
  • lon: 958
  • depth: 33
• Coordinates: (3)
  • depth
    (depth)
    float32
    0.0 3.0 5.0 ... 3e+03 4e+03 5e+03

    units :

    m

    positive :

    down

    standard_name :

    depth

    long_name :

    depth

    axis :

    Z

    array([0.00e+00, 3.00e+00, 5.00e+00, 1.00e+01, 1.50e+01, 2.00e+01, 2.50e+01,
           3.00e+01, 4.00e+01, 5.00e+01, 6.00e+01, 7.50e+01, 1.00e+02, 1.25e+02,
           1.50e+02, 1.75e+02, 2.00e+02, 2.25e+02, 2.50e+02, 3.00e+02, 3.50e+02,
           4.00e+02, 4.50e+02, 5.00e+02, 5.50e+02, 6.00e+02, 7.50e+02, 1.00e+03,
           1.50e+03, 2.00e+03, 3.00e+03, 4.00e+03, 5.00e+03], dtype=float32)

  • lat
    (lat)
    float32
    46.0 46.01 46.03 ... 62.73 62.74

    nav_model :

    grid_T

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    axis :

    Y

    array([46.     , 46.01351, 46.02703, ..., 62.71622, 62.72973, 62.74324],
          dtype=float32)

  • lon
    (lon)
    float32
    -16.0 -15.97 -15.94 ... 12.97 13.0

    nav_model :

    grid_T

    long_name :

    longitude

    standard_name :

    longitude

    units :

    degrees_east

    axis :

    X

    array([-16.     , -15.9697 , -15.93939, ...,  12.93939,  12.9697 ,  13.     ],
          dtype=float32)

• Data variables: (2)
  • deptho
    (lat, lon)
    float32
    ManifestArray<shape=(1240, 958),...

    long_name :

    Bathymetry

    standard_name :

    sea_floor_depth_below_geoid

    units :

    m

    ManifestArray<shape=(1240, 958), dtype=float32, chunks=(620, 479)>

  • mask
    (depth, lat, lon)
    float32
    ManifestArray<shape=(33, 1240, 9...

    long_name :

    Land-sea mask: sea = 1 ; land = 0

    standard_name :

    sea_binary_mask

    units :

    1

    ManifestArray<shape=(33, 1240, 958), dtype=float32, chunks=(33, 1240, 958)>

• Indexes: (3)
  • depth
    PandasIndex

    PandasIndex(Index([   0.0,    3.0,    5.0,   10.0,   15.0,   20.0,   25.0,   30.0,   40.0,
             50.0,   60.0,   75.0,  100.0,  125.0,  150.0,  175.0,  200.0,  225.0,
            250.0,  300.0,  350.0,  400.0,  450.0,  500.0,  550.0,  600.0,  750.0,
           1000.0, 1500.0, 2000.0, 3000.0, 4000.0, 5000.0],
          dtype='float32', name='depth'))

  • lat
    PandasIndex

    PandasIndex(Index([              46.0, 46.013511657714844,  46.02703094482422,
             46.0405387878418,  46.05405044555664, 46.067569732666016,
            46.08108139038086,  46.09458923339844,  46.10810852050781,
           46.121620178222656,
           ...
           62.621620178222656,  62.63513946533203, 62.648651123046875,
            62.66215896606445,   62.6756706237793,  62.68918991088867,
           62.702701568603516,  62.71622085571289,  62.72972869873047,
            62.74324035644531],
          dtype='float32', name='lat', length=1240))

  • lon
    PandasIndex

    PandasIndex(Index([              -16.0,  -15.96969985961914, -15.939390182495117,
           -15.909090042114258, -15.878789901733398, -15.848480224609375,
           -15.818180084228516, -15.787879943847656, -15.757579803466797,
           -15.727270126342773,
           ...
            12.727270126342773,  12.757579803466797,  12.787879943847656,
            12.818180084228516,  12.848489761352539,  12.878789901733398,
            12.909090042114258,  12.939390182495117,   12.96969985961914,
                          13.0],
          dtype='float32', name='lon', length=958))

• Attributes: (12)

  Conventions :

  CF-1.7

  contact :

  servicedesk.cmems@mercator-ocean.eu

  creation_date :

  2021-10-26T06:41:01Z

  credit :

  E.U. Copernicus Marine Service Information (CMEMS)

  forcing_data_source :

  ECMWF Global Atmospheric Model (HRES); UKMO NATL12; UKMO GLOWave; CMEMS-BALTIC; climatological rivers

  history :

  See source and creation_date attributes

  institution :

  UK Met Office

  licence :

  http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/

  netcdf-version-id :

  netCDF-4

  product :

  NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013

  references :

  http://marine.copernicus.eu/

  source :

  PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWATCH-III v4.18_NEMOVAR v6

Sea Surface Height (zos)

remote_files = sorted(
    s3.glob(f"{ssh_root}2022/*/*.nc") + s3.glob(f"{ssh_root}2023/*/*.nc")
)
print(f"found {len(remote_files)} .nc files.")

found 743 .nc files.

zos_virtual_datasets = [
    dask.delayed(open_virtual_dataset)(
        "s3://" + filepath,
        reader_options={"storage_options": storage_options},
        indexes={},
        decode_times=True,
        loadable_variables=["time", "lat", "lon"],
    )
    for filepath in remote_files
]

zvds = dask.compute(*zos_virtual_datasets)

zos_vds = xr.concat(
    zvds,
    dim="time",
    compat="override",
    coords="minimal",
    data_vars="minimal",
    combine_attrs="drop_conflicts",
)

zos_vds

<xarray.Dataset> Size: 42GB
Dimensions:  (time: 17832, lat: 1240, lon: 958)
Coordinates:
  * time     (time) datetime64[ns] 143kB 2022-01-01T01:00:00 ... 2024-01-14
  * lat      (lat) float32 5kB 46.0 46.01 46.03 46.04 ... 62.7 62.72 62.73 62.74
  * lon      (lon) float32 4kB -16.0 -15.97 -15.94 -15.91 ... 12.94 12.97 13.0
Data variables:
    zos      (time, lat, lon) int16 42GB ManifestArray<shape=(17832, 1240, 95...
Attributes:
    Conventions:          CF-1.7
    contact:              servicedesk.cmems@mercator-ocean.eu
    credit:               E.U. Copernicus Marine Service Information (CMEMS)
    forcing_data_source:  ECMWF Global Atmospheric Model (HRES); CMEMS-MERCAT...
    history:              See source and creation_date attributes
    institution:          UK Met Office
    licence:              http://marine.copernicus.eu/services-portfolio/serv...
    netcdf-version-id:    netCDF-4
    product:              NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013
    references:           http://marine.copernicus.eu/
    source:               PS-OS 45, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWAT...
    title:                hourly-instantaneous SSH (2D)

xarray.Dataset

• Dimensions:
  • time: 17832
  • lat: 1240
  • lon: 958
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2022-01-01T01:00:00 ... 2024-01-14

    axis :

    T

    standard_name :

    time

    long_name :

    Validity time

    array(['2022-01-01T01:00:00.000000000', '2022-01-01T02:00:00.000000000',
           '2022-01-01T03:00:00.000000000', ..., '2024-01-13T22:00:00.000000000',
           '2024-01-13T23:00:00.000000000', '2024-01-14T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    46.0 46.01 46.03 ... 62.73 62.74

    axis :

    Y

    standard_name :

    latitude

    units :

    degrees_north

    nav_model :

    grid_T

    cell_methods :

    longitude: mean

    long_name :

    latitude

    array([46.     , 46.01351, 46.02703, ..., 62.71622, 62.72973, 62.74324],
          dtype=float32)

  • lon
    (lon)
    float32
    -16.0 -15.97 -15.94 ... 12.97 13.0

    axis :

    X

    standard_name :

    longitude

    units :

    degrees_east

    nav_model :

    grid_T

    cell_methods :

    latitude: mean

    long_name :

    longitude

    array([-16.     , -15.9697 , -15.93939, ...,  12.93939,  12.9697 ,  13.     ],
          dtype=float32)

• Data variables: (1)
  • zos
    (time, lat, lon)
    int16
    ManifestArray<shape=(17832, 1240...

    add_offset :

    0.0

    long_name :

    Sea surface height above geoid

    missing_value :

    -32768

    scale_factor :

    0.001

    standard_name :

    sea_surface_height_above_geoid

    units :

    m

    valid_max :

    10000

    valid_min :

    -10000

    ManifestArray<shape=(17832, 1240, 958), dtype=int16, chunks=(1, 1240, 958)>

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-01-01 01:00:00', '2022-01-01 02:00:00',
                   '2022-01-01 03:00:00', '2022-01-01 04:00:00',
                   '2022-01-01 05:00:00', '2022-01-01 06:00:00',
                   '2022-01-01 07:00:00', '2022-01-01 08:00:00',
                   '2022-01-01 09:00:00', '2022-01-01 10:00:00',
                   ...
                   '2024-01-13 15:00:00', '2024-01-13 16:00:00',
                   '2024-01-13 17:00:00', '2024-01-13 18:00:00',
                   '2024-01-13 19:00:00', '2024-01-13 20:00:00',
                   '2024-01-13 21:00:00', '2024-01-13 22:00:00',
                   '2024-01-13 23:00:00', '2024-01-14 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=17832, freq=None))

  • lat
    PandasIndex

    PandasIndex(Index([              46.0, 46.013511657714844,  46.02703094482422,
             46.0405387878418,  46.05405044555664, 46.067569732666016,
            46.08108139038086,  46.09458923339844,  46.10810852050781,
           46.121620178222656,
           ...
           62.621620178222656,  62.63513946533203, 62.648651123046875,
            62.66215896606445,   62.6756706237793,  62.68918991088867,
           62.702701568603516,  62.71622085571289,  62.72972869873047,
            62.74324035644531],
          dtype='float32', name='lat', length=1240))

  • lon
    PandasIndex

    PandasIndex(Index([              -16.0,  -15.96969985961914, -15.939390182495117,
           -15.909090042114258, -15.878789901733398, -15.848480224609375,
           -15.818180084228516, -15.787879943847656, -15.757579803466797,
           -15.727270126342773,
           ...
            12.727270126342773,  12.757579803466797,  12.787879943847656,
            12.818180084228516,  12.848489761352539,  12.878789901733398,
            12.909090042114258,  12.939390182495117,   12.96969985961914,
                          13.0],
          dtype='float32', name='lon', length=958))

• Attributes: (12)

  Conventions :

  CF-1.7

  contact :

  servicedesk.cmems@mercator-ocean.eu

  credit :

  E.U. Copernicus Marine Service Information (CMEMS)

  forcing_data_source :

  ECMWF Global Atmospheric Model (HRES); CMEMS-MERCATOR12; UKMO GLOWave; CMEMS-BALTIC; climatological rivers

  history :

  See source and creation_date attributes

  institution :

  UK Met Office

  licence :

  http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/

  netcdf-version-id :

  netCDF-4

  product :

  NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013

  references :

  http://marine.copernicus.eu/

  source :

  PS-OS 45, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWATCH-III v4.18_NEMOVAR v6

  title :

  hourly-instantaneous SSH (2D)

Sea Water Potential Temperature (thetao)

thetao_virtual_datasets = [
    dask.delayed(open_virtual_dataset)(
        "s3://" + filepath,
        reader_options={"storage_options": storage_options},
        indexes={},
        decode_times=True,
        loadable_variables=[
            "time",
            "lat",
            "lon",
            "depth",
        ],  # note that we also load (and thus save) the depths
    )
    for filepath in sorted(
        s3.glob(f"{temp_root}2022/*/*.nc") + s3.glob(f"{temp_root}2023/*/*.nc")
    )
]

tvds = dask.compute(*thetao_virtual_datasets)

thetao_vds = xr.concat(
    tvds,
    dim="time",
    compat="override",
    coords="minimal",
    data_vars="minimal",
    combine_attrs="drop_conflicts",
)

thetao_vds

<xarray.Dataset> Size: 1TB
Dimensions:  (time: 17832, depth: 33, lat: 1240, lon: 958)
Coordinates:
  * time     (time) datetime64[ns] 143kB 2022-01-01T01:00:00 ... 2024-01-14
  * lat      (lat) float32 5kB 46.0 46.01 46.03 46.04 ... 62.7 62.72 62.73 62.74
  * lon      (lon) float32 4kB -16.0 -15.97 -15.94 -15.91 ... 12.94 12.97 13.0
  * depth    (depth) float32 132B 0.0 3.0 5.0 10.0 ... 2e+03 3e+03 4e+03 5e+03
Data variables:
    thetao   (time, depth, lat, lon) int16 1TB ManifestArray<shape=(17832, 33...
Attributes:
    Conventions:          CF-1.7
    contact:              servicedesk.cmems@mercator-ocean.eu
    credit:               E.U. Copernicus Marine Service Information (CMEMS)
    forcing_data_source:  ECMWF Global Atmospheric Model (HRES); CMEMS-MERCAT...
    history:              See source and creation_date attributes
    institution:          UK Met Office
    licence:              http://marine.copernicus.eu/services-portfolio/serv...
    netcdf-version-id:    netCDF-4
    product:              NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013
    references:           http://marine.copernicus.eu/
    source:               PS-OS 45, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWAT...
    title:                hourly-instantaneous potential temperature (3D)

xarray.Dataset

• Dimensions:
  • time: 17832
  • depth: 33
  • lat: 1240
  • lon: 958
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2022-01-01T01:00:00 ... 2024-01-14

    axis :

    T

    standard_name :

    time

    long_name :

    Validity time

    array(['2022-01-01T01:00:00.000000000', '2022-01-01T02:00:00.000000000',
           '2022-01-01T03:00:00.000000000', ..., '2024-01-13T22:00:00.000000000',
           '2024-01-13T23:00:00.000000000', '2024-01-14T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    46.0 46.01 46.03 ... 62.73 62.74

    axis :

    Y

    standard_name :

    latitude

    units :

    degrees_north

    nav_model :

    grid_T

    cell_methods :

    longitude: mean

    long_name :

    latitude

    array([46.     , 46.01351, 46.02703, ..., 62.71622, 62.72973, 62.74324],
          dtype=float32)

  • lon
    (lon)
    float32
    -16.0 -15.97 -15.94 ... 12.97 13.0

    axis :

    X

    standard_name :

    longitude

    units :

    degrees_east

    nav_model :

    grid_T

    cell_methods :

    latitude: mean

    long_name :

    longitude

    array([-16.     , -15.9697 , -15.93939, ...,  12.93939,  12.9697 ,  13.     ],
          dtype=float32)

  • depth
    (depth)
    float32
    0.0 3.0 5.0 ... 3e+03 4e+03 5e+03

    axis :

    Z

    standard_name :

    depth

    units :

    m

    positive :

    down

    long_name :

    depth

    array([0.00e+00, 3.00e+00, 5.00e+00, 1.00e+01, 1.50e+01, 2.00e+01, 2.50e+01,
           3.00e+01, 4.00e+01, 5.00e+01, 6.00e+01, 7.50e+01, 1.00e+02, 1.25e+02,
           1.50e+02, 1.75e+02, 2.00e+02, 2.25e+02, 2.50e+02, 3.00e+02, 3.50e+02,
           4.00e+02, 4.50e+02, 5.00e+02, 5.50e+02, 6.00e+02, 7.50e+02, 1.00e+03,
           1.50e+03, 2.00e+03, 3.00e+03, 4.00e+03, 5.00e+03], dtype=float32)

• Data variables: (1)
  • thetao
    (time, depth, lat, lon)
    int16
    ManifestArray<shape=(17832, 33, ...

    add_offset :

    25.0

    long_name :

    Sea Water Potential Temperature

    missing_value :

    -32768

    scale_factor :

    0.001

    standard_name :

    sea_water_potential_temperature

    units :

    degrees_C

    valid_max :

    30000

    valid_min :

    -30000

    ManifestArray<shape=(17832, 33, 1240, 958), dtype=int16, chunks=(1, 11, 414, 320)>

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-01-01 01:00:00', '2022-01-01 02:00:00',
                   '2022-01-01 03:00:00', '2022-01-01 04:00:00',
                   '2022-01-01 05:00:00', '2022-01-01 06:00:00',
                   '2022-01-01 07:00:00', '2022-01-01 08:00:00',
                   '2022-01-01 09:00:00', '2022-01-01 10:00:00',
                   ...
                   '2024-01-13 15:00:00', '2024-01-13 16:00:00',
                   '2024-01-13 17:00:00', '2024-01-13 18:00:00',
                   '2024-01-13 19:00:00', '2024-01-13 20:00:00',
                   '2024-01-13 21:00:00', '2024-01-13 22:00:00',
                   '2024-01-13 23:00:00', '2024-01-14 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=17832, freq=None))

  • lat
    PandasIndex

    PandasIndex(Index([              46.0, 46.013511657714844,  46.02703094482422,
             46.0405387878418,  46.05405044555664, 46.067569732666016,
            46.08108139038086,  46.09458923339844,  46.10810852050781,
           46.121620178222656,
           ...
           62.621620178222656,  62.63513946533203, 62.648651123046875,
            62.66215896606445,   62.6756706237793,  62.68918991088867,
           62.702701568603516,  62.71622085571289,  62.72972869873047,
            62.74324035644531],
          dtype='float32', name='lat', length=1240))

  • lon
    PandasIndex

    PandasIndex(Index([              -16.0,  -15.96969985961914, -15.939390182495117,
           -15.909090042114258, -15.878789901733398, -15.848480224609375,
           -15.818180084228516, -15.787879943847656, -15.757579803466797,
           -15.727270126342773,
           ...
            12.727270126342773,  12.757579803466797,  12.787879943847656,
            12.818180084228516,  12.848489761352539,  12.878789901733398,
            12.909090042114258,  12.939390182495117,   12.96969985961914,
                          13.0],
          dtype='float32', name='lon', length=958))

  • depth
    PandasIndex

    PandasIndex(Index([   0.0,    3.0,    5.0,   10.0,   15.0,   20.0,   25.0,   30.0,   40.0,
             50.0,   60.0,   75.0,  100.0,  125.0,  150.0,  175.0,  200.0,  225.0,
            250.0,  300.0,  350.0,  400.0,  450.0,  500.0,  550.0,  600.0,  750.0,
           1000.0, 1500.0, 2000.0, 3000.0, 4000.0, 5000.0],
          dtype='float32', name='depth'))

• Attributes: (12)

  Conventions :

  CF-1.7

  contact :

  servicedesk.cmems@mercator-ocean.eu

  credit :

  E.U. Copernicus Marine Service Information (CMEMS)

  forcing_data_source :

  ECMWF Global Atmospheric Model (HRES); CMEMS-MERCATOR12; UKMO GLOWave; CMEMS-BALTIC; climatological rivers

  history :

  See source and creation_date attributes

  institution :

  UK Met Office

  licence :

  http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/

  netcdf-version-id :

  netCDF-4

  product :

  NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013

  references :

  http://marine.copernicus.eu/

  source :

  PS-OS 45, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWATCH-III v4.18_NEMOVAR v6

  title :

  hourly-instantaneous potential temperature (3D)

Merging the three data sources

merged_vds = xr.merge([static_vds, zos_vds, thetao_vds])

merged_vds.virtualize.to_kerchunk("combined_final.parq", format="parquet")

merged_vds

<xarray.Dataset> Size: 1TB
Dimensions:  (lat: 1240, lon: 958, depth: 33, time: 17832)
Coordinates:
  * depth    (depth) float32 132B 0.0 3.0 5.0 10.0 ... 2e+03 3e+03 4e+03 5e+03
  * lat      (lat) float32 5kB 46.0 46.01 46.03 46.04 ... 62.7 62.72 62.73 62.74
  * lon      (lon) float32 4kB -16.0 -15.97 -15.94 -15.91 ... 12.94 12.97 13.0
  * time     (time) datetime64[ns] 143kB 2022-01-01T01:00:00 ... 2024-01-14
Data variables:
    deptho   (lat, lon) float32 5MB ManifestArray<shape=(1240, 958), dtype=fl...
    mask     (depth, lat, lon) float32 157MB ManifestArray<shape=(33, 1240, 9...
    zos      (time, lat, lon) int16 42GB ManifestArray<shape=(17832, 1240, 95...
    thetao   (time, depth, lat, lon) int16 1TB ManifestArray<shape=(17832, 33...
Attributes: (12/13)
    Conventions:          CF-1.7
    contact:              servicedesk.cmems@mercator-ocean.eu
    creation_date:        2021-10-26T06:41:01Z
    credit:               E.U. Copernicus Marine Service Information (CMEMS)
    forcing_data_source:  ECMWF Global Atmospheric Model (HRES); UKMO NATL12;...
    history:              See source and creation_date attributes
    ...                   ...
    licence:              http://marine.copernicus.eu/services-portfolio/serv...
    netcdf-version-id:    netCDF-4
    product:              NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013
    references:           http://marine.copernicus.eu/
    source:               PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWAT...
    coordinates:          depth lat lon time

xarray.Dataset

• Dimensions:
  • lat: 1240
  • lon: 958
  • depth: 33
  • time: 17832
• Coordinates: (4)
  • depth
    (depth)
    float32
    0.0 3.0 5.0 ... 3e+03 4e+03 5e+03

    units :

    m

    positive :

    down

    standard_name :

    depth

    long_name :

    depth

    axis :

    Z

    array([0.00e+00, 3.00e+00, 5.00e+00, 1.00e+01, 1.50e+01, 2.00e+01, 2.50e+01,
           3.00e+01, 4.00e+01, 5.00e+01, 6.00e+01, 7.50e+01, 1.00e+02, 1.25e+02,
           1.50e+02, 1.75e+02, 2.00e+02, 2.25e+02, 2.50e+02, 3.00e+02, 3.50e+02,
           4.00e+02, 4.50e+02, 5.00e+02, 5.50e+02, 6.00e+02, 7.50e+02, 1.00e+03,
           1.50e+03, 2.00e+03, 3.00e+03, 4.00e+03, 5.00e+03], dtype=float32)

  • lat
    (lat)
    float32
    46.0 46.01 46.03 ... 62.73 62.74

    nav_model :

    grid_T

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    axis :

    Y

    array([46.     , 46.01351, 46.02703, ..., 62.71622, 62.72973, 62.74324],
          dtype=float32)

  • lon
    (lon)
    float32
    -16.0 -15.97 -15.94 ... 12.97 13.0

    nav_model :

    grid_T

    long_name :

    longitude

    standard_name :

    longitude

    units :

    degrees_east

    axis :

    X

    array([-16.     , -15.9697 , -15.93939, ...,  12.93939,  12.9697 ,  13.     ],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    2022-01-01T01:00:00 ... 2024-01-14

    axis :

    T

    standard_name :

    time

    long_name :

    Validity time

    array(['2022-01-01T01:00:00.000000000', '2022-01-01T02:00:00.000000000',
           '2022-01-01T03:00:00.000000000', ..., '2024-01-13T22:00:00.000000000',
           '2024-01-13T23:00:00.000000000', '2024-01-14T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (4)
  • deptho
    (lat, lon)
    float32
    ManifestArray<shape=(1240, 958),...

    long_name :

    Bathymetry

    standard_name :

    sea_floor_depth_below_geoid

    units :

    m

    ManifestArray<shape=(1240, 958), dtype=float32, chunks=(620, 479)>

  • mask
    (depth, lat, lon)
    float32
    ManifestArray<shape=(33, 1240, 9...

    long_name :

    Land-sea mask: sea = 1 ; land = 0

    standard_name :

    sea_binary_mask

    units :

    1

    ManifestArray<shape=(33, 1240, 958), dtype=float32, chunks=(33, 1240, 958)>

  • zos
    (time, lat, lon)
    int16
    ManifestArray<shape=(17832, 1240...

    add_offset :

    0.0

    long_name :

    Sea surface height above geoid

    missing_value :

    -32768

    scale_factor :

    0.001

    standard_name :

    sea_surface_height_above_geoid

    units :

    m

    valid_max :

    10000

    valid_min :

    -10000

    ManifestArray<shape=(17832, 1240, 958), dtype=int16, chunks=(1, 1240, 958)>

  • thetao
    (time, depth, lat, lon)
    int16
    ManifestArray<shape=(17832, 33, ...

    add_offset :

    25.0

    long_name :

    Sea Water Potential Temperature

    missing_value :

    -32768

    scale_factor :

    0.001

    standard_name :

    sea_water_potential_temperature

    units :

    degrees_C

    valid_max :

    30000

    valid_min :

    -30000

    ManifestArray<shape=(17832, 33, 1240, 958), dtype=int16, chunks=(1, 11, 414, 320)>

• Indexes: (4)
  • depth
    PandasIndex

    PandasIndex(Index([   0.0,    3.0,    5.0,   10.0,   15.0,   20.0,   25.0,   30.0,   40.0,
             50.0,   60.0,   75.0,  100.0,  125.0,  150.0,  175.0,  200.0,  225.0,
            250.0,  300.0,  350.0,  400.0,  450.0,  500.0,  550.0,  600.0,  750.0,
           1000.0, 1500.0, 2000.0, 3000.0, 4000.0, 5000.0],
          dtype='float32', name='depth'))

  • lat
    PandasIndex

    PandasIndex(Index([              46.0, 46.013511657714844,  46.02703094482422,
             46.0405387878418,  46.05405044555664, 46.067569732666016,
            46.08108139038086,  46.09458923339844,  46.10810852050781,
           46.121620178222656,
           ...
           62.621620178222656,  62.63513946533203, 62.648651123046875,
            62.66215896606445,   62.6756706237793,  62.68918991088867,
           62.702701568603516,  62.71622085571289,  62.72972869873047,
            62.74324035644531],
          dtype='float32', name='lat', length=1240))

  • lon
    PandasIndex

    PandasIndex(Index([              -16.0,  -15.96969985961914, -15.939390182495117,
           -15.909090042114258, -15.878789901733398, -15.848480224609375,
           -15.818180084228516, -15.787879943847656, -15.757579803466797,
           -15.727270126342773,
           ...
            12.727270126342773,  12.757579803466797,  12.787879943847656,
            12.818180084228516,  12.848489761352539,  12.878789901733398,
            12.909090042114258,  12.939390182495117,   12.96969985961914,
                          13.0],
          dtype='float32', name='lon', length=958))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-01-01 01:00:00', '2022-01-01 02:00:00',
                   '2022-01-01 03:00:00', '2022-01-01 04:00:00',
                   '2022-01-01 05:00:00', '2022-01-01 06:00:00',
                   '2022-01-01 07:00:00', '2022-01-01 08:00:00',
                   '2022-01-01 09:00:00', '2022-01-01 10:00:00',
                   ...
                   '2024-01-13 15:00:00', '2024-01-13 16:00:00',
                   '2024-01-13 17:00:00', '2024-01-13 18:00:00',
                   '2024-01-13 19:00:00', '2024-01-13 20:00:00',
                   '2024-01-13 21:00:00', '2024-01-13 22:00:00',
                   '2024-01-13 23:00:00', '2024-01-14 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=17832, freq=None))

• Attributes: (13)

  Conventions :

  CF-1.7

  contact :

  servicedesk.cmems@mercator-ocean.eu

  creation_date :

  2021-10-26T06:41:01Z

  credit :

  E.U. Copernicus Marine Service Information (CMEMS)

  forcing_data_source :

  ECMWF Global Atmospheric Model (HRES); UKMO NATL12; UKMO GLOWave; CMEMS-BALTIC; climatological rivers

  history :

  See source and creation_date attributes

  institution :

  UK Met Office

  licence :

  http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/

  netcdf-version-id :

  netCDF-4

  product :

  NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013

  references :

  http://marine.copernicus.eu/

  source :

  PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWATCH-III v4.18_NEMOVAR v6

  coordinates :

  depth lat lon time

Option: copy the model to S3

As before, we can use rclone to send the model to the bucket:

rclone copy combined_all.parq gfts:gfts-reference-data/demo/NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013/

Fixing virtualizarr bug and quick data checking

In the following, we show how to fix a shortcoming of virtualizarr which causes the first depth value to equal nan upon opening the dataset.

We conclude with some ways to quickly check that the your model is complete.

import xarray as xr

# not needed if already done
from distributed import LocalCluster

cluster = LocalCluster()
client = cluster.get_client()
client

/srv/conda/envs/notebook/lib/python3.12/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 39135 instead
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.12/site-packages/distributed/client.py:1617: VersionMismatchWarning: Mismatched versions found

+---------+--------+-----------+---------+
| Package | Client | Scheduler | Workers |
+---------+--------+-----------+---------+
| numpy   | 2.1.3  | 2.1.3     | 2.0.2   |
+---------+--------+-----------+---------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))

Client

Client-4b1e1554-b150-11ef-895a-e6c722a023c1

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:39135/status

Launch dashboard in JupyterLab

Cluster Info

LocalCluster

bcde258e

Dashboard: http://127.0.0.1:39135/status	Workers: 3
Total threads: 6	Total memory: 24.00 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-7568d076-6902-4711-b96b-f3f775f29bc4

Comm: tcp://127.0.0.1:37137	Workers: 3
Dashboard: http://127.0.0.1:39135/status	Total threads: 6
Started: Just now	Total memory: 24.00 GiB

Workers

Worker: 0

Comm: tcp://127.0.0.1:36779	Total threads: 2
Dashboard: http://127.0.0.1:37619/status	Memory: 8.00 GiB
Nanny: tcp://127.0.0.1:35173
Local directory: /tmp/dask-scratch-space/worker-8y7fz8oe

Worker: 1

Comm: tcp://127.0.0.1:35777	Total threads: 2
Dashboard: http://127.0.0.1:44747/status	Memory: 8.00 GiB
Nanny: tcp://127.0.0.1:43831
Local directory: /tmp/dask-scratch-space/worker-c2rn_1be

Worker: 2

Comm: tcp://127.0.0.1:33643	Total threads: 2
Dashboard: http://127.0.0.1:43425/status	Memory: 8.00 GiB
Nanny: tcp://127.0.0.1:38881
Local directory: /tmp/dask-scratch-space/worker-q8p14e8t

combined_ds = xr.open_dataset(
    "combined_final.parq", engine="kerchunk", chunks={}
)  # chunks implicitly calls `dask`

combined_ds.coords["depth"].values[0] = 0.0  # fixing 1st depth is nan

combined_ds

<xarray.Dataset> Size: 3TB
Dimensions:  (depth: 33, lat: 1240, lon: 958, time: 8760)
Coordinates:
  * depth    (depth) float32 132B 0.0 3.0 5.0 10.0 ... 2e+03 3e+03 4e+03 5e+03
  * lat      (lat) float32 5kB 46.0 46.01 46.03 46.04 ... 62.7 62.72 62.73 62.74
  * lon      (lon) float32 4kB -16.0 -15.97 -15.94 -15.91 ... 12.94 12.97 13.0
  * time     (time) datetime64[ns] 70kB 2022-01-01T01:00:00 ... 2023-01-01
Data variables:
    deptho   (lat, lon) float32 5MB dask.array<chunksize=(620, 479), meta=np.ndarray>
    mask     (depth, lat, lon) float32 157MB dask.array<chunksize=(33, 1240, 958), meta=np.ndarray>
    mdt      (lat, lon) float32 5MB dask.array<chunksize=(1240, 958), meta=np.ndarray>
    thetao   (time, depth, lat, lon) float64 3TB dask.array<chunksize=(1, 11, 414, 320), meta=np.ndarray>
    zos      (time, lat, lon) float64 83GB dask.array<chunksize=(1, 1240, 958), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.7
    contact:              servicedesk.cmems@mercator-ocean.eu
    creation_date:        2021-10-26T06:41:01Z
    credit:               E.U. Copernicus Marine Service Information (CMEMS)
    forcing_data_source:  ECMWF Global Atmospheric Model (HRES); UKMO NATL12;...
    history:              See source and creation_date attributes
    institution:          UK Met Office
    licence:              http://marine.copernicus.eu/services-portfolio/serv...
    netcdf-version-id:    netCDF-4
    product:              NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013
    references:           http://marine.copernicus.eu/
    source:               PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWAT...

xarray.Dataset

• Dimensions:
  • depth: 33
  • lat: 1240
  • lon: 958
  • time: 8760
• Coordinates: (4)
  • depth
    (depth)
    float32
    0.0 3.0 5.0 ... 3e+03 4e+03 5e+03

    axis :

    Z

    long_name :

    depth

    positive :

    down

    standard_name :

    depth

    units :

    m

    array([0.00e+00, 3.00e+00, 5.00e+00, 1.00e+01, 1.50e+01, 2.00e+01, 2.50e+01,
           3.00e+01, 4.00e+01, 5.00e+01, 6.00e+01, 7.50e+01, 1.00e+02, 1.25e+02,
           1.50e+02, 1.75e+02, 2.00e+02, 2.25e+02, 2.50e+02, 3.00e+02, 3.50e+02,
           4.00e+02, 4.50e+02, 5.00e+02, 5.50e+02, 6.00e+02, 7.50e+02, 1.00e+03,
           1.50e+03, 2.00e+03, 3.00e+03, 4.00e+03, 5.00e+03], dtype=float32)

  • lat
    (lat)
    float32
    46.0 46.01 46.03 ... 62.73 62.74

    axis :

    Y

    long_name :

    latitude

    nav_model :

    grid_T

    standard_name :

    latitude

    units :

    degrees_north

    array([46.     , 46.01351, 46.02703, ..., 62.71622, 62.72973, 62.74324],
          dtype=float32)

  • lon
    (lon)
    float32
    -16.0 -15.97 -15.94 ... 12.97 13.0

    axis :

    X

    long_name :

    longitude

    nav_model :

    grid_T

    standard_name :

    longitude

    units :

    degrees_east

    array([-16.     , -15.9697 , -15.93939, ...,  12.93939,  12.9697 ,  13.     ],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    2022-01-01T01:00:00 ... 2023-01-01

    axis :

    T

    standard_name :

    time

    long_name :

    Validity time

    chunksizes :

    [512]

    fletcher32 :

    False

    shuffle :

    True

    preferred_chunks :

    {'time': 512}

    zlib :

    True

    complevel :

    7

    source :

    <File-like object S3FileSystem, gfts-reference-data/NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013/MetO-NWS-PHY-hi-SSH_202012/2022/01/metoffice_foam1_amm15_NWS_SSH_b20220103_hi20220101.nc>

    original_shape :

    [24]

    dtype :

    float64

    array(['2022-01-01T01:00:00.000000000', '2022-01-01T02:00:00.000000000',
           '2022-01-01T03:00:00.000000000', ..., '2022-12-31T22:00:00.000000000',
           '2022-12-31T23:00:00.000000000', '2023-01-01T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (5)
  • deptho
    (lat, lon)
    float32
    dask.array<chunksize=(620, 479), meta=np.ndarray>

    long_name :

    Bathymetry

    standard_name :

    sea_floor_depth_below_geoid

    units :

    m

    Array Chunk Bytes 4.53 MiB 1.13 MiB Shape (1240, 958) (620, 479) Dask graph 4 chunks in 2 graph layers Data type float32 numpy.ndarray

  • mask
    (depth, lat, lon)
    float32
    dask.array<chunksize=(33, 1240, 958), meta=np.ndarray>

    long_name :

    Land-sea mask: sea = 1 ; land = 0

    standard_name :

    sea_binary_mask

    units :

    1

    Array Chunk Bytes 149.54 MiB 149.54 MiB Shape (33, 1240, 958) (33, 1240, 958) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • mdt
    (lat, lon)
    float32
    dask.array<chunksize=(1240, 958), meta=np.ndarray>

    long_name :

    Mean dynamic topography

    standard_name :

    sea_surface_height_above_geoid

    units :

    m

    Array Chunk Bytes 4.53 MiB 4.53 MiB Shape (1240, 958) (1240, 958) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • thetao
    (time, depth, lat, lon)
    float64
    dask.array<chunksize=(1, 11, 414, 320), meta=np.ndarray>

    long_name :

    Sea Water Potential Temperature

    standard_name :

    sea_water_potential_temperature

    units :

    degrees_C

    valid_max :

    30000

    valid_min :

    -30000

    Array Chunk Bytes 2.50 TiB 11.12 MiB Shape (8760, 33, 1240, 958) (1, 11, 414, 320) Dask graph 236520 chunks in 2 graph layers Data type float64 numpy.ndarray

  • zos
    (time, lat, lon)
    float64
    dask.array<chunksize=(1, 1240, 958), meta=np.ndarray>

    long_name :

    Sea surface height above geoid

    standard_name :

    sea_surface_height_above_geoid

    units :

    m

    valid_max :

    10000

    valid_min :

    -10000

    Array Chunk Bytes 77.53 GiB 9.06 MiB Shape (8760, 1240, 958) (1, 1240, 958) Dask graph 8760 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (4)
  • depth
    PandasIndex

    PandasIndex(Index([   0.0,    3.0,    5.0,   10.0,   15.0,   20.0,   25.0,   30.0,   40.0,
             50.0,   60.0,   75.0,  100.0,  125.0,  150.0,  175.0,  200.0,  225.0,
            250.0,  300.0,  350.0,  400.0,  450.0,  500.0,  550.0,  600.0,  750.0,
           1000.0, 1500.0, 2000.0, 3000.0, 4000.0, 5000.0],
          dtype='float32', name='depth'))

  • lat
    PandasIndex

    PandasIndex(Index([              46.0, 46.013511657714844,  46.02703094482422,
             46.0405387878418,  46.05405044555664, 46.067569732666016,
            46.08108139038086,  46.09458923339844,  46.10810852050781,
           46.121620178222656,
           ...
           62.621620178222656,  62.63513946533203, 62.648651123046875,
            62.66215896606445,   62.6756706237793,  62.68918991088867,
           62.702701568603516,  62.71622085571289,  62.72972869873047,
            62.74324035644531],
          dtype='float32', name='lat', length=1240))

  • lon
    PandasIndex

    PandasIndex(Index([              -16.0,  -15.96969985961914, -15.939390182495117,
           -15.909090042114258, -15.878789901733398, -15.848480224609375,
           -15.818180084228516, -15.787879943847656, -15.757579803466797,
           -15.727270126342773,
           ...
            12.727270126342773,  12.757579803466797,  12.787879943847656,
            12.818180084228516,  12.848489761352539,  12.878789901733398,
            12.909090042114258,  12.939390182495117,   12.96969985961914,
                          13.0],
          dtype='float32', name='lon', length=958))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-01-01 01:00:00', '2022-01-01 02:00:00',
                   '2022-01-01 03:00:00', '2022-01-01 04:00:00',
                   '2022-01-01 05:00:00', '2022-01-01 06:00:00',
                   '2022-01-01 07:00:00', '2022-01-01 08:00:00',
                   '2022-01-01 09:00:00', '2022-01-01 10:00:00',
                   ...
                   '2022-12-31 15:00:00', '2022-12-31 16:00:00',
                   '2022-12-31 17:00:00', '2022-12-31 18:00:00',
                   '2022-12-31 19:00:00', '2022-12-31 20:00:00',
                   '2022-12-31 21:00:00', '2022-12-31 22:00:00',
                   '2022-12-31 23:00:00', '2023-01-01 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=8760, freq=None))

• Attributes: (12)

  Conventions :

  CF-1.7

  contact :

  servicedesk.cmems@mercator-ocean.eu

  creation_date :

  2021-10-26T06:41:01Z

  credit :

  E.U. Copernicus Marine Service Information (CMEMS)

  forcing_data_source :

  ECMWF Global Atmospheric Model (HRES); UKMO NATL12; UKMO GLOWave; CMEMS-BALTIC; climatological rivers

  history :

  See source and creation_date attributes

  institution :

  UK Met Office

  licence :

  http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/

  netcdf-version-id :

  netCDF-4

  product :

  NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013

  references :

  http://marine.copernicus.eu/

  source :

  PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWATCH-III v4.18_NEMOVAR v6

Data checking

combined_ds.zos.mean(dim=["lon", "lat"]).plot()

[<matplotlib.lines.Line2D at 0x7f4e1eff10d0>]

Counting the data

combined_ds.sel(time="2022-01")  # 743 values along time

<xarray.Dataset> Size: 240GB
Dimensions:  (depth: 33, lat: 1240, lon: 958, time: 743)
Coordinates:
  * depth    (depth) float32 132B 0.0 3.0 5.0 10.0 ... 2e+03 3e+03 4e+03 5e+03
  * lat      (lat) float32 5kB 46.0 46.01 46.03 46.04 ... 62.7 62.72 62.73 62.74
  * lon      (lon) float32 4kB -16.0 -15.97 -15.94 -15.91 ... 12.94 12.97 13.0
  * time     (time) datetime64[ns] 6kB 2022-01-01T01:00:00 ... 2022-01-31T23:...
Data variables:
    deptho   (lat, lon) float32 5MB dask.array<chunksize=(620, 479), meta=np.ndarray>
    mask     (depth, lat, lon) float32 157MB dask.array<chunksize=(33, 1240, 958), meta=np.ndarray>
    mdt      (lat, lon) float32 5MB dask.array<chunksize=(1240, 958), meta=np.ndarray>
    thetao   (time, depth, lat, lon) float64 233GB dask.array<chunksize=(1, 11, 414, 320), meta=np.ndarray>
    zos      (time, lat, lon) float64 7GB dask.array<chunksize=(1, 1240, 958), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.7
    contact:              servicedesk.cmems@mercator-ocean.eu
    creation_date:        2021-10-26T06:41:01Z
    credit:               E.U. Copernicus Marine Service Information (CMEMS)
    forcing_data_source:  ECMWF Global Atmospheric Model (HRES); UKMO NATL12;...
    history:              See source and creation_date attributes
    institution:          UK Met Office
    licence:              http://marine.copernicus.eu/services-portfolio/serv...
    netcdf-version-id:    netCDF-4
    product:              NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013
    references:           http://marine.copernicus.eu/
    source:               PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWAT...

xarray.Dataset

• Dimensions:
  • depth: 33
  • lat: 1240
  • lon: 958
  • time: 743
• Coordinates: (4)
  • depth
    (depth)
    float32
    0.0 3.0 5.0 ... 3e+03 4e+03 5e+03

    axis :

    Z

    long_name :

    depth

    positive :

    down

    standard_name :

    depth

    units :

    m

    array([0.00e+00, 3.00e+00, 5.00e+00, 1.00e+01, 1.50e+01, 2.00e+01, 2.50e+01,
           3.00e+01, 4.00e+01, 5.00e+01, 6.00e+01, 7.50e+01, 1.00e+02, 1.25e+02,
           1.50e+02, 1.75e+02, 2.00e+02, 2.25e+02, 2.50e+02, 3.00e+02, 3.50e+02,
           4.00e+02, 4.50e+02, 5.00e+02, 5.50e+02, 6.00e+02, 7.50e+02, 1.00e+03,
           1.50e+03, 2.00e+03, 3.00e+03, 4.00e+03, 5.00e+03], dtype=float32)

  • lat
    (lat)
    float32
    46.0 46.01 46.03 ... 62.73 62.74

    axis :

    Y

    long_name :

    latitude

    nav_model :

    grid_T

    standard_name :

    latitude

    units :

    degrees_north

    array([46.     , 46.01351, 46.02703, ..., 62.71622, 62.72973, 62.74324],
          dtype=float32)

  • lon
    (lon)
    float32
    -16.0 -15.97 -15.94 ... 12.97 13.0

    axis :

    X

    long_name :

    longitude

    nav_model :

    grid_T

    standard_name :

    longitude

    units :

    degrees_east

    array([-16.     , -15.9697 , -15.93939, ...,  12.93939,  12.9697 ,  13.     ],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    2022-01-01T01:00:00 ... 2022-01-...

    axis :

    T

    standard_name :

    time

    long_name :

    Validity time

    chunksizes :

    [512]

    fletcher32 :

    False

    shuffle :

    True

    preferred_chunks :

    {'time': 512}

    zlib :

    True

    complevel :

    7

    source :

    <File-like object S3FileSystem, gfts-reference-data/NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013/MetO-NWS-PHY-hi-SSH_202012/2022/01/metoffice_foam1_amm15_NWS_SSH_b20220103_hi20220101.nc>

    original_shape :

    [24]

    dtype :

    float64

    array(['2022-01-01T01:00:00.000000000', '2022-01-01T02:00:00.000000000',
           '2022-01-01T03:00:00.000000000', ..., '2022-01-31T21:00:00.000000000',
           '2022-01-31T22:00:00.000000000', '2022-01-31T23:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (5)
  • deptho
    (lat, lon)
    float32
    dask.array<chunksize=(620, 479), meta=np.ndarray>

    long_name :

    Bathymetry

    standard_name :

    sea_floor_depth_below_geoid

    units :

    m

    Array Chunk Bytes 4.53 MiB 1.13 MiB Shape (1240, 958) (620, 479) Dask graph 4 chunks in 2 graph layers Data type float32 numpy.ndarray

  • mask
    (depth, lat, lon)
    float32
    dask.array<chunksize=(33, 1240, 958), meta=np.ndarray>

    long_name :

    Land-sea mask: sea = 1 ; land = 0

    standard_name :

    sea_binary_mask

    units :

    1

    Array Chunk Bytes 149.54 MiB 149.54 MiB Shape (33, 1240, 958) (33, 1240, 958) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • mdt
    (lat, lon)
    float32
    dask.array<chunksize=(1240, 958), meta=np.ndarray>

    long_name :

    Mean dynamic topography

    standard_name :

    sea_surface_height_above_geoid

    units :

    m

    Array Chunk Bytes 4.53 MiB 4.53 MiB Shape (1240, 958) (1240, 958) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • thetao
    (time, depth, lat, lon)
    float64
    dask.array<chunksize=(1, 11, 414, 320), meta=np.ndarray>

    long_name :

    Sea Water Potential Temperature

    standard_name :

    sea_water_potential_temperature

    units :

    degrees_C

    valid_max :

    30000

    valid_min :

    -30000

    Array Chunk Bytes 217.01 GiB 11.12 MiB Shape (743, 33, 1240, 958) (1, 11, 414, 320) Dask graph 20061 chunks in 3 graph layers Data type float64 numpy.ndarray

  • zos
    (time, lat, lon)
    float64
    dask.array<chunksize=(1, 1240, 958), meta=np.ndarray>

    long_name :

    Sea surface height above geoid

    standard_name :

    sea_surface_height_above_geoid

    units :

    m

    valid_max :

    10000

    valid_min :

    -10000

    Array Chunk Bytes 6.58 GiB 9.06 MiB Shape (743, 1240, 958) (1, 1240, 958) Dask graph 743 chunks in 3 graph layers Data type float64 numpy.ndarray

• Indexes: (4)
  • depth
    PandasIndex

    PandasIndex(Index([   0.0,    3.0,    5.0,   10.0,   15.0,   20.0,   25.0,   30.0,   40.0,
             50.0,   60.0,   75.0,  100.0,  125.0,  150.0,  175.0,  200.0,  225.0,
            250.0,  300.0,  350.0,  400.0,  450.0,  500.0,  550.0,  600.0,  750.0,
           1000.0, 1500.0, 2000.0, 3000.0, 4000.0, 5000.0],
          dtype='float32', name='depth'))

  • lat
    PandasIndex

    PandasIndex(Index([              46.0, 46.013511657714844,  46.02703094482422,
             46.0405387878418,  46.05405044555664, 46.067569732666016,
            46.08108139038086,  46.09458923339844,  46.10810852050781,
           46.121620178222656,
           ...
           62.621620178222656,  62.63513946533203, 62.648651123046875,
            62.66215896606445,   62.6756706237793,  62.68918991088867,
           62.702701568603516,  62.71622085571289,  62.72972869873047,
            62.74324035644531],
          dtype='float32', name='lat', length=1240))

  • lon
    PandasIndex

    PandasIndex(Index([              -16.0,  -15.96969985961914, -15.939390182495117,
           -15.909090042114258, -15.878789901733398, -15.848480224609375,
           -15.818180084228516, -15.787879943847656, -15.757579803466797,
           -15.727270126342773,
           ...
            12.727270126342773,  12.757579803466797,  12.787879943847656,
            12.818180084228516,  12.848489761352539,  12.878789901733398,
            12.909090042114258,  12.939390182495117,   12.96969985961914,
                          13.0],
          dtype='float32', name='lon', length=958))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-01-01 01:00:00', '2022-01-01 02:00:00',
                   '2022-01-01 03:00:00', '2022-01-01 04:00:00',
                   '2022-01-01 05:00:00', '2022-01-01 06:00:00',
                   '2022-01-01 07:00:00', '2022-01-01 08:00:00',
                   '2022-01-01 09:00:00', '2022-01-01 10:00:00',
                   ...
                   '2022-01-31 14:00:00', '2022-01-31 15:00:00',
                   '2022-01-31 16:00:00', '2022-01-31 17:00:00',
                   '2022-01-31 18:00:00', '2022-01-31 19:00:00',
                   '2022-01-31 20:00:00', '2022-01-31 21:00:00',
                   '2022-01-31 22:00:00', '2022-01-31 23:00:00'],
                  dtype='datetime64[ns]', name='time', length=743, freq=None))

• Attributes: (12)

  Conventions :

  CF-1.7

  contact :

  servicedesk.cmems@mercator-ocean.eu

  creation_date :

  2021-10-26T06:41:01Z

  credit :

  E.U. Copernicus Marine Service Information (CMEMS)

  forcing_data_source :

  ECMWF Global Atmospheric Model (HRES); UKMO NATL12; UKMO GLOWave; CMEMS-BALTIC; climatological rivers

  history :

  See source and creation_date attributes

  institution :

  UK Met Office

  licence :

  http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/

  netcdf-version-id :

  netCDF-4

  product :

  NORTHWESTSHELF_ANALYSIS_FORECAST_PHY_004_013

  references :

  http://marine.copernicus.eu/

  source :

  PS-OS 44, AMM-FOAM 1.5 km (tidal) NEMO v3.6_WAVEWATCH-III v4.18_NEMOVAR v6

# sums over lon and lat and compare the number of values to the one above
count = (
    combined_ds.zos.sel(time="2022-01").mean(dim=["lon", "lat"]).count().compute()
)  # 743 too

count

<xarray.DataArray 'zos' ()> Size: 8B
np.int64(743)

xarray.DataArray

'zos'

• 743

  np.int64(743)

• Coordinates: (0)
• Indexes: (0)
• Attributes: (0)