Reading CMEMS Copernicus data from GFTS s3 bucket

import intake
import hvplot.xarray  # noqa: F401

%%time
cat = intake.open_catalog("s3://gfts-reference-data/gfts_cmems_catalog.yaml")

CPU times: user 362 ms, sys: 60.4 ms, total: 423 ms
Wall time: 498 ms

list(cat)

['cmems_nws_2d', 'cmems_nws_3d', 'cmems_ibi_2d', 'cmems_ibi_3d']

cat["cmems_nws_2d"]

cmems_nws_2d:
  args:
    consolidated: false
    storage_options:
      fo: s3://gfts-reference-data/CMEMS_v6r1_NWS_PHY_NRT_NL_01hav_AN_2D_combined.parq
      remote_options:
        anon: false
      remote_protocol: s3
      target_options:
        anon: false
    urlpath: reference://
  description: Copernicus CMEMS_v6r1_NWS_PHY_NRT_NL_01hav_AN_2D data for GFTS
  driver: intake_xarray.xzarr.ZarrSource
  metadata:
    catalog_dir: s3://gfts-reference-data

%%time
ds = cat["cmems_nws_2d"].to_dask()

CPU times: user 2.13 s, sys: 229 ms, total: 2.36 s
Wall time: 2.7 s

/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

ds

<xarray.Dataset> Size: 521GB
Dimensions:    (latitude: 551, longitude: 936, time: 18048)
Coordinates:
  * latitude   (latitude) float32 2kB 46.0 46.03 46.06 ... 61.23 61.25 61.28
  * longitude  (longitude) float32 4kB -16.0 -15.97 -15.94 ... 9.921 9.949 9.977
  * time       (time) datetime64[ns] 144kB 2022-04-02T00:30:00 ... 2024-04-22...
Data variables:
    mlotst     (time, latitude, longitude) float64 74GB dask.array<chunksize=(1, 551, 936), meta=np.ndarray>
    thetao     (time, latitude, longitude) float64 74GB dask.array<chunksize=(1, 551, 936), meta=np.ndarray>
    ubar       (time, latitude, longitude) float64 74GB dask.array<chunksize=(1, 551, 936), meta=np.ndarray>
    uo         (time, latitude, longitude) float64 74GB dask.array<chunksize=(1, 551, 936), meta=np.ndarray>
    vbar       (time, latitude, longitude) float64 74GB dask.array<chunksize=(1, 551, 936), meta=np.ndarray>
    vo         (time, latitude, longitude) float64 74GB dask.array<chunksize=(1, 551, 936), meta=np.ndarray>
    zos        (time, latitude, longitude) float64 74GB dask.array<chunksize=(1, 551, 936), meta=np.ndarray>
Attributes: (12/13)
    Conventions:     CF-1.8
    comment:         
    contact:         https://marine.copernicus.eu/contact
    domain_name:     NWS36
    field_date:      20220402
    field_type:      mean
    ...              ...
    forecast_type:   analysis
    institution:     Nologin (Spain)
    licence:         https://marine.copernicus.eu/user-corner/service-commitm...
    references:      http://marine.copernicus.eu/
    source:          NEMO3.6
    title:           Ocean surface hourly mean fields for the North West Shel...

xarray.Dataset

• Dimensions:
  • latitude: 551
  • longitude: 936
  • time: 18048
• Coordinates: (3)
  • latitude
    (latitude)
    float32
    46.0 46.03 46.06 ... 61.25 61.28

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    step :

    0.02777863f

    unit_long :

    Degrees North

    units :

    degrees_north

    array([46.003643, 46.03142 , 46.0592  , ..., 61.22633 , 61.25411 , 61.281887],
          dtype=float32)

  • longitude
    (longitude)
    float32
    -16.0 -15.97 -15.94 ... 9.949 9.977

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    step :

    0.02777863f

    unit_long :

    Degrees East

    units :

    degrees_east

    array([-15.996015, -15.968236, -15.940457, ...,   9.921447,   9.949225,
             9.977004], dtype=float32)

  • time
    (time)
    datetime64[ns]
    2022-04-02T00:30:00 ... 2024-04-...

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array(['2022-04-02T00:30:00.000000000', '2022-04-02T01:30:00.000000000',
           '2022-04-02T02:30:00.000000000', ..., '2024-04-22T21:30:00.000000000',
           '2024-04-22T22:30:00.000000000', '2024-04-22T23:30:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (7)
  • mlotst
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 551, 936), meta=np.ndarray>

    easting :

    longitude

    latitude_max :

    61.2819f

    latitude_min :

    46.0036f

    long_name :

    Ocean mixed layer thickness defined by density

    longitude_max :

    9.977f

    longitude_min :

    -15.996f

    northing :

    latitude

    standard_name :

    ocean_mixed_layer_thickness_defined_by_sigma_theta

    unit_long :

    Meters

    units :

    m

    valid_max :

    20000

    valid_min :

    -20000

    z_max :

    0.494025f

    z_min :

    0.494025f

    Array Chunk Bytes 69.35 GiB 3.93 MiB Shape (18048, 551, 936) (1, 551, 936) Dask graph 18048 chunks in 2 graph layers Data type float64 numpy.ndarray

  • thetao
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 551, 936), meta=np.ndarray>

    easting :

    longitude

    latitude_max :

    61.2819f

    latitude_min :

    46.0036f

    long_name :

    Temperature

    longitude_max :

    9.977f

    longitude_min :

    -15.996f

    northing :

    latitude

    standard_name :

    sea_water_potential_temperature

    unit_long :

    degrees_C

    units :

    degrees_C

    valid_max :

    22000

    valid_min :

    -12000

    z_max :

    0.494025f

    z_min :

    0.494025f

    Array Chunk Bytes 69.35 GiB 3.93 MiB Shape (18048, 551, 936) (1, 551, 936) Dask graph 18048 chunks in 2 graph layers Data type float64 numpy.ndarray

  • ubar
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 551, 936), meta=np.ndarray>

    easting :

    longitude

    latitude_max :

    61.2819f

    latitude_min :

    46.0036f

    long_name :

    zonal barotropic velocity

    longitude_max :

    9.977f

    longitude_min :

    -15.996f

    northing :

    latitude

    standard_name :

    barotropic_eastward_sea_water_velocity

    unit_long :

    Meters per second

    units :

    m s-1

    valid_max :

    9000

    valid_min :

    -9000

    z_max :

    0.494025f

    z_min :

    0.494025f

    Array Chunk Bytes 69.35 GiB 3.93 MiB Shape (18048, 551, 936) (1, 551, 936) Dask graph 18048 chunks in 2 graph layers Data type float64 numpy.ndarray

  • uo
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 551, 936), meta=np.ndarray>

    easting :

    longitude

    latitude_max :

    61.2819f

    latitude_min :

    46.0036f

    long_name :

    Eastward velocity

    longitude_max :

    9.977f

    longitude_min :

    -15.996f

    northing :

    latitude

    standard_name :

    eastward_sea_water_velocity

    unit_long :

    Meters per second

    units :

    m s-1

    valid_max :

    9000

    valid_min :

    -9000

    z_max :

    0.494025f

    z_min :

    0.494025f

    Array Chunk Bytes 69.35 GiB 3.93 MiB Shape (18048, 551, 936) (1, 551, 936) Dask graph 18048 chunks in 2 graph layers Data type float64 numpy.ndarray

  • vbar
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 551, 936), meta=np.ndarray>

    easting :

    longitude

    latitude_max :

    61.2819f

    latitude_min :

    46.0036f

    long_name :

    meridional barotropic velocity

    longitude_max :

    9.977f

    longitude_min :

    -15.996f

    northing :

    latitude

    standard_name :

    barotropic_northward_sea_water_velocity

    unit_long :

    Meters per second

    units :

    m s-1

    valid_max :

    9000

    valid_min :

    -9000

    z_max :

    0.494025f

    z_min :

    0.494025f

    Array Chunk Bytes 69.35 GiB 3.93 MiB Shape (18048, 551, 936) (1, 551, 936) Dask graph 18048 chunks in 2 graph layers Data type float64 numpy.ndarray

  • vo
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 551, 936), meta=np.ndarray>

    easting :

    longitude

    latitude_max :

    61.2819f

    latitude_min :

    46.0036f

    long_name :

    Northward velocity

    longitude_max :

    9.977f

    longitude_min :

    -15.996f

    northing :

    latitude

    standard_name :

    northward_sea_water_velocity

    unit_long :

    Meters per second

    units :

    m s-1

    valid_max :

    9000

    valid_min :

    -9000

    z_max :

    0.494025f

    z_min :

    0.494025f

    Array Chunk Bytes 69.35 GiB 3.93 MiB Shape (18048, 551, 936) (1, 551, 936) Dask graph 18048 chunks in 2 graph layers Data type float64 numpy.ndarray

  • zos
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 551, 936), meta=np.ndarray>

    easting :

    longitude

    latitude_max :

    61.2819f

    latitude_min :

    46.0036f

    long_name :

    Sea surface height

    longitude_max :

    9.977f

    longitude_min :

    -15.996f

    northing :

    latitude

    standard_name :

    sea_surface_height_above_geoid

    unit_long :

    Meters

    units :

    m

    valid_max :

    10000

    valid_min :

    -10000

    z_max :

    0.494025f

    z_min :

    0.494025f

    Array Chunk Bytes 69.35 GiB 3.93 MiB Shape (18048, 551, 936) (1, 551, 936) Dask graph 18048 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • latitude
    PandasIndex

    PandasIndex(Index([ 46.00364303588867,  46.03142166137695, 46.059200286865234,
           46.086978912353516,   46.1147575378418,  46.14253616333008,
            46.17031478881836,  46.19809341430664,  46.22587203979492,
             46.2536506652832,
           ...
            61.03187942504883,  61.05965805053711,  61.08743667602539,
            61.11521530151367,  61.14299392700195, 61.170772552490234,
           61.198551177978516,   61.2263298034668,  61.25410842895508,
            61.28188705444336],
          dtype='float32', name='latitude', length=551))

  • longitude
    PandasIndex

    PandasIndex(Index([-15.996014595031738, -15.968235969543457, -15.940457344055176,
           -15.912678718566895, -15.884900093078613, -15.857121467590332,
            -15.82934284210205,  -15.80156421661377, -15.773785591125488,
           -15.746006965637207,
           ...
             9.726996421813965,   9.754775047302246,   9.782553672790527,
             9.810332298278809,    9.83811092376709,   9.865889549255371,
             9.893668174743652,   9.921446800231934,   9.949225425720215,
             9.977004051208496],
          dtype='float32', name='longitude', length=936))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-04-02 00:30:00', '2022-04-02 01:30:00',
                   '2022-04-02 02:30:00', '2022-04-02 03:30:00',
                   '2022-04-02 04:30:00', '2022-04-02 05:30:00',
                   '2022-04-02 06:30:00', '2022-04-02 07:30:00',
                   '2022-04-02 08:30:00', '2022-04-02 09:30:00',
                   ...
                   '2024-04-22 14:30:00', '2024-04-22 15:30:00',
                   '2024-04-22 16:30:00', '2024-04-22 17:30:00',
                   '2024-04-22 18:30:00', '2024-04-22 19:30:00',
                   '2024-04-22 20:30:00', '2024-04-22 21:30:00',
                   '2024-04-22 22:30:00', '2024-04-22 23:30:00'],
                  dtype='datetime64[ns]', name='time', length=18048, freq=None))

• Attributes: (13)

  Conventions :

  CF-1.8

  comment :

  contact :

  https://marine.copernicus.eu/contact

  domain_name :

  NWS36

  field_date :

  20220402

  field_type :

  mean

  forecast_range :

  04-day_analysis

  forecast_type :

  analysis

  institution :

  Nologin (Spain)

  licence :

  https://marine.copernicus.eu/user-corner/service-commitments-and-licence

  references :

  http://marine.copernicus.eu/

  source :

  NEMO3.6

  title :

  Ocean surface hourly mean fields for the North West Shelf (NWS) region

da = ds["thetao"].sel(latitude=48.23, longitude=-7.154, method="nearest").load()

da

<xarray.DataArray 'thetao' (time: 18048)> Size: 144kB
array([11.50500007, 11.51700007, 11.53400007, ..., 12.0570001 ,
       12.0500001 , 12.0440001 ])
Coordinates:
    latitude   float32 4B 48.23
    longitude  float32 4B -7.162
  * time       (time) datetime64[ns] 144kB 2022-04-02T00:30:00 ... 2024-04-22...
Attributes: (12/14)
    easting:        longitude
    latitude_max:   61.2819f
    latitude_min:   46.0036f
    long_name:      Temperature
    longitude_max:  9.977f
    longitude_min:  -15.996f
    ...             ...
    unit_long:      degrees_C
    units:          degrees_C
    valid_max:      22000
    valid_min:      -12000
    z_max:          0.494025f
    z_min:          0.494025f

xarray.DataArray

'thetao'

• time: 18048

• 11.51 11.52 11.53 11.55 11.57 11.58 ... 12.07 12.06 12.06 12.05 12.04

  array([11.50500007, 11.51700007, 11.53400007, ..., 12.0570001 ,
         12.0500001 , 12.0440001 ])

• Coordinates: (3)
  • latitude
    ()
    float32
    48.23

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    step :

    0.02777863f

    unit_long :

    Degrees North

    units :

    degrees_north

    array(48.225933, dtype=float32)

  • longitude
    ()
    float32
    -7.162

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    step :

    0.02777863f

    unit_long :

    Degrees East

    units :

    degrees_east

    array(-7.1624103, dtype=float32)

  • time
    (time)
    datetime64[ns]
    2022-04-02T00:30:00 ... 2024-04-...

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array(['2022-04-02T00:30:00.000000000', '2022-04-02T01:30:00.000000000',
           '2022-04-02T02:30:00.000000000', ..., '2024-04-22T21:30:00.000000000',
           '2024-04-22T22:30:00.000000000', '2024-04-22T23:30:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-04-02 00:30:00', '2022-04-02 01:30:00',
                   '2022-04-02 02:30:00', '2022-04-02 03:30:00',
                   '2022-04-02 04:30:00', '2022-04-02 05:30:00',
                   '2022-04-02 06:30:00', '2022-04-02 07:30:00',
                   '2022-04-02 08:30:00', '2022-04-02 09:30:00',
                   ...
                   '2024-04-22 14:30:00', '2024-04-22 15:30:00',
                   '2024-04-22 16:30:00', '2024-04-22 17:30:00',
                   '2024-04-22 18:30:00', '2024-04-22 19:30:00',
                   '2024-04-22 20:30:00', '2024-04-22 21:30:00',
                   '2024-04-22 22:30:00', '2024-04-22 23:30:00'],
                  dtype='datetime64[ns]', name='time', length=18048, freq=None))

• Attributes: (14)

  easting :

  longitude

  latitude_max :

  61.2819f

  latitude_min :

  46.0036f

  long_name :

  Temperature

  longitude_max :

  9.977f

  longitude_min :

  -15.996f

  northing :

  latitude

  standard_name :

  sea_water_potential_temperature

  unit_long :

  degrees_C

  units :

  degrees_C

  valid_max :

  22000

  valid_min :

  -12000

  z_max :

  0.494025f

  z_min :

  0.494025f

da.hvplot()