07 NDarray Demo

07 NDarray Demo#

UW Geospatial Data Analysis
CEE467/CEWA567
David Shean, Eric Gagliano, Quinn Brencher

Objectives#

In modules 4 and 6 we used rioxarray to read and analyze rasters which were represented as xarray DataArray objects. Module 7 introduces advanced capabilities of Xarray for handling multidimensional geospatial data. In particular, we will work with N-dimensional arrays (NDarrays) that incorporate a time dimension alongside spatial coordinates, enabling time series analysis of raster data. Xarray’s tutorial material is well-developed, and their Xarray in 45 minutes is especially helpful in getting up to speed with some of the most important capabilities. With slight modification to include reprojection and clipping as well as excluding a couple of items, we’ve copied their tutorial notebook below. For more Xarray learning material, check out the Xarray Tutorial website.

Xarray citations#

S. Hoyer and J. Hamman. Xarray: N-D labeled arrays and datasets in Python. Journal of Open Research Software, 2017. URL: https://doi.org/10.5334/jors.148, doi:10.5334/jors.148.

Stephan Hoyer, Clark Fitzgerald, Joe Hamman, and others. Xarray: v2022.3.0. May 2022. URL: https://doi.org/10.5281/zenodo.59499, doi:10.5281/zenodo.59499.

Xarray in 45 minutes#

In this lesson, we cover the basics of Xarray data structures. By the end of the lesson, we will be able to:

Understand the basic data structures in Xarray
Inspect DataArray and Dataset objects.
Read and write netCDF files using Xarray.
Understand that there are many packages that build on top of xarray

We’ll start by reviewing the various components of the Xarray data model, represented here visually:

https://docs.xarray.dev/en/stable/_images/dataset-diagram.png

import matplotlib.pyplot as plt
import numpy as np
import xarray as xr

xr.set_options(keep_attrs=True, display_expand_data=False)
np.set_printoptions(threshold=10, edgeitems=2)

%xmode minimal
%matplotlib inline
%config InlineBackend.figure_format='retina'

Exception reporting mode: Minimal

Xarray has a few small real-world tutorial datasets hosted in the xarray-data GitHub repository.

xarray.tutorial.load_dataset is a convenience function to download and open DataSets by name (listed at that link).

Here we’ll use air temperature from the National Center for Environmental Prediction. Xarray objects have convenient HTML representations to give an overview of what we’re working with:

Note that behind the scenes the tutorial.open_dataset downloads a file. It then uses xarray.open_dataset function to open that file (which for this datasets is a netCDF file).

A few things are done automatically upon opening, but controlled by keyword arguments. For example, try passing the keyword argument mask_and_scale=False… what happens?

Why Xarray?#

Metadata provides context and provides code that is more legible. This reduces the likelihood of errors from typos and makes analysis more intuitive and fun!

Analysis without xarray: `X(`#

# plot the first timestep
lat = ds.air.lat.data  # numpy array
lon = ds.air.lon.data  # numpy array
temp = ds.air.data  # numpy array

plt.figure()
plt.pcolormesh(lon, lat, temp[0, :, :]);

../../_images/0f96503c4c06cc68a0a7ea80a03a5ac621575cb36e7643582ede264dd5377350.png

temp.mean(axis=1)  ## what did I just do? I can't tell by looking at this line.

array([[279.398 , 279.6664, ..., 280.3152, 280.6624],
       [279.0572, 279.538 , ..., 280.27  , 280.7976],
       ...,
       [279.398 , 279.666 , ..., 280.342 , 280.834 ],
       [279.27  , 279.354 , ..., 279.97  , 280.482 ]])

Analysis with xarray `=)`#

How readable is this code?

ds.air.isel(time=0).plot(x="lon");

../../_images/763b33f6bc92060b4274e756ca10d9b6192fda13b0c3655e594e06679b6ddd56.png

Use dimension names instead of axis numbers

ds.air.mean(dim="time").plot(x="lon")

<matplotlib.collections.QuadMesh at 0x7f2ff96e3920>

../../_images/6434c79f91e400e0b911b1e792327f144c5058302e555376746d5c801e1bc665.png

Visualization#

(.plot)

We have seen very simple plots earlier. Xarray also lets you easily visualize 3D and 4D datasets by presenting multiple facets (or panels or subplots) showing variations across rows and/or columns.

# facet the seasonal_mean
seasonal_mean.air.plot(col="season", col_wrap=2);

../../_images/f934f8dd8a75a8e627a17052ba666cbce9e8209cbc4a7f6c666373a3cb8e9fec.png

# contours
seasonal_mean.air.plot.contour(col="season", levels=20, add_colorbar=True);

../../_images/1e7c3f6e6c44aefbb90095922b5796f5d90fcef70ca7b0375623708754f233ff.png

# line plots too? wut
seasonal_mean.air.mean("lon").plot.line(hue="season", y="lat");

../../_images/357e1ebfde673736dd3d8dbccd9ac7da370813d2cfe8c552ad8f33fa79ad9156.png

For more see the user guide, the gallery, and the tutorial material.

The scientific python ecosystem#

Xarray ties in to the larger scientific python ecosystem and in turn many packages build on top of xarray. A long list of such packages is here: https://docs.xarray.dev/en/stable/related-projects.html.

Now we will demonstrate some cool features.

Pandas: tabular data structures#

You can easily convert between xarray and pandas structures. This allows you to conveniently use the extensive pandas ecosystem of packages (like seaborn) for your work.

# convert to pandas dataframe
df = ds.isel(time=slice(10)).to_dataframe()
df

			air	spatial_ref
lat	time	lon
75.0	2013-01-01 00:00:00	200.0	-25.510	0
		202.5	-23.170	0
		205.0	-21.370	0
		207.5	-20.470	0
		210.0	-20.290	0
...	...	...	...	...
15.0	2013-01-03 06:00:00	320.0	74.930	0
		322.5	75.452	0
		325.0	74.750	0
		327.5	74.552	0
		330.0	75.110	0

13250 rows × 2 columns

Alternative array types#

This notebook has focused on Numpy arrays. Xarray can wrap other array types! For example:

distributed parallel arrays & Xarray user guide on Dask

pydata/sparse : sparse arrays

GPU arrays & cupy-xarray

pint : unit-aware arrays & pint-xarray

Dask#

Dask cuts up NumPy arrays into blocks and parallelizes your analysis code across these blocks

# demonstrate dask dataset
dasky = xr.tutorial.open_dataset(
    "air_temperature",
    chunks={"time": 10},  # 10 time steps in each block
)

dasky.air

<xarray.DataArray 'air' (time: 2920, lat: 25, lon: 53)> Size: 31MB
dask.array<chunksize=(10, 25, 53), meta=np.ndarray>
Coordinates:
  * lat      (lat) float32 100B 75.0 72.5 70.0 67.5 65.0 ... 22.5 20.0 17.5 15.0
  * lon      (lon) float32 212B 200.0 202.5 205.0 207.5 ... 325.0 327.5 330.0
  * time     (time) datetime64[ns] 23kB 2013-01-01 ... 2014-12-31T18:00:00
Attributes:
    long_name:     4xDaily Air temperature at sigma level 995
    units:         degK
    precision:     2
    GRIB_id:       11
    GRIB_name:     TMP
    var_desc:      Air temperature
    dataset:       NMC Reanalysis
    level_desc:    Surface
    statistic:     Individual Obs
    parent_stat:   Other
    actual_range:  [185.16 322.1 ]

All computations with dask-backed xarray objects are lazy, allowing you to build up a complicated chain of analysis steps quickly

# demonstrate lazy mean
dasky.air.mean("lat")

<xarray.DataArray 'air' (time: 2920, lon: 53)> Size: 1MB
dask.array<chunksize=(10, 53), meta=np.ndarray>
Coordinates:
  * lon      (lon) float32 212B 200.0 202.5 205.0 207.5 ... 325.0 327.5 330.0
  * time     (time) datetime64[ns] 23kB 2013-01-01 ... 2014-12-31T18:00:00
Attributes:
    long_name:     4xDaily Air temperature at sigma level 995
    units:         degK
    precision:     2
    GRIB_id:       11
    GRIB_name:     TMP
    var_desc:      Air temperature
    dataset:       NMC Reanalysis
    level_desc:    Surface
    statistic:     Individual Obs
    parent_stat:   Other
    actual_range:  [185.16 322.1 ]

To get concrete values, call .compute or .load

# import hvplot.xarray
# ds.air.hvplot(groupby="time", clim=(270, 300), widget_location='bottom')

Other cool packages#

hvplot : quickly generate interactive plots from your data
xgcm : grid-aware operations with xarray objects
xrft : fourier transforms with xarray
xclim : calculating climate indices with xarray objects
intake-xarray : forget about file paths
rioxarray : raster files and xarray
xesmf : regrid using ESMF
MetPy : tools for working with weather data

Check the Xarray Ecosystem page and this tutorial for even more packages and demonstrations.

Next#

Read the tutorial material and user guide
See the description of common terms used in the xarray documentation:
Answers to common questions on “how to do X” with Xarray are here
Ryan Abernathey has a book on data analysis with a chapter on Xarray
Project Pythia has foundational and more advanced material on Xarray. Pythia also aggregates other Python learning resources.
The Xarray Github Discussions and Pangeo Discourse are good places to ask questions.
Tell your friends! Tweet!

Welcome!#

Xarray is an open-source project and gladly welcomes all kinds of contributions. This could include reporting bugs, discussing new enhancements, contributing code, helping answer user questions, contributing documentation (even small edits like fixing spelling mistakes or rewording to make the text clearer). Welcome!

07 NDarray Demo

Contents

07 NDarray Demo#

Objectives#

Xarray citations#

Xarray in 45 minutes#

What’s in a Dataset?#

What’s in a DataArray?#

Name (optional)#

Named dimensions#

Coordinate variables#

Arbitrary attributes#

Underlying data#

Review#

Why Xarray?#

Analysis without xarray: `X(`#

Analysis with xarray `=)`#

Extracting data or “indexing”#

Label-based indexing#

Position-based indexing#

Concepts for computation#

Broadcasting: expanding data#

Alignment: putting data on the same grid#

High level computation#

groupby#

resample#

weighted#

Visualization#

Band math / convert variables#

Reprojection and clipping#

Reading and writing files#

The scientific python ecosystem#

Pandas: tabular data structures#

Alternative array types#

Dask#

Other cool packages#

Next#

Welcome!#

07 NDarray Demo

Contents

07 NDarray Demo#

Objectives#

Xarray citations#

Xarray in 45 minutes#

What’s in a Dataset?#

What’s in a DataArray?#

Name (optional)#

Named dimensions#

Coordinate variables#

Arbitrary attributes#

Underlying data#

Review#

Why Xarray?#

Analysis without xarray: X(#

Analysis with xarray =)#

Extracting data or “indexing”#

Label-based indexing#

Position-based indexing#

Concepts for computation#

Broadcasting: expanding data#

Alignment: putting data on the same grid#

High level computation#

groupby#

resample#

weighted#

Visualization#

Band math / convert variables#

Reprojection and clipping#

Reading and writing files#

The scientific python ecosystem#

Pandas: tabular data structures#

Alternative array types#

Dask#

Other cool packages#

Next#

Welcome!#

Analysis without xarray: `X(`#

Analysis with xarray `=)`#