diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index fe5f4559..4fa4c46d 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -47,6 +47,5 @@ repos: additional_dependencies: [black] - id: nbqa-pyupgrade additional_dependencies: [pyupgrade] - exclude: foundations/quickstart.ipynb - id: nbqa-isort additional_dependencies: [isort] diff --git a/README.md b/README.md index 128d7c3b..8831fcb0 100644 --- a/README.md +++ b/README.md @@ -9,25 +9,28 @@ [](https://binder.projectpythia.org/v2/gh/ProjectPythia/unstructured-grid-viz-cookbook.git/main?labpath=notebooks) [](https://doi.org/10.5281/zenodo.10403389) -This Cookbook is a comprehensive showcase of workflows & techniques for visualizing Unstructured Grids using [UXarray](https://uxarray.readthedocs.io/). +This Cookbook is a comprehensive showcase of workflows & techniques for visualizing Unstructured Grids using [UXarray](https://uxarray.readthedocs.io/), +also providing foundational information on unstructured grids. ## Motivation -The ability to natively visualize unstructured grids is a much-needed ability within the Scientific Python Ecosystem, +The ability to natively visualize unstructured grids is much needed within the Scientific Python Ecosystem, which poses multiple challenges and needs to: -- Not regrid the source unstructured grid to structure +- Not regrid the source unstructured grid to structured grid - Take advantage of grid information, such as connectivity variables - Limit the amount of pre-processing needed to prepare the data for Python visualization tools -UXarray enables such visualization methods that operate directly on unstructured grid data, -providing Xarray-styled functionality to better read in -and use unstructured grid datasets that follow standard conventions. -UXarray supports a variety of unstructured grid formats including UGRID, MPAS, SCRIP, and Exodus, -and is extendable for other formats. +UXarray enables such visualization methods that operate directly on unstructured grid data, providing +Xarray-styled functionality to better read in and use unstructured grid datasets that follow standard +conventions. -This cookbook covers an introduction to unstructured grids and UXarray, -provides an overview of the visualization methods and libraries, and showcases several UXarray visualization functions. +UXarray supports a variety of unstructured grid formats and file types including UGRID, MPAS, ICON, CAM-SE, +SCRIP, Exodus, ESMF, GEOS, and FESOM2, and is extensible for other formats. + +This cookbook covers an introduction to unstructured grids and UXarray from a visualization standpoint, +providing foundational information about unstructured grids, visualization methods and libraries, and +introducing UXarray, and showcasing several UXarray visualization functions and workflows. ## Authors @@ -47,19 +50,32 @@ provides an overview of the visualization methods and libraries, and showcases s ## Structure -This cookbook is split up into a few chapters that provide a detailed overview of how to use UXarray to work with and visualize unstructured grid datasets: +This cookbook is split up into several chapters to communicate the content efffectively with different +levels of readers: + +**1. Foundations** + +Here, we cover overview of the foundational topics necessary to understand the content in this cookbook, +e.g. what unstructured grids are and how they are different than structured grids, what plotting libraries +and visualization techniques exist that can be helpful for unstructured grid visualization, and we briefly +mention how UXarray is related to these. -**1. Introduction to UXarray & Unstructured Grids** +**2. Introduction to UXarray** -Here we cover what unstructured grids are and how they are different than structured grids as well as whay UXarray could play a significant role in unstructured grid visualization. +In this chapter, we provide an overview of UXarray: An Xarray-extension for unstructured grid-formatted +climate and global weather data analysis and visualization. -**2. Methods & Libraries for Unstructured Grid Visualization** +**3. Plotting with UXarray** -In this chapter, we briefly introduce plotting libraries and their specific technologies as well as rendering techniques that could be used for unstructured grid plotting and are used as part of UXarray. +We provide an overview of the UXarray plotting API along with several visualization functionality, and cases +and examples that can be realized using such UXarray functionality; Grid visualization, Data visualization, +Geographic projections and features, to name a few. Also in this section, customization and interactivaity +with UXarray plotting and considerations with high-resolution plotting are also provided. -**3. UXarray Visualization** +**4. Visualization Recipies** -Several visualization cases and examples that can be realized using UXarray are provided in this chapter; grid topology plots, polygons, points, to name a few. Also in this section, the usage of UXarray plotting API and a discussion of visualization at scale are also provided. +In this last chapter, we offer to the interested readers a set of focused workflows that can be realized +with UXarray, including visualizations of MPAS and E3SM model output. ## Running the Notebooks diff --git a/_config.yml b/_config.yml index bef5b978..db383ccc 100644 --- a/_config.yml +++ b/_config.yml @@ -5,7 +5,7 @@ title: Unstructured Grids Visualization Cookbook description: Comprehensive showcase of workflows and techniques for visualizing Unstructured Grids using UXarray author: the Project Pythia Community logo: notebooks/images/logos/pythia_logo-white-rtext.svg -copyright: "2024" +copyright: "2025" execute: # To execute notebooks via a Binder instead, replace 'cache' with 'binder' diff --git a/_static/custom.css b/_static/custom.css index b0e83df1..59f43131 100644 --- a/_static/custom.css +++ b/_static/custom.css @@ -1,6 +1,6 @@ .bd-main .bd-content .bd-article-container { - max-width: 100%; /* default is 60em */ + max-width: 100%; /* default is 60em */ } .bd-page-width { - max-width: 100%; /* default is 88rem */ + max-width: 100%; /* default is 88rem */ } diff --git a/_static/images/grids.png b/_static/images/grids.png new file mode 100644 index 00000000..d5bfc8e7 Binary files /dev/null and b/_static/images/grids.png differ diff --git a/_static/images/uxarray-design.png b/_static/images/uxarray-design.png new file mode 100644 index 00000000..1d9d386e Binary files /dev/null and b/_static/images/uxarray-design.png differ diff --git a/_templates/footer-extra.html b/_templates/footer-extra.html index 27cb915d..2b794d89 100644 --- a/_templates/footer-extra.html +++ b/_templates/footer-extra.html @@ -1,27 +1,27 @@
diff --git a/_toc.yml b/_toc.yml index bac8f544..9b277ee2 100644 --- a/_toc.yml +++ b/_toc.yml @@ -4,23 +4,37 @@ parts: - caption: Preamble chapters: - file: notebooks/how-to-cite - - caption: Introduction to UXarray & Unstructured Grids + + - caption: Foundations + chapters: + - file: notebooks/01-foundations/unstructured-grids + - file: notebooks/01-foundations/plotting-libs + - file: notebooks/01-foundations/rendering-techniques + + - caption: Introduction to UXarray chapters: - - file: notebooks/01-intro/01-unstructured-grid-overview - - file: notebooks/01-intro/02-data-structures - - file: notebooks/01-intro/03-data-mapping + - file: notebooks/02-intro-to-uxarray/overview + - file: notebooks/02-intro-to-uxarray/grid + - file: notebooks/02-intro-to-uxarray/uxds-uxda + - file: notebooks/02-intro-to-uxarray/selection - - caption: Methods & Libraries for Unstructured Grid Visualization + - caption: Plotting with UXarray chapters: - - file: notebooks/02-methods/01-plotting-libraries - - file: notebooks/02-methods/02-rendering-techniques + - file: notebooks/03-plotting-with-uxarray/grid-viz + - file: notebooks/03-plotting-with-uxarray/data-viz + - file: notebooks/03-plotting-with-uxarray/geo + - file: notebooks/03-plotting-with-uxarray/customization + - file: notebooks/03-plotting-with-uxarray/high-res + - file: notebooks/03-plotting-with-uxarray/compare-xarray - - caption: UXarray Visualization + - caption: Visualization Recipes chapters: - - file: notebooks/03-uxarray-vis/01-plot-api - - file: notebooks/03-uxarray-vis/002-xarray-to-uxarray - - file: notebooks/03-uxarray-vis/02-grid-topology - - file: notebooks/03-uxarray-vis/03-polygons - - file: notebooks/03-uxarray-vis/04-points - - file: notebooks/03-uxarray-vis/07-animations - - file: notebooks/03-uxarray-vis/06-performance + - file: notebooks/04-recipes/mpas-atmo + - file: notebooks/04-recipes/mpas-ocean + # - file: notebooks/04-recipes/mpas-regional + - file: notebooks/04-recipes/e3sm +# - caption: Compatibility with Visualization Packages +# chapters: +# - file: notebooks/05-viz-packages/matplotlib +# - file: notebooks/05-viz-packages/datashader +# - file: notebooks/05-viz-packages/lonboard diff --git a/meshfiles/hex.data.nc b/meshfiles/hex.data.nc new file mode 100644 index 00000000..5fcd36bb Binary files /dev/null and b/meshfiles/hex.data.nc differ diff --git a/meshfiles/hex.grid.nc b/meshfiles/hex.grid.nc new file mode 100644 index 00000000..aa38e3d4 Binary files /dev/null and b/meshfiles/hex.grid.nc differ diff --git a/meshfiles/hex.node.data.nc b/meshfiles/hex.node.data.nc new file mode 100644 index 00000000..fa13a90d Binary files /dev/null and b/meshfiles/hex.node.data.nc differ diff --git a/meshfiles/ne30pg2.data.nc b/meshfiles/ne30pg2.data.nc new file mode 100644 index 00000000..edc7db42 Binary files /dev/null and b/meshfiles/ne30pg2.data.nc differ diff --git a/meshfiles/ne30pg2.grid.nc b/meshfiles/ne30pg2.grid.nc new file mode 100644 index 00000000..ae86055f Binary files /dev/null and b/meshfiles/ne30pg2.grid.nc differ diff --git a/meshfiles/x1.655362.data.nc b/meshfiles/x1.655362.data.nc new file mode 100644 index 00000000..87f02794 Binary files /dev/null and b/meshfiles/x1.655362.data.nc differ diff --git a/meshfiles/x1.655362.grid.nc b/meshfiles/x1.655362.grid.nc new file mode 100644 index 00000000..148a63ab Binary files /dev/null and b/meshfiles/x1.655362.grid.nc differ diff --git a/notebooks/01-foundations/plotting-libs.ipynb b/notebooks/01-foundations/plotting-libs.ipynb new file mode 100644 index 00000000..23bdb08e --- /dev/null +++ b/notebooks/01-foundations/plotting-libs.ipynb @@ -0,0 +1,371 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a107dfa27c1b5b7b", + "metadata": { + "panel-layout": { + "height": 423.611, + "visible": true, + "width": 100 + } + }, + "source": [ + "Note:
\n", + " The selection between vector graphics and rasterization needs to be made taking into account several factors such as how large is the dataset (i.e. how fine-resolution the data is), what data fidelity with the visualization is desired, what performance is expected, etc.\n", + "Note:
\n", + " This is a very basic example to show how an unstructured grid with triangles would look like. In actual model outputs, often the region of interest is meshed with a finer resolution, while it is coarsened in areas where high resolution is not needed. This is done to reduce the number of elements and improve computational efficiency.\n", + "Note:
\n", - " This is a very basic example of an unstructured grid with triangles. There are very specialized libraries to create\n", - " unstructured grids. Often the region of interest is meshed with a finer resolution. The mesh is then coarsened in\n", - " areas where the resolution is not needed. This is done to reduce the number of elements and improve computational\n", - " efficiency.\n", - "Note:
\n", - " This notebook serves as an introduction to unstructured grids and UXarray. For more information, please visit the\n", - " UXarray documentation and specifically see the \n", - " Usage examples section.\n", - "Note:
\n", + " In most cases, checking the ``Grid`` object of either ``UxDataset`` or ``UxDataArray`` is anticipated be the common scenario since majority of the UXarray workflows will rely on a data set and its variable(s) of interest. Such cases will be showcased as part of many of the following notebooks; thus, this tutorial will focus on the latter where we explore a standalone ``Grid`` object.\n", + "Note:
\n", + " The following notebooks will detail each of these data structures; hence, we are keeping it short here in this section.\n", + "Note!
\n", + " The visualizations throughout this tutorial are only for demonstrating the results of the subsetting and cross-sections versus the global grid. Since the details of plotting with UXarray will be covered in the next chapter, we will not go over any details of the plots here.\n", + "<xarray.UxDataset> Size: 43kB\n", + "Dimensions: (n_face: 5400)\n", + "Dimensions without coordinates: n_face\n", + "Data variables:\n", + " psi (n_face) float64 43kB ...
<uxarray.Grid>\n", + "Original Grid Type: UGRID\n", + "Grid Dimensions:\n", + " * n_node: 5402\n", + " * n_face: 5400\n", + " * n_max_face_nodes: 4\n", + " * n_nodes_per_face: (5400,)\n", + "Grid Coordinates (Spherical):\n", + " * node_lon: (5402,)\n", + " * node_lat: (5402,)\n", + "Grid Coordinates (Cartesian):\n", + "Grid Connectivity Variables:\n", + " * face_node_connectivity: (5400, 4)\n", + "Grid Descriptor Variables:\n", + " * n_nodes_per_face: (5400,)\n", + "
<xarray.UxDataArray 'psi' (n_face: 5400)> Size: 43kB\n", + "[5400 values with dtype=float64]\n", + "Dimensions without coordinates: n_face
<uxarray.Grid>\n", + "Original Grid Type: UGRID\n", + "Grid Dimensions:\n", + " * n_node: 5402\n", + " * n_face: 5400\n", + " * n_max_face_nodes: 4\n", + " * n_nodes_per_face: (5400,)\n", + "Grid Coordinates (Spherical):\n", + " * node_lon: (5402,)\n", + " * node_lat: (5402,)\n", + "Grid Coordinates (Cartesian):\n", + "Grid Connectivity Variables:\n", + " * face_node_connectivity: (5400, 4)\n", + "Grid Descriptor Variables:\n", + " * n_nodes_per_face: (5400,)\n", + "
<xarray.Dataset> Size: 30kB\n", + "Dimensions: (lat: 45, lon: 80)\n", + "Coordinates:\n", + " * lat (lat) int64 360B -90 -86 -82 -78 -74 -70 -66 ... 66 70 74 78 82 86\n", + " * lon (lon) float64 640B -180.0 -175.5 -171.0 ... 166.5 171.0 175.5\n", + "Data variables:\n", + " psi (lat, lon) float64 29kB ...
<xarray.DataArray 'psi' (lat: 45, lon: 80)> Size: 29kB\n", + "[3600 values with dtype=float64]\n", + "Coordinates:\n", + " * lat (lat) int64 360B -90 -86 -82 -78 -74 -70 -66 ... 66 70 74 78 82 86\n", + " * lon (lon) float64 640B -180.0 -175.5 -171.0 ... 166.5 171.0 175.5\n", + "Attributes:\n", + " regrid_method: nearest_s2d
See also:
\n", + " For further information, please visit the \n", + " UXarray Inheritance from Xarray documentation\n", + "Note:
\n", - " The following design diagram is actually provided in the Plotting\n", - " API section along with key takeaways about it. We highly recommend to check them out as well.\n", - "Important!
\n", - " UXarray takes care of the vital tasks such as recognizing unstructured grids from various formats (such as UGRID,\n", - " MPAS, Scrip, Exodus, etc.) and representing them in a unified UGRID-like format, providing the data structures and\n", - " functionality necessary for convenient preprocessing of the grid, and wrapping up HoloViz packages' functionality to\n", - " enable unstructured grids-specialized, high-level visualization functions.\n", - "Note:
\n", - " UXarray allows both Matplotlib and Bokeh backends to be chosen in visualization functions as they are provided by Holoviews (in addition to Plotly).\n", - "Warning!
\n", - " While these conversion functions have already been released, we have observed some issues with the resulting Matplotlib plots after the exceution of\n", - " these functions, and the bug-fixing of that is WIP. Hence, we don't have an officially released documentation/example about these functions yet.\n", - "<xarray.Dataset> Size: 30kB\n", + "Dimensions: (lat: 45, lon: 80)\n", + "Coordinates:\n", + " * lat (lat) int64 360B -90 -86 -82 -78 -74 -70 -66 ... 66 70 74 78 82 86\n", + " * lon (lon) float64 640B -180.0 -175.5 -171.0 ... 166.5 171.0 175.5\n", + "Data variables:\n", + " psi (lat, lon) float64 29kB ...
<xarray.UxDataset> Size: 43kB\n", + "Dimensions: (n_face: 5400)\n", + "Dimensions without coordinates: n_face\n", + "Data variables:\n", + " psi (n_face) float64 43kB ...
See also:
\n", + " To learn more about hvPlot and Xarray, please refer to the\n", + " hvPlot Documentation\n", + "<cartopy.crs.Orthographic object at 0x000001CB15F7E570>" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 9 + }, + { + "metadata": { + "ExecuteTime": { + "end_time": "2025-01-08T20:05:38.572835Z", + "start_time": "2025-01-08T20:05:37.914878Z" + } + }, + "cell_type": "code", + "source": [ + "uxds[\"bottomDepth\"].plot.polygons(projection=projection)" + ], + "id": "635c3b24f12592b7", + "outputs": [ + { + "data": {}, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
<cartopy.crs.Orthographic object at 0x000001CB13D4BB60>" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 5 + }, + { + "metadata": { + "ExecuteTime": { + "end_time": "2025-01-08T06:16:20.874469Z", + "start_time": "2025-01-08T06:16:20.231620Z" + } + }, + "cell_type": "code", + "source": [ + "uxds[\"bottomDepth\"].plot.polygons(projection=projection)" + ], + "id": "53e10d6d99c2d68d", + "outputs": [ + { + "data": {}, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "