Updated 25 Apr 2020, CPAS backend version 0.7.2

 

Part 1: Mesh Generation

 

Algorithm and Implementation

ClusterTech Platform Atmospheric Simulation (CPAS) consists of an in-house implementation of several ideas published in [1].

Our implementation from scratch made modifications to the original algorithm to ensure high mesh quality with smooth transition of resolution by Customized Unstructured Mesh Generation (CUMG), and to optimize computations in the Hierarchical Time-Stepping (HTS) algorithm.

The resulting resolutions of the mesh will not exactly match those specified for each region. CPAS mesh generation trades-off resolution conformance for 100% well-staggering, which is more important. The bisection-based algorithm has a discrete nature (bisect or not bisect for the target resolution) and together with properties of spherical tessellation, an out-coming result of ±50% variation around the target resolution is considered as normal behavior of the algorithm.

 

Resolution Boost for Orography

For boosting resolution for orography, we pre-computed the topographic gradient based on the gridded topography data for the entire globe.

For coastlines, we provide increased resolution via the USGS land cover data to identify the regions with land-water contrast.

 

Part 2: Atmospheric Simulation with Hierarchical Time-Stepping 

The following features are implemented in CPAS to enable using MPAS-A (v6.3) on customized meshes with large resolution variable and Hierarchical Time-Stepping (HTS):

 

Populating geographic data

This process is based on the init_atmosphere program in MPAS-A (v6.3). In order to handle large variation in cell spacing, the init_atmosphere program was modified to allow reading static data of various resolutions. This can speed up the initialization of static fields as well as ensure their validity for small cells.

 

Assignment of Time Steps

CPAS implemented an in-house algorithm for assigning time steps to the cells in the mesh which minimizes computational resources. This algorithm partitions the global mesh into partitions, each having a fixed time step level. Denoting the largest time step as dt_0, all time steps are assigned a value 2^{-n} dt_0 for some non-negative integer HTS level n. HTS levels are firstly determined by cell spacing, and go through an optimization process.

 

Dynamic Core

The dynamic core in CPAS with HTS is based on MPAS-A (v6.3) and uses the same set of prognostic and diagnostic variables. CPAS also uses the same vertical coordinates and vertical information transfer as MPAS-A. The time-integration scheme is mostly adopted from MPAS-A with an in-house implementation of using time steps assigned to the cell and handling the boundaries where the time step has a jump. Being the same as MPAS-A, the filtering of spurious higher-order waves by the horizontal diffusion term is essential for handling resolution transition.

 

Saving of Computation Resource by HTS

The optimization of computation resources with respect to MPAS-A (v6.3) depends largely on the customized mesh, in particular, the proportion of numbers of cells in different HTS levels.

CPAS calculates the theoretical resource usage by HTS and its saving from MPAS-A. The unit Giga-cell-step is used to quantify the resource usage, i.e. the computation for one mesh cell for one time integration step is one cell-step, and one Giga-cell-step is 10^9 cell-steps. The result is displayed with time step level breakdown in the mesh generation job report.

 

Physics Schemes

The physics schemes are based on MPAS-A (v6.3) with in-house modifications. The convection parameterization schemes, Kain-Fritsch (cu_kain_fritsch), Tiedtke (cu_tiedtke) and New Tiedtke (cu_ntiedtke) are turned off for cells with spacing smaller than a specified threshold.

 

Parallelization and Load Balancing

The parallelization is based on MPAS-A (v6.3) with in-house modifications for load-balancing. This allows the MPI processes to run efficiently in a cluster environment and minimizes total execution time.

Graph partition for MPI parallelization is done by METIS (version 5.1.0) [2].

 

Speedup

The speedup that can be achieved with respect to MPAS-A (v6.3) depends on: (1) the reduction through HTS, (2) CPAS’ modified parallelization and load-balancing function, as well as (3) the server and network hardware environment used to run the job. Some benchmarks were published in White Paper 2.

 

Part 3: Accessing Result Data and Visualization 

The following functionality has been implemented, while much more visualization automation being planned.

 

Jupyter Notebooks

While the resulting file sizes of practical simulations are too large for downloads generally, CPAS provides Jupyter Notebook Service for users to use Python code to analyze their data. This service is a customized hosting of Jupyter Notebook with security and computational resource restrictions. Each user should be able to access only his/her own mesh and real simulation data in a read-only manner. CPAS provides in-house and third-party Python packages for common result inspection and visualization tasks. User can perform customized analysis tasks, that extract data and post-process them for data reduction purposes. Resources used by the Jupyter Notebook including network bandwidth, network transfer size, storage, memory and CPU will be limited according to the subscription, terms and conditions and applicable usage contracts.

 

Full mesh visualization

CPAS provides an easy-to-use, semi-interactive, Jupyter Notebook style user interface for plotting the full mesh by PyNGL, and visualizing it via ipyleaflet.

 

Plotting Simulation Result

CPAS provides an easy-to-use, semi-interactive, Jupyter Notebook style user interface for plotting 2-dimensional horizontal data (of specified time slab) by PyNGL, and visualizing it via ipyleaflet.

 

List of Python packages that can be used in the environment

The following packages are installed and can be used in the Python environment:

In-house:

  • cpas

Thirdparty:

  • netcdf4
  • numpy
  • xarray
  • PyNGL
  • ipywidgets
  • ipyleaflet
  • matplotlib
  • geopy
  • geopandas
  • cartopy
  • seaborn
  • scikit-learn
  • pydot

 

References

[1] Walko, R. L., and R. Avissar (2011): A direct method for constructing refined regions in unstructured conforming triangular-hexagonal computational grids: Application to OLAM. Mon. Wea. Rev., 139, 3923-3937.

[2] METIS - Serial Graph Partitioning and Fill-reducing Matrix Ordering http://glaros.dtc.umn.edu/gkhome/metis/metis/overview