Updated 20 June 2019, CPAS backend version 0.3.2

 

Part 1: Mesh Generation

 

Algorithm and Implementation

ClusterTech Platform Atmospheric Simulation (CPAS) consists of an in-house implementation of several ideas published in [1].

Our implementation from scratch made modifications to the original algorithm to ensure conformance of target resolution specified by users, and to optimize computations in the hierarchical time-stepping (HTS) algorithm.

We are continuing our research and development on the generation of initial meshes (i.e. zero optimization iteration) and mesh optimization. Candidate techniques for initial meshes include the modified Frontal-Delaunay algorithm [1] and bisection techniques [2]. Options for mesh optimization include those mentioned in [1] and the Lloyd-based algorithm [2]. Users can expect significant changes in the mesh generation algorithm in future releases of online services. CPAS backend version 0.3.2 (current version) uses our modified Frontal-Delaunay algorithm for initial mesh generation, and Lloyd-based algorithm for mesh optimization.

 

Resolution Boost for Orography

We are currently using variance of subgrid-scale orography (VARSSO) data between latitude -60 to 60 degrees for orographic resolution boost, we plan to use the gradient of topography data for the entire globe.

For coastlines, we will provide increased resolution via the USGS land cover data to identify the regions with land-water contrast.

 

 

Part 2: Atmospheric Simulation with Hierarchical Time-stepping (coming soon)

The following features are implemented in CPAS to enable using MPAS-A (v6.3) on customized meshes with large resolution variable and Hierarchical Time-stepping (HTS):

 

Populating geographic data

This process is based on the init_atmosphere program in MPAS-A (v6.3). In order to handle large variation in cell spacing, the init_atmosphere program was modified to allow reading static data of various resolutions. This can speed up the initialization of static fields as well as ensure their validity for small cells.

 

Assignment of Time Steps

CPAS implemented an in-house algorithm for assigning time steps to the cells in the mesh which minimizes computational resources. This algorithm partitions the global mesh into partitions, each having a fixed time step level. Denoting the largest time step as dt_0, all time steps are assigned a value 2^{-n} dt_0 for some non-negative integer HTS level n. HTS levels are firstly determined by cell spacing, and go through an optimization process.

 

Dynamic Core

The dynamic core in CPAS with HTS is based on MPAS-A (v6.3) and uses the same set of prognostic and diagnostic variables. CPAS also uses the same vertical coordinates and vertical information transfer as MPAS-A. The time-integration scheme is mostly adopted from MPAS-A with an in-house implementation of using time steps assigned to the cell and handling the boundaries where the time step has a jump. Being the same as MPAS-A, the filtering of spurious higher-order waves by the horizontal diffusion term is essential for handling resolution transition.

 

Saving of Computation Resource by HTS

The optimization of computation resources with respect to MPAS-A (v6.3) depends largely on the customized mesh, in particular, the proportion of numbers of cells in different HTS levels.

CPAS calculates the theoretical resource usage by HTS and its saving from MPAS-A. The unit Giga-cell-step is used to quantify the resource usage, i.e. the computation for one mesh cell for one time integration step is one cell-step, and one Giga-cell-step is 10^9 cell-steps. The result is displayed with time step level breakdown in the mesh generation job report.

 

Physics Schemes

The physics schemes are based on MPAS-A (v6.3) with in-house modifications. The convection parameterization schemes, Kain-Fritsch (cu_kain_fritsch), Tiedtke (cu_tiedtke) and New Tiedtke (cu_ntiedtke) are turned off for cells with spacing smaller than a specified threshold.

 

Parallelization and Load Balancing

The parallelization is based on MPAS-A (v6.3) with in-house modifications for load-balancing. This allows the MPI processes to run efficiently in a cluster environment and minimizes total execution time.

Graph partition for MPI parallelization is done by METIS (version 5.1.0) [3].

 

Speedup

The speedup that can be achieved with respect to MPAS-A (v6.3) depends on: (1) the reduction through HTS, (2) CPAS’ modified parallelization and load-balancing function, as well as (3) the server and network hardware environment used to run the job. Some benchmarks will be published as a new White Paper when available.

 

 

Part 3: Accessing Result Data and Visualization (preliminary, coming soon)

The following functionality has been implemented, while much more visualization automation being planned.

 

Jupyter Notebooks

While the resulting file sizes of practical simulations are too large for downloads generally, CPAS will provide Jupyter Notebook Service for users to use Python code to analyze their data. This service is a customized hosting of Jupyter Notebook with security and computational resource restrictions. Each user should be able to access only his/her own mesh and real simulation data in a read-only manner. CPAS provides in-house and third-party Python packages for common result inspection and visualization tasks. User can perform customized analysis tasks, that extract data and post-process them for data reduction purposes. Resources used by the Jupyter Notebook including network bandwidth, network transfer size, storage, memory and CPU will be limited according to the subscription, terms and conditions and applicable usage contracts.

 

Full mesh visualization

CPAS provides an easy-to-use, semi-interactive, Jupyter Notebook style user interface for plotting the full mesh by PyNGL, and visualizing it via ipyleaflet.

 

Plotting Simulation Result

CPAS provides an easy-to-use, semi-interactive, Jupyter Notebook style user interface for plotting 2-dimensional horizontal data (of specified time slab) by PyNGL.

 

List of Python packages that can be used in the environment

The following packages are installed and can be used in the Python environment:

In-house:

  • cpas

Thirdparty:

  • numpy
  • xarray
  • PyNGL
  • ipywidgets
  • ipyleaflet

 

References

[1] Engwirda, D. (2017). JIGSAW-GEO (1.0): locally orthogonal staggered unstructured grid generation for general circulation modelling on the sphere , Geosci. Model Dev., 10, pp. 2117-2140, https://doi.org/10.5194/gmd-10-2117-2017

[2] Lecture slides “Generating meshes for MPAS” in Boulder 2018 MPAS tutorial http://www2.mmm.ucar.edu/projects/mpas/tutorial/Boulder2018/slides/12._mesh_generation.pdf

[3] METIS - Serial Graph Partitioning and Fill-reducing Matrix Ordering http://glaros.dtc.umn.edu/gkhome/metis/metis/overview