Development of a Massively Parallel NOGAPS Forecast
Model

Background:  The Navy Operational Global Atmospheric Prediction System (NOGAPS) is the primary Numerical Weather Prediction (NWP) system providing global weather guidance for all branches of the DOD. The system has been run operationally by Fleet Numerical Meteorology and Oceanography Center (FNMOC) since 1982. All research and development of NOGAPS has been by the Marine Meteorology Division of the Naval Research Laboratory (NRL). The operational system has been ported to three generations of supercomputers, the latest being a 16 processor Cray C90. The heart of the system is a global spectral forecast model using spherical harmonics as horizontal basis functions and vertical finite differencing. Over its life, NOGAPS resolution has increased in parallel with the computational power of scientific supercomputers of the day, yielding increased forecast skill (Fig. 1) with each resolution improvement. Currently the operational NOGAPS forecast model runs with 0.75 deg horizontal resolution (spectral resolution=T159) and 24 levels in the vertical, from the Earth's surface to 50 km. The model runs operationally on 12 processors with a sustained performance of 5 Gflops. This corresponds to a wall-time performance of 20 minutes/forecast day, which is necessary to meet FNMOC operational schedules. The operational NOGAPS produces about 25,000 global meteorological grid fields each day, from twice daily forecasts to 144 hours. These products support virtually every DOD activity and application that depends on environmental information, from surface forcing for ocean circulation models to ballistic missile targeting inputs. It also provides time dependent boundary conditions to drive the Navy's operational mesoscale model, COAMPS, for 7 regional forecast areas.

Besides the operational requirements, NOGAPS supports a wide range of in-house NRL research projects, including atmospheric predictability studies, coupled atmosphere/ocean model development, and data assimilation with adaptive observations. Some examples of this research are available at http://www.nrlmry.navy.mil/projects/projects.html. NRL also supplies delayed operational NOGAPS products to non-DOD users with the NRL Master Environmental Library (MEL), also accessible on the above NRL website. In addition to actual gridded field products, weather maps of traditional NWP guidance are available on an FNMOC website: http://www.fnmoc.navy.mil/PUBLIC/WXMAP/GLOBAL. This site has images of a wide range of NOGAPS guidance products for areas all over the world, and is one of the busiest sites for weather information, receiving about 3,000,000 hits/day.

Project Goals:  The next step in NOGAPS evolution is porting to scalable architectures. The DOD Common HPC Software Support Initiative (CHSSI) is supporting the conversion effort by NRLMRY, in anticipation of the delivery of a scalable system to FNMOC in early FY2000. The new computer system will replace the FNMOC operational C90's sometime in early FY2001, when all large application codes must be converted from the parallel vector programming model of the C90 to distributed memory message-passing designs suitable for the new architecture.

The first goal of the development of a scalable NOGAPS code is for the FNMOC procurement of a scalable system, where the NOGAPS forecast model is the primary benchmark code. In anticipation of this, a primary design goal was portability and flexibility of the code to run on any of the candidate computer platforms. The scalable NOGAPS model is coded in strict adherence to Fortran 90, and communication is with message passing (MPI). This ensures graceful porting of the code from single processor workstations to massively parallel, distributed memory supercomputers. A second goal is ensuring that the scalable code is essentially identical to the operational parallel vector code in meteorological performance. Because of necessary algorithmic differences in a distributed memory design, bit reproducibility is not possible between the two codes, but every effort has been made to minimize differences. However, bit reproducibility of scalable NOGAPS results for different processor numbers is an important goal.

Accomplishments and Results:  A prototype scalable NOGAPS code was completed in early FY98. The code was tested successfully on a Cray T3E, DEC alpha 8400 SMP, SGI Origin 2000, and a Cray C90. On scalable architectures most of the inter-processor communication in the NOGAPS code is in the spherical harmonic transforms (Fig. 2), in the form of matrix transposes. Good scalability of the transform code was critical for overall model performance (Fig. 3). The transforms are the heart of the dynamical core of the NOGAPS model, where the equations of motion for meteorological flow on a sphere are solved. Most of the computational code of the model, however, is in the diabatic processes, such as radiation and cumulus convection. These processes are basically embarrassingly parallel, with independent columns of grid points spread across the processors. However, load imbalances are inevitable because of the inhomogeneous distribution of the processes on the Earth's surface. Cumulus convection is the most conspicuous example, with most of the workload in the tropics, and also during daylight hours. Special interleaving of model latitudes and longitudes among the processors is necessary to minimize cumulus load imbalances (Fig. 4).

In early FY99 the prototype scalable NOGAPS code was given to FNMOC for their procurement. Special scripts and data sets were prepared for prospective vendors to use in the competition. A special FNMOC web site was established for communication with vendors and handle their questions. Few vendor questions or complaints about the NOGAPS code were received, however, indicating few problems with the code or accompanying information. Four vendors competed: Sun, SGI, IBM, and Compaq/DEC. All successfully ran the code on up to 120 processors, and based on benchmark results, SGI has been awarded the contract. Initial delivery will be a 128 processor Origin 2000 in early FY 2000, with upgrades in processor number and technology over the next 2 years.

With the delivery of the initial system, optimizing the scalable NOGAPS code for the Origin 2000 will begin. Target resolutions will be tested, based on the 20 minute/forecast day criteria describe above. Communication design and cache optimization will be top priorities. With the expected technology upgrades, the goal is 100 Gflop sustained performance by FY2001.

Software:  The NOGAPS model source code is not proprietary, however, the model must be run as part of the complete NOGAPS NWP environment. The complexity of the system, and its meteorological data base requirements, make it impractical to distribute it freely to outside users. For in-house NRLMRY users special run scripts and source code are maintained under CVS control, and expert consultants are available. NRLMRY is not staffed to provide this consultant support to outside users, however, so we rarely release the code except to groups working on joint projects. During the development of the scalable NOGAPS code a group at the UCSD supercomputer center was provided the model source code with a simple input data set suitable for code development, but not for meteorological research. We do distribute non-real time NOGAPS products to other research groups on a case by case basis.

Publications:

Rosmond, T.E, 1998: A Scalable Version of the Navy Operational Global Atmospheric Prediction System Spectral Forecast Model. Workshop on Software Engineering and Code Design in Parallel Meteorological and Oceanographic Applications, June 15-18, 1998, Scottsdale, AZ.

Rosmond, T.E., 1998: Investigation of the Scalability of Pre-Processing, Post-Processing, and Computational Modules of the NOGAPS Spectral Forecast Model. Toward Teracomputing: Eighth ECMWF Workshop on Parallel Applications in Meteorology, 16-20 November, Reading, England.

Rosmond, T.E., 1999: A Scalable Version of the Navy Operational Global Atmospheric Prediction System Spectral Forecast Model. To appear in Journal of Scientific Programming.