By Thomas LeFebvre and Tracy Hansen
FSL's role in the development of the Interactive Forecast Preparation System (IFPS) was featured in the December 2000 issue of the FSL Forum. Also included in these comprehensive articles was the Graphical Forecast Editor (referred to as the GFESuite), an important aspect of IFPS. Three years ago, only 15 National Weather Service (NWS) forecast offices were using this prototype software quasi-operationally to produce a small number of experimental products. Since then, GFESuite software has been installed at every NWS Weather Forecast Office (WFO), and forecasters are using it operationally around the clock to express the weather forecast digitally. This initial operating capability (IOC) represents a major milestone for the GFESuite software and the NWS.
Forecasters now interact with a suite of tools that manipulate gridded numerical representations of the forecast instead of spending most of their shift typing text products. With these useful products no longer needing to be typed by hand, forecasters can focus more on the forecast as the products are formatted automatically from digital grids that "feed" the National Digital Forecast Database (NDFD), soon to become the official representation of all WFO-generated forecasts. Much has
been accomplished to reach this IOC milestone.
GFESuite developers designed and implemented a new text
product formatter framework that is flexible enough
to accommodate local preferences ranging from
simple wording issues to major changes in the text
product format. The InterSite Coordination (ISC) facility
allows adjacent offices to compare their forecasts
graphically so that they can produce gridded forecasts that look
as seamless as possible when the data are mosaicked in
the NDFD. To complement implementation of the
software, GFESuite developers devoted a significant amount
of time preparing and presenting training material to
convey critical information to NWS forecasters.
The GFESuite comprises not only the Graphical Forecast Editor, but also a collection of software (Figure 1) that permits forecasters to define the weather forecast numerically or digitally. Forecasters use the GFE interactive editor to manipulate gridded representations of the forecast. The GFE houses dozens of tools that can modify the gridded forecast in meteorologically useful ways. Three separate editors provide different views of the forecast database (Figure 2). The Spatial
Editor presents an areal perspective of the grids so
that forecasters can edit the gridded data over a
particular area. The Temporal Editor displays a time series view
of the data over any selected area. This editor lets
the forecaster modify the grids for the selected area as
a function of time. The Grid Manager displays an
inventory view of the database where forecasters can
inject or replace grids based on numerical model output,
change the time period over which grids are valid, add new
grids by interpolating based on existing grids, and perform
a variety of other operations.
Figure 1. Diagram illustrating major components of the GFESuite software.
Figure 2. GFE editors include the Grid Manager (an inventory of weather elements), Spatial Editor (grids in plan view), and Temporal Editor (time series representation of weather elements).
The GFE also contains a programmable interface we call the Smart Tool framework, which is used by forecasters to implement their own tools to perform a virtually limitless number of operations. Smart Tools have access to a growing array of data sources including numerical model output, surface observations, and topography. The Smart Tool architecture is shown in Figure 3.
Figure 3. The Smart Tool architecture.
The other GFESuite programs are dedicated to generating forecast products or processing meteorological data that are used by some other component of the GFE.
The ifpInit facility ingests raw numerical model output
and calculates various sensible weather elements at
the earth's surface. These grids are sometimes used
by forecasters to initialize the gridded forecast,
particularly in the 5–7 day forecast period. The Daily
Forecast Critique (DFC) archives surface observations and
plots them in a time series format along with point
forecasts extracted from the grids. This program gives
forecasters the capability to compare their forecast with
observations, providing insight into systematic forecast errors.
Several GFESuite programs produce forecast products based on the forecast grids. The ifpnetCDF program creates a binary representation of the forecast. This format is useful for transferring the values of the forecast to users who require the highest level of forecast detail, such as fire behavior modelers. The ifpImage program produces pictures of the forecast in Portable Network Graphics (PNG) format. Many NWS Forecast Offices use this as part of their Website to convey forecast information to the general public. GFESuite text formatters sample the gridded data over some specified location and time period and create words that represent the forecast. This suite of text formatters provides a very important function to NWS forecasters. High quality, automatically generated text products are very important to the success of IFPS, since the time that forecasters previously spent typing text products can now be better spent improving the quality of the gridded forecast.
Several new features have been added to the GFESuite since our last report (December 2000 Forum). A new text product formatter infrastructure was designed and implemented so that local forecast offices have better control over the words derived from the forecast grids. We implemented the InterSite Coordination (ISC) facility so that forecasters can view grids from adjacent offices and more efficiently coordinate their forecasts. The Daily Forecast Critique, built in 2002, gives forecasters the opportunity to compare their forecasts to surface observations.
GFESuite Text Product Infrastructure
The task of automatically generating text products from the GFESuite forecast grids has been a major focus over the last year. We introduced an experimental prototype in 1998 at the Modernized Product Workshop as a way to create new innovative text products directly from forecast grids (Figure 4). The field forecasters quickly adopted this new approach, which uses the Python programming language, to make it extensible and easy to customize to the local site. They explored the use of
this technology to create new products as well as to work with the legacy text products. The popularity of
this approach prompted the NWS in August 2002 to ask
FSL, in concert with field forecasters, to write a set of
core products as an alternative to the existing products.
This was quite an undertaking, and to set it in motion,
we immediately formed a Focus Group of about 25
motivated forecasters who had been working with the text
Figure 4. Events that comprise the evolution of
the current text formatter infrastructure.
The National Weather Service issues directives describing the content and format of text products to be issued. The core products (about 12 in all) include the Zone Forecast Product, Coastal Waters Forecast, Fire Weather Forecast as well as tables such as the Coded Cities Forecast. The directives (Figure 5) needed to be encoded into the baseline software, but that's only half the story. These directives act only as guidelines, and each local office has its own variations and preferences based on the geographic and meteorological characteristics of its domain. We began a series of 14 releases of the software to the forecasters on our team. They tested the code and made sure it could handle their local customizations, and then gave us their "bug" reports and requests for enhancements. On the surface, the idea of representing numbers as simple phrases does not seem too difficult: "Sunny. Highs in the 80s." How hard could that be?
Figure 5. NWS Directives are the starting point for building a text product formatter. Code developed in the field that addresses details not mentioned in the directives are also folded into the baseline software.
However, we discovered that there are many subtleties and complexities that we did not foresee. For example, the probability of precipitation (PoP) phrase, "Chance of rain...," "Chance of snow..." depends on the current weather events. Or the snow accumulation we report in the "Today" period will contribute to the total snow amount reported in a later period. We could have "a chance of rain and fog in the morning" then "a chance of rain in the afternoon," and the formatters have to be smart enough to see that there is a chance of rain all day and fog only in the morning.
The temporal resolution of the grids can lead to more complexity with many different weather scenarios in a 12-hour period. The formatters have to be able to consolidate this digital information and produce concise yet accurate phrases. Beyond that, we needed to account for local effects and report differences between areas, such as the "coast" versus "inland," or "mountains" versus "valleys."
In order to tackle these complexities and dependencies, we invented a new way of processing, which we call "holographic." This means that the narrative components and phrases are represented in a tree-like structure in the computer memory (Figure 6). Then different dependency and logical rules can be applied to any part of the tree before producing the final product. The weather conditions can be examined to properly word the PoP phrase. The snow accumulation can be summed over consecutive periods to yield a total snow amount that is consistent and accurate. Phrases can be more detailed, new ones can be added, or the order of the phrases can be changed on the fly. Forecasters do not have
to commit to any part of the product until all
dependencies are accounted for and the whole is complete.
Figure 6. Text products are represented in a tree-like structure before the words are generated.
Once this framework was in place, we worked with our Focus Group to determine which rules to apply to the tree-like structure. It turns out that these are the rules forecasters have used almost intuitively over the years as they typed their text forecasts. The Rapid Prototype Process, working interactively with the users, is a very effective way of extracting this information. This approach was successful, and in June of 2003 we released the core products for national deployment.
This means that if you look up the official forecast for
a National Weather Service office, it is likely that
the words you read were generated by software written
at FSL. Similarly, if you listen to NOAA weather radio,
the words you hear are likely to be a result of this
system. And now this infrastructure, bolstered by our
experience with the core products, can be used to create new
and innovative products such as pointcasts,
travelers' forecasts, and products for other programs such
as Aviation and River Forecast products.
Under the previous text-based paradigm, forecasts from several offices were rarely, if ever, presented on the same display, thus offices seldom coordinated routine forecasts. However, with the NDFD presenting a national mosaic of digital forecasts from literally dozens of offices, it became clear that forecast offices must spend more effort coordinating their forecasts. To help with this coordination effort, FSL built the ISC facility that lets forecasters look at forecast grids from surrounding offices before they commit their official grids to the NDFD.
As the gridded forecast is being developed at a local office, the forecaster has the option of sending the grids to adjacent offices over the AWIPS Wide Area Network (WAN). A few minutes later these grids are received at the adjacent offices and remapped or transformed into a format so that they can be viewed properly by the adjacent sites. The forecasters only need to select a
single button on the GFE, and gridded data from
surrounding sites instantly appear on the display. Forecasters
can then very easily see any discrepancies between
their own forecast and that of their neighbors. Figure 7
shows a GFE display with data from surrounding sites interleaved.
Figure 7. InterSite Coordination software performs an automatic mosaic of the gridded data from surrounding sites so forecasters can compare their forecasts with neighboring offices.
The capability to look at the forecast from an adjacent site is not the only advantage in using the ISC. New tools have been developed that flag forecast data discrepancies beyond a configurable threshold. Grids that exceed these thresholds are color-coded in the GFE Grid Manager so forecasters can see at a glance which grids do not meet the criteria. If the discrepancies are relatively minor, tools built into the GFE can smooth them out. Major discrepancies generally require some additional communication between the two offices. Over time, forecasters have learned that coordinating the forecast before editing the gridded forecast usually leads to better results than waiting until the forecast is complete. Many forecast offices use Internet chat software to collaborate and reach consensus on the forecast before beginning the process of editing the grids. As a result, forecasts produced at these offices usually contain fewer spatial discrepancies than those produced at offices where forecasters do not collaborate.
Daily Forecast Critique
The Daily Forecast Critique (DFC) permits forecasters to compare their gridded forecasts to actual observations. Since observations are collected at selected points within their forecast area, point forecasts are extracted from the grids before the comparison takes place. Forecasters use the DFC to give them a sense of forecast accuracy and to learn about any systematic errors so that they may use that knowledge to make a better forecast next time.
The DFC consists of two main programs, an archiver and display program. Since the AWIPS software saves on average only 36 hours of surface observations, a DFC archive program was developed so that offices could save as many observations as they thought useful. The archiver is started about every 15 minutes, searches the AWIPS database for any new observations, extracts them, and archives them. In addition to saving observations, the archiver also saves forecast points extracted from the official database as well as any or all numerical model databases. The model database archive lets forecasters compare the numerical model-predicted values against observations, giving insight into systematic model errors.
The display program consists of a user interface window and a display window. With the user interface, forecasters can select from various categories such as the weather element, station, data source (observations, model, official forecast), and model time, followed by a command to plot the data. Once the data are plotted, a different data source or model time can be selected and that dataset can be combined with the previously plotted dataset. This type of interface allows users to plot any dataset against any other and have complete control over which datasets are compared. The DFC configuration files let forecasters control which stations and models are archived, as well as display preferences such as colors, plotting symbols, and line textures.
Though the GFESuite system was judged adequate to meet IOC requirements, we will continue to develop new tools and interact with operational forecasters to create a system that makes the task of weather forecasting more efficient. Expressing the forecast as a gridded field is a very new concept for forecasters. Therefore, the methodology, or the techniques employed to edit the grids, is generally quite primitive. Many forecasters use interactive tools that require a significant amount of forecaster input to accomplish a task. This practice is both time-consuming and can lead to grids that are internally inconsistent. GFESuite needs a more sophisticated level of tools that apply changes to the forecast in a meteorologically meaningful way while keeping the database internally consistent in time, space, and across weather elements. With an eye toward that goal, we are now working to inject more science in the forecast process.
The Smart Tool framework developed several years ago has worked reasonably well thus far, but needs new enhancements that will make writing new tools less cumbersome, and injecting good science into the forecast process easier. We are developing a new set of meteorological tools that will provide the fundamental building blocks for creating scientifically sound tools. These tools include methods to perform numerical derivatives on meteorological fields, a set of atmospheric physics methods that enable tools to more easily calculate important derived quantities, and utility methods that make the job of writing new Smart Tools easier. GFE currently offers an interpolation facility that calculates new grids based on grids that have been already defined. Interpolation allows forecasters to easily and quickly fill in temporal gaps in the forecast, saving precious time. Unfortunately, this interpolation program is purely mathematical and lacks all knowledge of meteorological concepts. A new interpolation facility is needed that knows how various weather elements interact with each other. To make the results more predictable, we plan to allow user input in the interpolation process, such as the position and strength of a cold front as a function of time, so that these interpolation algorithms can make better, more intelligent decisions when filling in the
temporal gaps. We are now designing a new interpolation
facility that provides for this user input and also
contains knowledge about how weather elements interact so
that interpolated grids are more consistent with the
Forecasters have demonstrated strong personal preferences when creating the words that represent the forecast. We expect that the text formatter infrastructure and the text formatters themselves will require refinement as traditional products evolve and new products are developed.
Despite our success in the IFPS project, we still have much work ahead. GFE can be used as an effective tool for making gridded digital forecasts, but many improvements are needed to make this forecasting process more efficient and more scientific. Refinements to the product formatters will free up time spent on making products and permit forecasters to better focus on the meteorology. Better science in the forecast process will likely lead to forecasts of unprecedented detail and accuracy.
Editor's Note: A complete list of references and more information on this and related topics are available at the main FSL Website www.fsl.noaa.gov, by clicking on "Publications" and "Research Articles."
(Tom LeFebvre is a meteorologist in the Modernization Division. He can be reached at Thomas.J.LeFebvre @noaa.gov, or 303-497-6582.)