tJet Roars to Life
New ESRL supercomputer supports hurricane research, including experimental FIM forecasts
Visitors to the home of tJet, one of NOAA’s most powerful new supercomputers, have to yell to be heard over the roar in the room. tJet is running simulations of tropical cyclones, and it seems as if the computer is invoking the sound of the storms.
But ESRL’s Scott Nahman (Global Systems Division, GSD), points to the closet-sized air conditioning system that keeps the supercomputer in optimum conditions. The fans may be loud, but “this is the first high-efficiency system in the building,” Nahman said. The new unit could save NOAA several thousand dollars on electricity next year, he estimated.
That’s just one record among several for ESRL’s summertime supercomputing upgrades, including the Intel-based tJet supercomputer. Installed in early August and running hurricane forecast experiments by the end of that month, tJet is one of the two most powerful research supercomputers in NOAA.
With more than 10,000 new cores, tJet is faster and more efficient than its predecessor—the 3,500-core Nehalem Jet (nJet) installed in the summer of 2009 and already full, said ESRL’s Leslie Hart, the high-performance computing senior software engineer for NOAA’s Office of the Chief Information Officer.
“Science is a core vacuum,” Hart said. “We needed more resolution for this hurricane work.”
By September, ESRL researchers were using tJet’s parallel processors to make detailed comparisons of the hurricane forecasting prowess of several global forecast models, from the operational Global Forecast System (GFS) and European Center for Medium-Range Weather Forecasts (ECMWF) to ESRL’s experimental FIM (the Flow-following, finite-volume isocahedral model).
At issue is how closely those models forecast the real-world tracks of hurricanes and changes in the storms’ intensity, said Stan Benjamin, Chief of GSD’s Assimilation and Modeling branch. Efforts to improve global tropical storm forecasts, he said, are reaching in several directions, all enabled by tJet: Experiments with higher horizontal resolution; ensemble forecasting with multiple runs; experiments with new data-assimilation schemes; and inclusion of chemistry and aerosols.
ESRL researchers currently conduct a 15-km FIM run once per day out to 10-day projection, to examine the effect of higher-resolution for tropical cyclone forecasts. Last year, when ESRL conducted tests of an older version of FIM at 15-km resolution, it had to be done on an NSF-funded supercomputer in Texas with limited availability. tJet has also accelerated experiments to consider the effects of inline chemistry in FIM on tropical cyclone forecasts.
So far, FIM is performing well, Benjamin said, with track and intensity forecasts that are about as skillful as those generated by the operational GFS—FIM is sometimes better, sometimes worse, but with improved intensity forecasts in the 15-km FIM. The experimental FIM model performs very well at predicting high-level winds, he said, and perhaps most importantly, FIM represents an approach that’s independent of other global models. That makes it especially useful for forecasters who project many possible futures with ensembles of independent models. Ensembles let forecasters pin probabilities on predictions.
“FIM and GFS may be equal in overall skill, but FIM comes to different solutions,” Benjamin said. “So we’re very confident that the global ensemble will be better with the additional diversity of FIM.”
FIM is a strong candidate to be added as an experimental model to the global multi-model ensemble system at the National Centers for Environmental Prediction (NCEP) in 2012.
ESRL researchers are also using tJet to study the impact of various data assimilation techniques on hurricane forecasts. Assimilation schemes are used to incorporate meteorological observations into the initial conditions of forecast models.
Mike Fiorino (GSD) is evaluating the performance of two key assimilation schemes—the still-experimental Ensemble Kalman Filter (EnKF), developed by Jeff Whitaker, Tom Hamill, Phil Pegion, and others in ESRL’s Physical Sciences Division, and the operational NCEP Gridpoint Statistical Interpolation—when used with FIM. The EnKF is also run on ESRL’s tJet.
“We are getting signals that the EnKF improves Eastern Pacific track forecasts,” Fiorino said. “This is the kind of result that will capture the attention of forecasters…but it may mean that this season’s set of storms happen to be the sort the EnKF/model handles well.”
Fiorino said we will need many more experimental runs to make firm conclusions about the hurricane forecast performance of various models and assimilation schemes. But already, interesting patterns are emerging—such as the fact that FIM consistently did better with Eastern Pacific storms than those in the Western Pacific Ocean. Figuring out why may well help his colleagues improve the models and assimilation schemes.
It’s quite helpful to have in-house the computing power necessary for such research, Benjamin said. In the past, he and his colleagues had to port FIM code to other centers—such as the Texas Advanced Computing Center—and wait in queues for the processing time required to run a forecast at resolution as fine as 15-km scale.
“Now we can do more science and less queue management,” Benjamin said.
tJet also means that he and his colleagues can make quick variations to runs, depending on recent results, and to test high-resolution ensemble forecasts (with 30+ ensemble members) using FIM and GFS. These ESRL-tJet ensemble-based forecasts have had particularly high skill for tropical cyclone forecasts this fall.
tJet was funded through the 2009 Hurricane Forecast Improvement Project (HFIP), in which NOAA is working with academic and private-sector partners to improve scientific understanding and prediction of tropical storms, hurricane intensity, hurricane tracking, and the dangerous storm surges associated with tropical storms.
More on FIM, including output of daily runs: http://fim.noaa.gov/.