THE USE OF HISTORICAL, CURRENT, AND FORECAST WEATHER AND CLIMATE DATA IN A GIS

The availability of historical, current and forecast weather data that is easily importable into a GIS is presented. Point data such as weather parameters observed at, or forecast for, specific locations, including temperature and precipitation, are discussed. The availability of historical and climatic data in similar formats is presented. Non-traditional data such as weather warnings and hurricane tracks are demonstrated.

INTRODUCTION

There is a tremendous amount of weather data available today. A host of new observing technologies have led to a large increase in the amount of weather data available in near real-time to users of GIS systems. Much of this data is also available in a historical record. These same improvements in weather observation technology, combined with increases in computing performance, have led to dramatic improvements in weather forecasts.

Unfortunately, much of this weather data has only been available in a variety of arcane formats, or has required extensive processing, or very expensive receiving equipment. Furthermore, very little weather data was georegistered in a standard way that permitted easy and quick use by non-meteorologists. DTN Weather Services has now developed software and systems that make weather data available to Esri applications in standard shape and arcgrid formats. This makes many types of weather information available to any Esri user.

DTN Weather Services collects and processes virtually every imaginable type of weather information, including radar, satellite, lightning, surface and upper air observations, gridded and text forecasts, etc. Data is collected from every part of the globe, processed, checked and quality controlled, and then distributed to users, or archived. While much of this data is available in its raw formats from other sources, DTN Weather Services has added significant value to the data by its collection, processing, and quality control. Combined with the convenience of a single source for such weather information, and the conversion of this data into Esri formats, this data service is second to none for convenience and reliability.

Meteorologists traditionally divide weather data into two categories: diagnostic (observed) data and prognostic (forecast) data. This paper divides weather data into three categories: historical, current, and forecast data. These categories seem to better reflect the use of weather data in GIS environments. This paper examines various types of weather data in all three categories, with the emphasis being on that data which delivers the most information, while requiring the least amount of interpretation, or secondary processing.

CURRENT WEATHER DATA

Current weather data includes real-time observations of clouds, precipitation, or other weather phenomena, and short-term forecasts (or nowcasts) of that information. This weather data is familiar to most readers as that which is typically displayed on the local weather segment of the evening news. Much of this data is available globally, as well as locally.

Satellite

One of the best types of weather data to view large areas of the globe with is satellite cloud data. This data is available from geostationary weather satellites, and provides 24 hour per day coverage of the earth. Raw satellite data is not georegistered, has significant volume, and requires expensive receive equipment. DTN Weather Services collects raw data from satellites around the world. This data is then processed into a standard format called GCAB (Ground Corrected Altitude Based) remapped satellite data. The GCAB process uses a dynamic three-dimensional atmospheric model to eliminate non-cloud features from infrared (IR) satellite data.

The result is a georegistered earth coordinate grid (4km resolution) of cloud information (automatically converted into a shape file), where the cloud values are directly proportional to the altitude of the clouds. All non-cloud features have been eliminated. It is available every 30 minutes in North America and Europe, and hourly elsewhere. Incidentally, this data is the basis of the fly- through satellite imagery seen on television.

Radar

Over the United States, radar data is available in near real-time to depict areas of precipitation, severe weather, and other weather phenomena. DTN Weather Services maintains dedicated 14.4Kb connections to 150 National Weather Service (NWS) WSR88D Doppler weather radar's in the United States. Each WSR88D radar scans a 460km cylinder of air above it every 5 minutes. Base data products, such as reflectivity (the precipitation echoes) and radial velocity (the speed of those echoes toward or away from the radar) are generated at a variety of elevation angles. When the radar has completed scanning the entire volume of the cylinder around it, other derived products are generated. These include storm attributes and tracking information, precipitation estimates, and other products.

Obviously, this represents a very large volume of information, coming in in real-time. There are other shortcomings of radar data from a single radar. Some of the echoes may not be caused by actual precipitation. Ground clutter in and around the radar itself may mask actual precipitation. Because of the curvature of the earth, echoes at longer ranges may be misrepresented, or missed altogether. Because of these shortcomings, DTN Weather Services developed a composite, or mosaic, of all single site radar's, to provide a better product.

Composite radar is generated by taking a snapshot of all the single site radar's every five minutes. At each radar site, data from two different elevation angles is compared to discriminate between valid precipitation and non-precipitating echoes. The remaining data is then mapped onto a 1km earth coordinate grid. Where data from two radar's overlaps, data from the stronger echo (worst case) is used. Next, the radar data is compared to the GCAB satellite data, and all echoes that do not have corresponding clouds are eliminated. Finally, this data is compared to the DTN Weather Services database, and the echoes are categorized by precipitation type: rain, snow, or a mix of the two.

The result is a 1km grid of actual precipitation that is reaching the ground across the continental US (CONUS), on the same grid as the satellite data grid described above. Data is available every five minutes, and automatically converted into shape files.

Precipitation Estimates

One of the products generated by each WSR88D radar is a precipitation estimate. The radar uses a sophisticated algorithm to correlate the strength of reflectivity echoes and the near storm environment to estimate the relationship between reflectivity and rainfall rate (with some help from the NWS radar operator). This algorithm then accumulates the rainfall over the entire area scanned by the radar with each volume scan, and once per hour produces an estimate of the rainfall.

DTN Weather Services takes this digital rainfall product from each radar, and maps it into the same 4km CONUS grid used by the satellite and radar products described above. Once per hour, the estimated rainfall for the past hour is available as a shape file. This hourly data is also accumulated so that a running three-hour total and a 24-hour total rainfall are also available as shape files, every hour. Once per day the daily data is accumulated to produce shape files of estimated precipitation for the CONUS for the past seven days, 30 days, and year to date.

Severe Weather Warnings

Severe weather watches and warnings are actually short fused forecasts (nowcasts) of threatening conditions. The NWS issues severe weather watches when conditions are ripe for the development of severe weather. When actual severe weather develops, the NWS issues warnings for specific counties. With hurricanes or winter storms, the areas covered by these warnings could extend to areas the size of states. There are a variety of specialized warnings available from DTN Weather Services, listed in Table 1.

Each watch or warning issued by the NWS is received at DTN and parsed, decoded, and validated. The watches and warnings are checked for format, duplication, and all relevant information, including issue and expiration times, the nature of the warning, and the FIPS codes of the counties affected are placed into a table for conversion into a shape file.

With the advent of new radar technology in the 1990's, NWS radar's now automatically identify and track individual storm cells. This software can actually detect the presence of tornado's, hail, strong winds, and the mesocyclones that spawn this weather. The radar builds a table of storm cells, with each cell ranked by whether it contains a tornado, hail, or other severe weather. This table also contains each cells location, speed and direction of movement. This information is available with each volume scan from each NWS radar, and is often used by NWS personnel to issue severe weather warnings. DTN Weather Services provides this data in real-time, automatically converted into a shape file for easy use in any GIS application.

Hourly Weather Observations

Weather observations are taken for aviation and weather forecasting purposes everywhere in the world. These observations include data about current temperature, humidity, winds, weather, and precipitation at specific locations (points) all over the world. These observations are available over the internet, but only as collections of raw data in their native code forms. Currently, there are about 7500 weather observation locations around the world, with about 2500 in North America.

Weather data at the surface of the earth is available in two forms. Aviation observations are taken on an hourly basis, and collected by the NWS in a special code form called METAR:

KORD 311756Z 33010KT 10SM -RA SCT027 BKN042 OVC140 20/18 A3002 RMK AO2 RAB1658 SLP162 P0001 60002 T02000183 10244 20183 50010=

Outside of North America, routine weather observations are taken on a three or six hourly basis, and exchanged among countries under the auspices of the World Meteorological Organization (WMO). This data is also available from the NWS, but in a different code form, called SYNOP:

AAXX 31184 72530 11566 83310 10200 20183 39921 40162 50010 69951 761// 91756 333 10244 20183 555 93118=

Both of these reports are from the same time, for the same location. Obviously, there is a considerable amount of work to do to parse these reports and extract the available weather elements. This work is exacerbated by common coding or transmission errors, which can make dramatic changes in any individual weather parameter.

DTN Weather Services receives, parses, and decodes weather observations as they come in from the NWS. All data is strictly and exhaustively quality controlled to detect, and correct if possible, bad data. These checks include reasonableness checks (e.g. making sure that the dewpoint is not higher than the temperature), temporal checks (making sure that conditions have not changed unreasonably from last observations), and spatial checks (comparing this observation to those around it). Finally, quantities not observed or reported in the observation, but derived from it, are calculated. For example, relative humidity is not an observed parameter, but is calculated and added to the data record.

The end result is a table of surface weather data, available either in a comma delimited ASCII text file, or converted into a point shape file, and including:

OBSERVED WEATHER CODES:

1	Fair	14	Windy
2	Partly Cloudy	15	Rain-snow mix
3	Cloudy	16	Blizzard
4	Dust	17	Blowing Snow
5	Mostly Sunny	18	Rain
6	Fog	19	Snow
7	Very hot and humid	20	Thunderstorms
8	Haze	21	Sunny
9	Very Cold	22	Clear
10	Snow showers	23	Rain showers
11	Smoke	24	Sleet (ice pellets)
12	Drizzle	25	Freezing rain
13	Flurries	26	Freezing drizzle

Daily Weather Data

Not all weather phenomena occur at the time of weather observations. The maximum and minimum daily temperatures can occur at any time during a 24-hour period. Precipitation is accumulated and most often reported only once per day. Because of this, DTN Weather Services maintains a separate set of daily weather information, providing a concise set of daily weather parameters for the same set of locations as the hourly weather data. For many (but not all) stations this includes 30 year normal's, as well as the actual observed values.

This daily data is also available from DTN Weather Services as comma delimited ASCII text files, or as point shape files.

HISTORICAL WEATHER DATA

Historical weather data is all of the current weather data described above, after it is a day to years old. DTN Weather Services does not maintain archives of all weather data because of the enormous volume of that data. However selected types of data are archived, and available by subscription.

Radar/Satellite/Rainfall Estimate

The radar, satellite, and rainfall estimate products as described in current weather data for the United States are archived at DTN Weather Services. This data is available from 1999 to the present. The data is actually stored on tape in its native master grid format. This information has to be manually pulled from tape, converted into shape files, and distributed, which is an expensive process. If there is market interest or demand, this process could be automated, considerably reducing the cost of such data.

Hourly Weather Observations

Hourly weather observations are available for all 7500 locations in the world for all of the year 2000. Hourly data for all 2500 locations in North America is available for all of 1999 as well. All of this data is available in comma delimited ASCII text files, and is in the same format described in Table 1. A separate table with reporting location (geo-reference) information is also available.

Additionally, a 30-year record of hourly data is available for 106 locations in the United States, dating from 1969 to present. These locations represent large urban areas, and are also spaced geographically to provide broad areal representation. The format of this data is as described in Table 1.

Daily Weather Data

Daily weather data (as described above) including maximum and minimum temperature and daily precipitation, is available for all 7500 locations in the world from 1999 to present. All of this data is available in comma delimited ASCII text files, and is in the same format as described in Table 2.

FORECAST WEATHER DATA

Knowing what the weather will be is of equal or greater interest than knowing what it has been. Weather forecasting has improved markedly in the past 20 years, driven primarily by increases in computational power, and better observations. Forecast accuracy is very good out to two or three days into the future, and forecasts out to seven to ten days are show significant skill. Even forecasts of one, three, or 12 months, while less specific in nature, have shown remarkable improvement in the past few years.

Forecast weather data is available in two fundamentally different formats, point data, and gridded data. Forecasts of point data are essentially forecasts of the elements of surface weather observations at some future time. The parameters are very similar. These point forecasts are generated from Numerical Weather Prediction (NWP) models. The models start with current observed conditions, analyzed in three dimensions very much like a weather map, and then use mathematical equations to forecast what those parameters will look like at some time in the future. The basic output of an NWP model is a series of grids. This gridded data is available in varying resolutions, and for varying domains.

There are three types of NWP models available from the NWS. Global models cover the entire world. They are used to forecast the interrelations of all of the earth's weather, and provide forecast's out 10 to 15 days into the future, at relatively course resolutions. Regional Models have domains of continent sized (10,000km) areas. They have finer resolution than global models, but only provide forecasts out three to five days. Mesoscale models provide very high- resolution forecasts for very small (<1,000km) areas, usually for very short time periods. The are used to forecast the evolution of very small-scale phenomena, such as thunderstorms. All three types of models are available from the NWS, as depicted in Table 3.

Gridded Forecast Data

NWP models represent the atmosphere in three dimensions as a series of grids. Each grid represents one weather parameter (say temperature) at one atmospheric level (from the surface to above 15,000m in altitude) at one point in time. The currently available parameters include those elements listed in Table 4.

Gridded data is packed in a special WMO code form called GRIB (GRIdded Binary). This code compresses and packs the data, but also makes it unreadable by anything other than an unpacking program. Furthermore, the large number of grids and grid parameters make data management a significant issue.

DTN Weather Services has been using GRIB data for many years in a variety of mapping applications. DTN collects NWP data from the NWS as it becomes available. DTN manages the volume of data by packaging together NWP data by model and forecast period, and delivering it to customers. This GRIB data can be received on a DTN Metwork Fileserver, which will automatically unpack the GRIB data, store it in its own directory, and convert it into the Esri ARCgrid file format, so that it is ready for use by Esri applications like Spatial Analyst.

This means that weather forecast information is available for any part of the globe, for a variety of parameters and a variety of forecast periods, in native Esri format readable by Spatial Analyst without any further processing. This makes it easy and affordable to use gridded forecast data in an operational setting.

Point Forecast Data

NWP models produce forecasts of various basic weather parameters on an evenly spaced grid, at various levels in the atmosphere. These forecasts only contain certain basic parameters (See Table 4), and may not provide the specific weather parameters desired. Also, the actual conditions at specific points on the ground may be significantly affected by local effects, such as terrain and proximity to bodies of water which are features that are smaller than the grid resolution of the model. For these reasons, interpolating from the raw grid data may not provide an accurate forecast at a specific point.

The model grid data can be adjusted statistically to overcome these deficiencies. Statistical techniques can also provide information derived from the grid data, such as probabilities. These statistical forecasts, called MOS (Model Output Statistics), are valid for specific locations, many of which are identical to the hourly weather observation locations. Over 1000 locations are available with point forecast information.

This point-forecast information is contained in a table that contains the output from various NWP forecast models, and from NWS manual forecasts. This forecast data is converted by DTN Weather Services into point shape files, and is displayable just as with the hourly weather observation data.

CONCLUSION

There is a tremendous amount of weather information available today. However most of the information available via the Internet is of limited value to the non-meteorological user of GIS equipment because the data is not GIS ready. DTN Weather Services has developed software and systems to make all of this weather information available with little or no additional effort to the GIS user. DTN Weather Services also provides information like radar, satellite, and rainfall composites that are not available from any other source.

This combination of a broad offering of weather data in a system that will automatically convert this information into GIS ready formats transforms marginally useful data into valuable information. It allows the GIS user to focus on their particular problem, and apply their specific knowledge to weather information without having to become a meteorologist, a weather data detective, a communications specialist, or some combination of the three.

Much more information on available data, systems, costs, and distribution methods can be found at DTN Weather Services (http://DTNWeather.com/GIS).