Web Soil Data Viewer
NRCS Implements an N-tier Enterprise Class GIS Web Application for Internet Access to the USDA Soil Data Warehouse.
Web Soil Data Viewer
NRCS Implements an N-tier Enterprise Class GIS Web Application for Internet Access to the USDA Soil Data Warehouse.
Steven L. Ekblad, Eric J. Strand, and Terry L. Aho
Esri International User Conference July 9-13, 200
Web Soil Data Viewer: Implementing an N-tier enterprise class GIS web application for Internet access to the USDA-NRCS Soil Data Warehouse.
Steven L. Ekblad, NRCS Development Team Leader, Fort Collins, Colorado
Eric J. Strand, Vice President, Synergetics Incorporated, Fort Collins, Colorado
Terry L. Aho, Business Area Specialist/Soil Scientist, Fort Collins, Colorado
The Web Soil Data Viewer is a proof of concept Internet application showcasing the use of current hardware and software technologies to produce custom soils maps from real-time interpretation of soils data attributes. Complete USDA - NRCS SSURGO Soils data, both spatial and tabular, were integrated into a natural resource data warehouse. The Web Soil Data Viewer utilizes Windows 2000 Advanced Server, Microsoft SQL Server, and Microsoft Internet Information Server implemented on a Compaq ProLiant four server rack system with 1.2 Terabyte SAN Storage Works disk storage. Esri Spatial Data Engine and Esri ArcIMS provided geo-spatial data management and GIS processing.
Software architecture includes four tiers: 1) User presentation implemented with DHTML, XML, and ArcIMS ActiveX controls. 2) Middleware software components implemented with Internet Information Server and ArcIMS connectors. Custom middleware to provide data access to a business rules database, relational soils data, and queries connecting SDE spatial data to non-SDE soils data. 3) Navigation and map imaging services provided from an eight-processor application server. 4) Data stored and accessed in three separate SQL Server databases: SDE SSURGO spatial database, National Soils Information System Soil Survey Distribution database, and business rules database. The business rules database embodies information for aggregating, processing, and querying the complex relational soils attribute database. This business rules database provides software system agility for changing user interface and business level rules to meet future business needs.
Data Rich and Information Poor
USDA - Natural Resource Conservation Service (NRCS) is the federal agency responsible for mapping soils and delivering soil information to the public. Since its inception in the mid-1930s, the NRCS has worked in partnership with other government agencies, as well as agricultural and community conservation groups, to provide management assistance for America's private land. A majority of the technical support the NRCS provides helps farmers and ranchers develop agricultural and conservation systems tailored to their individual needs.
In 1995, a Blue Ribbon panel of university and private industry specialists conducted a study and found that the NRCS had a huge storehouse of natural resource information from a variety of sources (USDA-NRCS, November 1995). However, the public did not have electronic access to that information, because the NRCS, like many other government agencies, relied on paper publications. In addition, much of the resource information was geographically dispersed. As a result, locating requested data was a lengthy process that could take days, even weeks and often produced less than desirable results. The panel provided recommendations to NRCS to remedy this situation, including the following software lifecycle development recommendations (USDA-NRCS, November 1995):
Soil Data Viewer Development History
USDA - NRCS is responsible for soil survey activities on non-federal lands. Soil surveys are carried out cooperatively with state agricultural experiment stations and other agencies. Completed soil surveys are digitized and the soil information delivered to the public in a Soil Survey Geographic Database (SSURGO). Over the last 15 years numerous technology distribution efforts have been designed and implemented to provide soils data to the public.
1997-1998: The seminal project for the Soil Data Viewer was the Soils Explorer project. The concept of this product is outlined in Future Directions: A vision of Automation for Field Conservationists. The business concept is that soils information would be packaged with a royalty-free viewer provided by NRCS to provide information electronically. This was a pilot project addressing the business needs to furnish tabular and spatial soils data to professional conservationists, natural resource land managers, and agriculture consultants. The development environment was Sybase Power Builder, Esri MapObjects Lite, targeted for distribution on an Intel Windows PC. Problems with this approach included the inability to field all digital ortho imagery with a complete soil survey due to CD storage capability. This software application is still seen as needed by the Soil Division. (For more information refer to the web page at http://www.statlab.iastate.edu/soils/nssc/explorer/soilex.htm.)
1998-2000: The second-generation effort at providing spatial and tabular soils information was the Soil Data Viewer (SDV). SDV was developed in Microsoft Visual Studio 6.0 using Visual Basic with minor use of Visual C++. The spatial data was enabled using ArcView 3.2 and the Soil Survey tabular data is stored in Microsoft Access. The Soil Data Viewer embodies business rules, interface and option information, and pre-design SQL searches in a Microsoft Access database that is accessed at runtime to configure the Graphical User Interface and to provide business logic for generation of soil interpretive maps. (For more information refer to the web page at http://www.itc.nrcs.usda.gov/soildataviewer/about.htm.)
2000-2001: The third generation of the Soil Data Viewer has been implemented in the Lighthouse Project. To bring the agency into the 21st century, the NRCS evaluated their total information technology architecture and identified the need to create a Web-enabled, centralized repository of geo-spatial and tabular soils data.
In January 2000, the NRCS began a joint effort with Compaq and Microsoft to design and implement an integrated system that would offer easy online access to geographic information systems (GIS) and geo-spatial information. The goal of the pilot program - called the "Lighthouse Project" - was to integrate three prototype applications using emerging technologies, in order to create a more seamless and efficient system that would involve sharing of common navigation services and images. If successful, this one-stop service would eliminate paper-based maps and information, while supporting the data research needs of 3,000 field offices, 13,000 employees, and thousands of public customers who rely on the USDA for technical assistance and conservation planning information.
To achieve this integrated vision, the NRCS would need to build large data sets using large-scale, Web-based applications on equipment that could eventually scale up to enterprise level. Considering the fact that building maps and navigating in real time over the Web had not been done before to this scale and in this type of environment, it was a monumental task. With assistance from Microsoft and Compaq Solutions Provider an initial technical architecture was identified and described (Report on Architecture Workshop, NRCS Natural Resource Data Gateway Project). Major considerations in the analysis and design of the architecture were data and application scale out capabilities.
Web Soil Data Viewer Prototype
The Web Soil Data Viewer is a prototype Internet application that provides online access to soils information while taking advantage of soil surveys that have been digitized. Access the Web Soil Data Viewer at:
http://lighthouse.nrcs.usda.gov/lighthouse/websdv/Navigator.html
This web-based tool makes it easy for USDA employees to produce maps that display soils information, combine soils themes with other geographic information, and query soil databases for answers to customer questions. The Web Soil Data Viewer helps answer questions such as how the soils located in a specific site should be used to conserve the resource and promote water quality (Figure 1 - Web Soil Data Viewer).
Figure 1 - Web Soil Data Viewer
The Web Soil Data Viewer prototype was built to test the usefulness of providing a single geospatial soil database for creating user specified soil interpretation and soil property maps. For each interpretation or soil property choice, the user is presented with and can select the valid methods for processing the component data for each mapunit (Figure 2 - User Selected Options). The dominant soil choice processes the soil attribute data for each spatial soil polygon and classes the soil map using the dominant soil for each mapunit. Dominant soil is on a mapunit basis not a field basis. So if a farm field has five different soil mapunits, each mapunit is processed and classed based on the dominant soil for that mapunit.
Figure 2 - User Selected Options
The Web Soil Data Viewer generates tabular reports and maps derived for the selected soil interpretation based on the user-entered parameters. The following output is a customized soil interpretation tabular report generated using XML.
Figure 3 - Soil Interpretation Report
The following is a customized soil interpretation map generated from the tabular and spatial soils data.
Figure 4 - Soil Interpretation Map
NRCS is in the process of updating the Soil Survey Geographic Database (SSURGO) with a new attribute data structure. Implementation of the new SSURGO data structure began as of April 2001. As existing SSURGO data sets are created or refreshed with the new attribute data structure they will become available through the Web Soil Data Viewer site. It is likely to take as much as a year or more to update all the existing SSURGO data sets with the new attribute data structure.
The current Web Soil Data Viewer site is a prototype version built to discover hardware and software requirements for delivering soil data over the Internet. The spatial database currently contains 936 soil survey areas. The related tabular database derived from NASIS downloaded data contains 391 test soil survey areas. NRCS State Soil Scientists have authorized 17 soil survey areas for release to public viewing with the prototype web Soil Data Viewer application. Additional soil survey areas can be added to the database in a matter of hours from receipt of an approved data set.
Table 1 - Soils Data Authorized for Viewing
|
State |
County |
|
California |
Sacramento |
|
Florida |
Glades, |
|
Georgia |
Hall |
|
Idaho |
Latah |
|
Indiana |
Scott |
|
Kansas |
Miami, |
|
Maryland |
Worcester |
|
Michigan |
Marquette |
|
New Mexico |
Valencia |
|
North Dakota |
Rolette |
|
Pennsylvania |
Bucks |
|
Oregon |
Hood River, |
|
Texas |
Taylor |
The Web Soil Data Viewer technical architecture is implemented through the use of commercial off-the-shelf information technology from Microsoft, Compaq, and Esri. The hardware, software, and data used are:
At the Fort Collins site, two full-height racks house the equipment for this pilot program - one rack for storage and one rack to hold the four Compaq ProLiant
Ô servers. Three Compaq ProLiant 8500 systems house the database and GIS software and one Compaq ProLiant 1850 is used for authentication.The Storage Area Networks (SAN) rack is able to hold up to six terabytes of data. The modular design of the Compaq hardware provides fast and easy swap capabilities for these critical components, which is important in pilot test mode. The NRCS is using Veritas Backup Exec with a Compaq Tape Library for the enterprise backup solution.
The Web Soil Data Viewer application was implemented using Esri ArcIMS, Esri Spatial Data Engine (SDE), Microsoft IIS, SQL Server, and custom developed COM+ software running on Microsoft Windows 2000 Server and Advanced Server operating systems.
The Web Soil Data Viewer relies on the four databases to provide real-time creation of user selected interpretive soil maps and reports. Custom developed application software integrates the soil geo-spatial data with relational soil data contained in two separate SQL Server databases. The Microsoft TerraServer provides the digital orthophotography and the Lighthouse ArcIMS server creates the soils map using the digital ortho image as a backdrop.
N-tier Enterprise Class Architecture
To get soils information, the user starts the browser at the Soil Data Viewer navigator - which allows the user to navigate to an area of interest based on the map data displayed. The request triggers an XML stream that sends the long coordinates of the selected area to a Microsoft Internet Information Server (IIS) server, and then to the application server where ArcIMS is running.
ArcIMS takes the information and sends it to the SQL Server database through ArcSDE. The ArcSDE software turns the request into an index search into the data tables on SQL Server and pulls back the mapping tables. These go back to the IIS server and then on to the browser.
The next step is to zoom in to a scale where the aerial photography is displayed. At the Web server, custom-written COM+ middle-tier DLLs detect that the map is at the appropriate scale to retrieve a TerraServer image. At this point an image request is made to TerraServer.NET. The http request string generated is:
http://terraserver.microsoft.net/GetImageArea.ashx?t=T&s=S&lon=LON&lat=LAT&w=IW&h=IH&f=Arial&fs=6&fc=ff000000
Table 2 - TerraServer Image Request
|
Parameter |
Description |
|
T |
Theme of image data: T = 0 for DOQ T = 1 for DRG |
|
S |
Size of pixel per side: S = 10 for 1 meter S = 11 for 2 meter S = 12 for 4 meter S = 13 for 8 meter S = 14 for 16 meter S = 15 for 32 meter S = 16 for 64 meter |
|
LON |
Longitude at the center of the requested image, in decimal degrees. |
|
LAT |
Latitude at the center of the requested image, in decimal degrees. |
|
IW |
Image width in pixels. |
|
IH |
Image height in pixels |
TerraServer.NET replies with a jpeg of the ortho imagery. Next, the vector data being pulled from the SQL Server database, such as roads, bridges and stream courses, is stacked on top of the base ortho imagery to produce the final image sent back to the navigator.
Figure 5 - TerraServer Orthophotography
The Web Soil Data Viewer browser uses DHTML and behaviors under Internet Explorer 5.0 or greater for rendering application content. Data requests and data services to and from IIS Active Server Pages (ASP) are implemented with XML.
Figure 6 - Web Soil Data Viewer Interface
Data is stored and accessed as three separate SQL Server databases: SDE SSURGO spatial database, National Soils Information System Soil Survey Distribution database, and the business rules database. The Soil Data Viewer uses these three databases by binding the data and retrieving imagery data from the TerraServer database to render a custom soils map for the area of interest.
Once a particular interpretation is selected to make a soils map, the application goes through the same processes as the navigator, beginning with the ortho-image from the TerraServer. The application also retrieves the spatial soils data from SDE and SQL Server for the selected latitude and longitude. Customized COM+ DLLs bind the spatial and tabular search in SDE and SQL Server. The software retrieves the tabular information from a complex soil database containing more than 60 tables and generates XML to create the tabular report returned to the user's browser.
Middleware software components are implemented with Internet Information Server and ArcIMS connectors. Custom middleware was developed to provide data access to a business rules database, relational soils data, and queries connecting SDE spatial data to non-SDE soils data.
Figure 7 - Client to Web Server Tiers
The Web Soil Data Viewer provides an easy to use tool for geospatial analysis of soil information for resource assessment and management. The application shields users from the complexity of the soil database and incorporates the "business rules" for appropriate use of soil data when creating soil-based thematic maps.
The soil survey attribute database associated with the spatial soil map is a complicated database with more than 60 tables. The Web Soil Data Viewer provides users with Internet access to soil interpretations and soil properties while shielding them from the complexity of the soil database. Each soil map unit (polygon) may contain multiple soil components that have different use and management. The Web Soil Data Viewer makes it easy to compute a single value for a polygon and display the result, relieving the user from the burden of querying the database, processing the data and linking the results to the spatial soil map.
The Web Soil Data Viewer contains processing rules for appropriate use of the data. These rules for the type of soil interpretation or soil property that the user can select as follows:
The soil interpretation or soil property of the soil mapunit component that has the largest percent composition is used to class the map unit. Where two or more components have equal percent composition the component that has the most restrictive interpretation is used.
For each component in the map unit the interpretation rating class or soil property are grouped into like rating classes or property value. The component percent compositions are summed for each group of rating classes or soil property. The rating class or soil property that has the largest percent composition is used to class the map unit. Where two or more interpretation groups have equal percent composition, the group that has the most restrictive interpretation is used.
Out of all the selected components in the map unit, the component that has the most limiting restriction for the interpretation is used.
Out of all the selected components in the map unit, the component that has the least limiting restriction for the interpretation is used.
Weighted average is a method of processing numeric soil properties or productivity values (crop yields) for multiple components and arriving at a single value for the map unit. Weighted average means that each component's value contributes to the final answer based (weighted) on the components percent composition of the map unit. Components with larger percent composition contribute more to the final answer than minor components of small percent composition. For each selected component, the numeric property or productivity value is multiplied by the decimal percent of the component percent composition. These values are summed for all selected components and represent the weighted average value for the map unit.
All the components in a map unit are evaluated for the presence or absence of an interpretation or soil property. The map unit can be classed where the interpretation is present in all components, absent in all components, partially present in all components, or unknown. This processing method is used for the hydric soil map unit rating.
The application server is an 8-way computer that is the geoprocessing workhorse of the system. The image servers, feature servers, and query servers running under ArcIMS are located on the application server. These spatial servers are configured to generate vector-based navigation maps, compile annotated feature data over TerraServer ortho imagery, retrieve spatial soils data within a given extent, and return feature attribute information. The spatial servers connect to SDE in the database tier.
On-site data is stored and accessed in three separate SQL Server databases: SSURGO spatial database, National Soils Information System (NASIS) Soil Survey Distribution database, and Business Rules database. Off-site digital ortho data is stored and accessed from the Microsoft TerraServer.
Figure 8 - Application Server and Database Server Tiers
Soil Survey Geographic Database (SSURGO) data are available for selected counties and areas throughout the United States and its territories. The map extent for a SSURGO data set is a soil survey area, which may consist of a county, multiple counties, or parts of multiple counties. A SSURGO data set consists of map data, attribute data, and metadata.
Field mapping methods using national standards are used to construct the soil maps in the Soil Survey Geographic (SSURGO) database. Mapping scales generally range from 1:12,000 to 1:63,360; SSURGO is the most detailed level of soil mapping done by the NRCS. SSURGO digitizing duplicates the original soil survey maps. This level of mapping is designed for use by landowners, townships, and county natural resource planning and management.
Digitizing is done by line segment (vector) format in accordance with NRCS digitizing standards. The mapping bases meet national map accuracy standards and are either orthophotoquads or 7.5-minute topographic quadrangles. SSURGO data are collected and archived in 7.5-minute quadrangle units, and distributed as complete coverage for a soil survey area. An automated process joins soil boundaries ending at quad neatlines to adjoining maps to achieve an exact match.
The spatial portion of SSURGO is linked to the map unit information in the attribute database. The attribute database gives the proportionate extent of the component soils and their properties for each map unit. Examples of information that can be queried from the data base are available water capacity, soil reaction, salinity, flooding, water table, and bedrock; building site development and engineering uses; cropland, woodland, rangeland, pastureland, and wildlife; and recreational development.
NASIS Soil Survey Distribution database
NASIS (the National Soil Information System) is the information system used to create and manage soil surveys. It takes advantage of database technology to provide an automated means for storing all information about soil surveys. NASIS maintains the hierarchical structure of soil survey data, through the use of table-oriented editors, but allows for new flexibility in creating and maintaining soil survey data.
Exports from NASIS produce the attribute data for SSURGO and support the Soil Data Viewer application. While a typical export from NASIS includes all data for a single soil survey area, an export may contain data for more than one soil survey area. The exported tabular soils data are used to populate the SQL Server database for the Web Soil Data Viewer.
The Web Soil Data Viewer Business Rules database embodies information for aggregating, processing, and querying the complex relational soils attribute database. This business rules database provides software system agility for changing user interface and business level rules to meet future business needs.
Figure 9 - Business Rules Database
The user interface menus and options, business rules to process data, and predefined SQL query statements are implemented in a third SQL Server database which is accessed at run-time by the web based application.
Microsoft TerraServer database
The Microsoft TerraServer (see http://terraserver.microsoft.com) provides access to off-site digital orthophotography data. The TerraServer database is hosted on a four-node Windows 2000 Datacenter cluster. Each node is a Compaq Proliant 8500 computer, each with eight 733-Mhz processors and 4 GB of memory. The 2.4 TB SQL Server database includes USGS topographic maps, digital ortho-quadrangle aerial imagery, Microsoft Encarta Virtual Globe shaded relief maps, Gazetteer and related metadata. The first generation TerraServer became available in June 1998. One year later, over 48 TB of data has been transferred to Internet users. The Web Soil Data Viewer uses TerraServer imagery as the background imagery for navigation and generation of interpretative soils maps.
Figure 10 - TerraServer Orthophotography
Information Rich Products from Data Rich Warehouses
This rapid assimilation of tabular and image data has eliminated a manual process that has been in place for more than fifty years. In the past, field personnel would research a published document to find the desired soils information and look up what soils were on that site. Once they found those paper images, they could then assess the limitations of that soil for building a foundation or planting a particular crop. Because this involved a series of flat paper documents, they would have to visualize an overlay of the images.
Using the pilot system, users could go to the Internet, navigate to a specific location, and then quickly generate soil interpretative maps and reports using the latest GIS technology. Starting with a user request in the Geospatial Data Gateway, users could download both complex GIS images and detailed tabular data from customized queries - within minutes of access.
The Web Soil Data Viewer is the next step in evolving NRCS information technology to overcome the "data rich, information poor" condition identified in the 1995 Blue Ribbon panel report. Today USDA can begin to use their data assets to produce useful "information rich" products that serve the needs of their customers and the public.
USDA-NRCS November 1995 Data Rich and Information Poor, A report to the Chief of the Natural Resources Conservation Service by the Blue Ribbon Panel on Natural Resource Inventory and Performance Measurement
USDA-NRCS August 1997 Future Directions: A Vision of Information Technology for Field Conservationists
USDA 1998 Common Computing Environment ( USDA-NRCS Lighthouse Report on Architecture Workshop, NRCS Resource Data Gateway Project - Workshop held on January 18 - January 22, 2000, Compaq Solutions Group and Microsoft Federal Consulting.
Microsoft, Access, SQL Server, Visual Studio, .NET, and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the U.S.A. and/or other countries.
ArcView, Spatial Data Engine and Avenue are either registered trademarks or trademarks of Environmental Systems Research Institute, Inc. in the U.S.A. and/or other countries.
Steven Ekblad is the Development Team Leader for the USDA Service Center Initiative, Mobile Computing and Desktop GIS, Business Process Re-engineering project. Steve is a graduate from Texas A&M University with BS and MS degree specializing in natural resource information systems development and implementation. For the last 12 years he has managed software development projects for USDA - NRCS, Information Technology Center, Fort Collins Colorado.
http://www.itc.nrcs.usda.gov/toolkit Eric Strand is Vice President of Synergetics Incorporated and is the contractor project manager for the Customer Service Toolkit project team. For the last 20 years he has led information technology development projects emphasizing GIS for federal government and private industry applications. He is contributing editor to GEOWORLD magazine.
http://www.synergetics.com Terry Aho is the Business Area Soil Scientist for NRCS information technology development of applications for customer delivery of soils information. With 25 years of experience in soil survey activities and data delivery and the last 5 years providing soil science business area analysis for the technology development projects of USDA - NRCS, Information Technology Center, Fort Collins, Colorado.