Hydrography Framework Clearinghouse for the Pacific Northwest (www.reo.gov)
Through a collaborative effort with all federal, state and local agencies, this partnership has established a single hydrography clearinghouse and support structure. This structure includes data integration protocols, route/event creation and editing tools, core (required) data attributes, and roles and responsibilities for each partner.
The clearinghouse software suite was developed by Esri and the Regional Ecosystem Office (REO), and based on Arc 8.1 and SDE software. Through the Internet, users first 'check out' or lock an area for update then 'checked in' updated hydrography. The system validates the data for accuracy and completeness. It is then posted to the clearinghouse and this update is available for viewing (ArcIMS) or downloading.
This effort in collaboration is a model for other regions of the country in order to implement framework data across agencies. For further information or to visit the clearinghouse visit www.reo.gov.
Development and Implementation of an Interagency Clearinghouse for Hydrography:
Authors:
Dale Guenther, Regional Ecosystem Office (dguenther@fs.fed.us)
Dan Wickwire, BLM Oregon State Office (dan_wickwire@or.blm.gov)
Ken Adee, US Forest Service Region 6 (kadee@fs.fed.us)
Bob Harmond, Oregon State Water Resources (robert.c.harmon@wrd.state.or.us)
Additional Support: Carrie Wolfe, Washington Dept. Natural Resource (carrie.wolfe@wadnr.gov)
Background
The agencies in the Northwest have a long history of intergovernmental coordination. The management of Salmonid, Spotted Owls, and other Threatened or Endangered Species has required state, private and federal partners to coordinate on a variety of fronts. The national Federal Geographic Data Committee framework effort also promotes the development of interagency core datasets. A very high priority at the national level has been the coordination of hydrography and fisheries data standards and associated GIS formats.
In 1996 the Pacific Northwest (PNW) executives agreed to a set of core data standards for hydrography and fisheries. This allowed agencies to share critical information in support of an ecosystem based management. The sharing of this information then uncovered the need for a mechanism for a coordinated dataset for hydrography.
Many datasets for hydrography information are maintained at the local level in the PNW. Snapshots can be created for a combined dataset, but these are expensive to develop and are quickly out of date. State, private and federal partners agreed to the development of an interagency Framework Clearinghouse in order to store and share an integrated dataset for 1:24,000 hydrography (or better).
It was recognized early on that, in order for this clearinghouse to be successful, a high level of coordination and support was necessary at all levels for each agency. To provide adequate input into the clearinghouse design and in order for the clearinghouse to meet agency business needs, a series of ‘Hydro Summits’ were convened. At these summits State and Federal hydrologists, fisheries biologists, GIS, database experts came together to outline the many functional areas which needed to be addressed for this Clearinghouse.
These summits began in 1999 and continued for the ensuing year. The products of these summits now provide the foundation for the Framework Clearinghouse. These are a set of agreements, which cover a variety of coordination points and decisions. The following documents, which outline key aspects of this process, may be found on the Regional Ecosystem Office (REO) web site http://www.reo.gov/:
- Hydro Project Scope
- Charter
- Business and Data Requirements
- Decision Making Process
- Roles and Responsibilities
- Spatial Features and Core Data Model
- Framework Protocols
- Data Dictionary
The term ‘clearinghouse’ has a variety of descriptions. The Hydrography Framework Clearinghouse may be described as a central repository where partners co-manage the interagency hydrography layer. Updates are submitted in a controlled environment and the resulting dataset is available to all for analysis and mapping purposes.
Esri Corporation was contracted to provide consulting services in the design and to provide the technical expertise for developing the system. As the data dictionary and protocols were developed and agreed upon, Esri provided critical input to ensure that these design elements could then be successfully integrated into the clearinghouse system.
There were many agency criteria and issues to recognize as the clearinghouse was being developed. While a full list of these issues may be found at the above web site, several are considered paramount and are provided below:
- The system must be easy to use in order to minimize resources needed to maintain it. The process for data stewards to submit and verify updates must not adversely impact existing agency priorities and workloads.
- It must be able to support multiple scales. The goal is to develop an active repository for the best available data, as it is being developed for specific areas.
- Water courses (e.g. streams), water bodies (e.g. Lakes), waterbody shorelines (e.g. coastal areas), and water points (e.g. springs) will be managed by the Clearinghouse.
- It must link to existing agency specific hydrography systems. Most agencies have systems to track hydrography. This clearinghouse must seamlessly link to these to reduce conflicts.
- It must be internet-based providing ease of access to all potential users. Updates and requests for new data must be easily accessible. Anyone may request hydrography data from the clearinghouse. An ID/Password is required to submit information.
- The data must be centrally based to reduce redundancies, yet based on field information. This will provide the most detailed and up to date information.
- The Clearinghouse system must be easily maintained. Significant resources are not available for maintaining an intricate system.
- The basic data model will be the Esri Dynamic Segmentation system for tracking events on a routed coverage.
The Hydrography Clearinghouse is based on a client/server architecture. The server (described below) is housed in the Regional Ecosystem Office (REO) in Portland, Oregon. It may be accessed through any Internet connect to: http://hydro.reo.gov. Downloading or querying of Clearinghouse information is not restricted.
In order to submit information to the Clearinghouse a client package is first downloaded from the web site and installed. This package provides the tools necessary to ftp, edit, and quality check data for the Clearinghouse.
Example: Selecting an area to ‘check out’ and update.
The user then accesses the web site and submits a polygon, or selects the 5th Level Watersheds (www.usgs.gov) for the area in which they wish to update. In order to prevent duplicate or inconsistent updates the Clearinghouse system then ‘locks’ this area from further updates. The status map (above) is updated to show in red those areas checked out. Other users may then see which areas are checked-out, and by who and when.
The Clearinghouse then bundles all hydrography information that falls entirely within the selected polygon, including water courses, waterbodies, waterbody shorelines, waterbody points, and all related event tables. The surrounding data is also included in order to ensure edge matching of old to new data. This area is bundled as a background coverage. The Clearinghouse then FTP’s this dataset to the user’s server, and notifies the user via email that the data is now available.
Initially the Clearinghouse will be populated with existing 1:100,000 hydrography linework. The first step in the update process is to conflate Routes and unique identifiers from the existing Clearinghouse hydrography. This provides a link to the existing 1:100:000 data and associated relational databases.
To support the development of new stream routes, or to edit routes, a series of tools have been developed (see Supporting Systems below). These tools greatly facilitate the development of hydrography data into the proper format for the clearinghouse. These tools may be used in conjunction with agency specific hydrography systems.
Once updates have been completed for the downloaded data the client tools are then used to quality check (QC) the data. The QC routine checks for line problems (dangles, gaps, etc), routing problems, and ensure that all attributes have been submitted following established protocols and dependencies. The client package then bundles and FTP’s the updates to the Clearinghouse. Only the 4 main coverages are sent back to the Clearinghouse. Since the background coverage was not edited, it does not need to be returned to the Clearinghouse. The Clearinghouse manager and user are notified via email when this process has been completed.
Once the data is received, the Clearinghouse will again QC the spatial and event data to ensure that it meets protocols. If it passes this check the Clearinghouse Manager then merges the updates into the SDE layer. The area checked out is then ‘unlocked’, and available for others to update. If the coverage does not pass QC a detailed listing of errors is sent back to the user for review. In the final step, the other partners in that area are notified via email that the data has been updated. The updated data is then available to all users through the web site.
The database engine supporting the Clearinghouse is Spatial Database Engine (ArcSDE). Incorporated into this is ArcInfo 8.x in order to manage the spatial data within the Clearinghouse. An ArcIMS server is used to display and check out areas from the Clearinghouse. MO IMS is used to manage transaction ID’s.
A series of AML’s were developed in order to manage the check out, check in and QC edit routines. These AML’s automate the processes relative to the Clearinghouse. The Clearinghouse Manager intervenes only when exceptions occur (faulty data) and when the final merge to the master is required.
A java-enabled browser is used for data viewing, check-out, and check-in operations. Once coverages are checked out, an AML-based client package is used to manage, QC, and submit updates to the Clearinghouse.
A detailed User’s Guide and a series of routing and route/event editing tools have been developed to aid users in the development and update of hydrography data. These tools were developed through a combined effort of the State and Federal agencies in the PNW. These tools are available from www.reo.gov:
- Stream Routing Tool (SRT) – conflates existing routes, names and unique identifiers from one dataset to another. Creates new routes and identifiers, check for line errors, etc.
- Route and Event Editing Tools – Helps users in modifying existing routes and associated events during updates. For example a stream course may change.
- Associated tools – These are a variety of tools in order to calibrate events, match measures from two datasets, convert from Arcs to events, etc.
The Hydrography Framework Clearinghouse is now undergoing beta testing. We then plan on going into a full production fall, 2001.
Since this system is new and unique we anticipate a series of improvements will be necessary in order for it to meet the requirements of the Framework Partners here in the Pacific Northwest. A list of planned enhancements is already forming. Once we begin full production then we anticipate further enhancements will be necessary.
In the long term, we will analyze the conversion of this system to ArcHydro and the ArcGIS GeoDatabase model. The design approach for the Clearinghouse includes this migration path. The obvious path for this will be through the Esri Hydro Consortium. Esri and the Center for Research in Water Resources (CRWR) at the University of Texas at Austin have established a research consortium for GIS in water resources Through this venue we plan on working with Esri and other experts to expand the capabilities of the Clearinghouse. The primary goal is seamless hydrography information. The Clearinghouse provides the functionality to develop this goal, Esri products and the Hydro Consortium provide the tools and the standards to make it work together.
The clearinghouse data model utilizes a unique stream identification system titled LLID (Longitude/Latitude ID). This system is based on the early US Forest Service national system, with this organization (NRIS Water) providing the major funding for this project. LLID was adopted by the PNW partners as the logical system, rather than the USGS NHD (National Hydro Data). The analysis and rational for this decision is available at www.iricc.org. The primary issues focused with critical problems in implementing the NHD model at the 1:24,000 scale, since it was designed for 1:100,000 data. We have been actively participating in the FGDC hydrography standards subgroup, which is tasked to develop and propose a national data model. Supporting the development of a usable national standard, and migrating the Hydro Framework Clearinghouse to that standard is one of our highest priorities.
By establishing the Hydrography Framework Clearinghouse the partners in the PNW have instituted a model for data standards and protocols, which will ensure an integrated and up to date hydrography dataset. The level of coordination and cooperation is unmatched for this level of effort. State and federal agencies have reached a compromise on each facet of this project. While we have only begun field-testing, the Clearinghouse is proving to be a critical tool in the support of an integrated Hydrography dataset.
Without the technical and logistical support of Esri Olympia, we would not have been able to develop a useful and efficient product.