G.S. Tudor
The Washington Department of Natural Resources revised Washington Cadastral Framework SDE layers and implemented them as Geodatabase layers. The dataset layers are based on the Federal Geographic Data Committee's Cadastral Data Content Standard. Esri Northwest was contracted to implement the cadastral GDB layers with feature history and develop ArcMap editing software. Point, boundary, legal description, parcel, and jurisdiction features were implemented using Visio and ArcCatalog. The DNR loaded feature classes and relationship classes using ArcCatalog and SDE Administration tools. The DNR and Esri Northwest developed practical methods of implementing GDB layers and loading data into the GDB layers.
The Washington Department of Natural Resources (DNR) manages 2 million acres of uplands and 3 million acres of aquatic lands held in trust for schools, universities, the capitol, counties, and the public. Since 1985, the DNR has been using Environmental Systems Research Institute's (Esri) ArcInfo to manage these lands. The public land survey, ownership, county, and admini
G.S. Tudor
The Washington Department of Natural Resources revised Washington Cadastral Framework SDE layers and implemented them as Geodatabase layers. The dataset layers are based on the Federal Geographic Data Committee's Cadastral Data Content Standard. Esri Northwest was contracted to implement the cadastral GDB layers with feature history and develop ArcMap editing software. Point, boundary, legal description, parcel, and jurisdiction features were implemented using Visio and ArcCatalog. The DNR loaded feature classes and relationship classes using ArcCatalog and SDE Administration tools. The DNR and Esri Northwest developed practical methods of implementing GDB layers and loading data into the GDB layers.
The Washington Department of Natural Resources (DNR) manages 2 million acres of uplands and 3 million acres of aquatic lands held in trust for schools, universities, the capitol, counties, and the public. Since 1985, the DNR has been using Environmental Systems Research Institute's (Esri) ArcInfo to manage these lands. The public land survey, ownership, county, and administration (POCA) (DNR, 2002), and public land survey points (PLS-PT) coverages have met basic needs up to this time. However, the land managers using the information, and the land surveyors maintaining the coverages have been interested in extending the usefulness of the information. Some revisions have taken place over the years, but a decision to redesign the layers was made in 1996.
The Cadastral Framework project began by adopting the Federal Geographic Data Committee's (FGDC) Cadastral Data Content Standard (FGDC, 1996). Modifications were made to the data model to accommodate the existing DNR data, and Spatial Database Engine (SDE) layers were built and loaded with data in the first phase of the Washington Cadastral Framework Project (Wolfe, 1998). In the second phase of the project, the data was made publicly accessible over the Internet using ArcView Internet Map Server (IMS) and a data transfer standard was created to allow partners to contribute data to the Framework database. Several partners, the Bureau of Land Management, Snohomish County, and Longview Fibre Company, contributed data and participated in testing the process. (Tudor, 2000.)
Maintenance of the data was still problematic since the data had to be check out to coverages and back in to SDE. In 1998, information was gathered about possible alternatives to moving the data in and out of coverages or using incomplete ArcView editing extensions. In 1999, Esri together with partner NovaLIS was the only respondent to the DNR's request for proposals. The DNR was encouraged to participate in the National Integrated Land System (NILS) being developed jointly by the United States Bureau of Land Management, Forest Service, and Esri. After working as a NILS partner for a year, DNR managers decided to refocus internally on explicit DNR needs. The DNR developed a service contract with Esri Northwest to implement the SDE database as a Geodatabase (GDB) and build editing tools that would support typical maintenance operations. One of the primary goals of the contract was to ensure that the cadastral GDB would be compatible with new releases of ArcGIS software that would include topology and creation of business rules.
The DNR-Esri contract specified six deliverables: Geodatabase design implementation, interface design, application development, test plan documentation, application installation, and application documentation and training. The focus of this paper is the first deliverable: Geodatabase design implementation, developed jointly. Esri began by collecting functional requirements and focused initial efforts on developing a toolset to provide that functionality. After Esri was certain that most of the functional requirements could be met, work started on the GDB implementation.
From the work with the DNR's Cadastral Framework data design, several improvements had been identified for the Geodatabase. The coverage data from POCA, and PLS-PT was being loaded first into a DNR SDE data schema called Cadastre, from there, the Cadastre_Frame SDE schema was loaded for public consumption. The Cadastre_Frame database was a deliverable of the Washington Cadastral Framework Project (Tudor, Wolfe, 1999). The Cadastre schema was overly complex, and the Cadastre_Frame schema did not meet all of DNR's internal needs. The decision had been made to simplify the Cadastre schema to match the Cadastre_Frame schema without losing DNR information. The redesign work was identified in the DNR work request referenced by the DNR-Esri contract. Since this was the DNR's first GDB project, DNR staff did not know the capabilities or limitations of the Visio computer aided systems engineering (CASE) tools, the ArcCatalog data tools, or the GDB itself. Esri staff worked with DNR staff to redesign Cadastre and Cadastre_Frame SDE layers and created the new GDB, Cadastre_GDB. As DNR staff learned more, DNR staff made more of the design modifications.
Before work on the Geodatabase started, the DNR completed a first draft of the revised design using its Oracle CASE environment, and implemented the first set of physical tables using this design. The model included: Parcel as the basic unit of land transactions such as ownership, easements, leases, and permits; Jurisdiction as the areas of administrative authority and management policy; Legal Description as the extent of land defined through legal and surveying processes; Boundaries as the edges of Legal Descriptions and Jurisdictions; and Points as corners anchoring boundaries and control defining measured locations. To be added at a later phase were: Agents as people or organizations having an interest in Parcels or Jurisdictions; and Documents as records of land transactions between agents and sources of the data. (Tudor, 1999.)
Esri transcribed the draft relational design into UML (Unified Modeling Language) with Visio by cutting, pasting, and editing from text reports of the Oracle model design. There was no way to reverse engineer designs between Visio and Oracle, and because of the customization of the Esri object classes such conversion might still have only a limited application. The ArcGIS software comes with a Visio model having the customized Esri object classes, which allows the user to focus on the business problem and not have to worry about the class definition as much. (The base model is under \arcgis\arcexe8x\CaseTools\ArcInfo UML Model.) After the inevitable transcription errors were fixed, more serious changes in the model were identified.
The SDE layers are generally analogous to the Geodatabase layers. The GDB feature classes are implemented as SDE layers, but there are many additional GDB extensions to the SDE repository template. GDB feature classes may be part of a feature data set that carries the projection and scaling information at a higher level. Metadata is built in. The relational database management system constraints are turned off, and constraints are enforced only through the ArcGIS application using relationship classes, domains, and business rules. Domains may be enforced as codes or ranges. Business rules may be developed for data validation procedures (Geometric Networks, and coming soon to Topology in ArcGIS 8.3).
Domains were identified early as the first conceptual difference between SDE and the Geodatabase. In Oracle and SDE look up tables are created and populated with data just as primary business tables are; the GDB stores lists of values in a coded domain kept as a single record/row of binary data that can be defined in the Visio model. Domains can also be created and loaded in ArcCatalog using a developer's sample, but this would also require manually synchronizing with all of the objects using the domain. So these were manually entered into Visio UML by the cut/paste process. At this point there is no developer tool to unload domains to Oracle tables, but this must be done unless another means of accessing the domain can be developed for external applications.
After the draft Geodatabase design in UML was finished, the DNR set up the SDE/Oracle instance and Esri then demonstrated to DNR staff how to generate the GDB using the ArcCatalog Schema Wizard. With this implementation of the Geodatabase, the DNR followed recommendations to allocate uniform local tablespaces in Oracle for the SDE database tuning, and thus reduced the Oracle maintenance overhead for the Database Administrator. The Schema Wizard imports the UML model, checks the model for consistency with the Esri objects, allows users to define the feature data set projection and scaling, and creates the GDB. The process is relatively quick, but is still subject to some user errors. After the DNR was finally able to install a version of Visio that would work with the GDB UML models the DNR took responsibility for regenerating models as changes were made.
The DNR was responsible for preparing the legacy data and loading the legacy data into the Geodatabase. Esri demonstrated the ArcCatalog loading process, and the DNR proceeded to load data using this process for a short time. It quickly became apparent that the ArcCatalog loading process, using a graphical user interface, was prone to user errors, took a lot of user time and attention, and - from previous experience - was slower than processes used to load SDE. The DNR began to use the SDE administration tool cov2sde and the Oracle tool sqlldr to load spatial and tabular data through a consistent programmatic process. Later, the sqlldr tool was dropped in favor of the greater reliability of the SDE administration tool tbl2sde to load non-spatial data.
Once users started testing the Geodatabase, the usability of the design came into question. Land surveyors using the editing tools were happy with the general functionality of the tools. However, the ability to determine what the tools were acting on was a problem. For example, the Legal Description feature class consisted of public land survey townships, township subdivisions such as sections and donation land claims, section subdivisions such as government lots and aliquot parts (1/16ths), and non-PLS subdivisions such as tracts, blocks, lots, metes and bounds, and tidelands; all of these were mixed together. When a Legal Description feature was selected it was hard to tell what kind was selected, and there was no means of selecting a particular type. From experience with region coverages, the DNR expected similar functionality in the Geodatabase. After some discussion with Esri, the major types of the feature class were implemented as subtypes. With the sub-layers defined, users could set the selectable layer to the sub-layer, as well as set the target layer to the sub-layer, with acceptable usability.
Another anticipated use of subtypes, variation of domains by subtype, had been forgotten until user testing revealed its absence. In this case Jurisdiction was in need of subtyping. Jurisdiction types included states, tribal reservations, counties, cities, national parks, state parks, etc. and all sorts of special districts. Each type of Jurisdiction had a domain designating the name of the Jurisdiction. For example, state designations included Washington, Oregon, Idaho, California... County designations included Adams, Asotin, Benton, Chelan... The domain to use depended on the type of Jurisdiction. Jurisdiction was subtyped and the oversight corrected so that each subtype had the appropriate domain.
The next issue uncovered by users was that default values were being inserted to existing features when the Shared Edit tool was applied. In fact, the shared edit tool made use of the Integrate function, which in turn changed existing values to default values. In this example, the Shared Edit tool was used to move a Point corner along with the Boundary, Legal Description, and Parcel. All of the features that had attributes with new values set were reset back to their default values. Only attributes without default values stayed the same. Obviously, this was a big problem for database integrity, unless the idea was to standardize the database to default values. After Esri worked on this for several weeks, a patch was developed and installed back at DNR, which fixed the problem. This patch was also incorporated into the 8.2 Release of ArcGIS.
After the DNR confirmed that the Shared Edit default problem had been fixed, another default value problem surfaced. When the feature was split, the new features had copied the attribute values of the original feature, except for attributes with default values. It was later found that the Split Policy was set to default (by default) instead of duplicate. Although the GDB design was changed to set Split Policy to duplicate, the question remained of why would anyone want it to be default instead of duplicate in any case? Esri decided that it would be better to change the Split Policy to duplicate (as the default) for its ArcGIS 8.2 Release.
Another aspect of the Geodatabase that attracted attention at the DNR is built-in metadata. The basic metadata about the projection, the feature dataset, and the feature classes is generated automatically through the ArcCatalog Schema Wizard design import process. The metadata can be viewed and saved in several formats including a graphical Esri format, the FGDC format (FGDC, 1995), and XML. Modifications to the generated metadata allow addition of definitions to enhance the extent and depth of material. However, the metadata gets overwritten if a revised design is imported. Nevertheless, saved metadata can be imported back into the metadata repository and used where it still applies.
The Department of Natural Resources currently supports many workstation ArcInfo applications and coverages. All of these applications have many interdependencies, being especially dependent on the base layers of public land survey, ownership, transportation, and hydrography. These dependencies have caused many problems in the past when making radical technical changes. In order to minimize the impact of the transition from workstation ArcInfo to ArcGIS, plans were made to recreate the POCA and PLS-PT coverages being replaced from the Cadastre_GDB. POCA and PLS-PT would be backfilled from Cadastre_GDB so that workstation ArcInfo applications using POCA and PLS-PT coverages would continue to function without any modification of the existing applications. New applications would be built using ArcGIS. Subsequently, as applications were moved into the ArcGIS environment, the dependencies would be reevaluated, and when dependencies fell away then backfill processes would be turned off.
The Visio design process has been set up to keep the technical aspect to a minimum and focus on the business requirements. However, the DNR started out knowing nothing about the process, no classes had been taken (created for that matter), not much documentation was available. Both DNR and Esri Northwest learned a lot through trial and error. A summary of the Visio design process would be: Create a folder, called a package, to define the workspace. Create another package to define the feature dataset. Create a static structure diagram to show class inheritance and relate new object classes to parent object classes. Copy the Esri Class:Object and the Esri Class:Feature to the class inheritance diagram. Create and name a new object class on the class inheritance diagram. Relate the new object class to the Esri parent class using the generalization relationship, for example connect the Jurisdiction class to the Esri Class:Feature to make it a feature class, or connect the Document class to the Esri Class:Object to make it a table. Create and name attributes for the new object class; define the attribute field type using Esri templates such as Esri Type::EsriFieldTypeInteger or select a domain to use for the field; enter a default value, if any; define the precision, or length using attribute tags. Create a static structure diagram to show class structures: feature classes, object classes, and the relationship classes between them. Create and name domains by creating an object class and set the stereotype to CodedValueDomain or RangeDomain; set the field type, split policy, merge policy, and values as attributes of the object class; set the precision or length as a tagged value of the field type. Drag composition or binary associations over to the class structure diagram; connect the ends and define the rules of the relationship. Copy an existing feature class and paste the copies to the class structure diagram to start building a subtype; drag a binary association to the diagram, connect the ends between the feature class and each subtype; rename the binary association to "Subtype". For more information, definitely take the Esri training (Esri, 2001, 2002). Visio design saves a great deal of time, and the training will prevent much frustration.
There were some quirks with the Visio CASE tool. To start with, Visio Enterprise must be used to get the UML extensions required to generate Geodatabase objects, and Enterprise was not available off the shelf. The DNR obtained a copy from Esri after getting a Microsoft license; this was a long process. Finally, to install Enterprise, the Professional edition must be uninstalled; successfully uninstalling Visio Professional so that the UML extensions worked was almost impossible.
Some other lessons learned. Delete the repository file before exporting, or export the repository to a new file instead of to an existing file; otherwise, the export process takes a great deal of time working through change differences. Set the configuration keyword in the Visio model by using a tagged value for the object; this saves having to reenter the keyword each time in the Schema Wizard. Do not cut relationship classes from any Visio diagrams; the relationship class does not have any other representation in the workspace, but seems to hang around in memory and causes warnings, not fatal, but annoying. Do not create attributed relationship classes; the relationship's attributes are almost impossible to maintain using ArcMap; instead, create a table having two relationship classes. Create a folder for each feature class subtype and move the subtypes into them to reduce confusion about which class is the master. Pay close attention to the capitalization of all reserved words. The list goes on, but these are the highlights.
There are several data loading cycles that the DNR developed for the initial load of the Geodatabase, the backfill load of production data to the legacy coverages, and the load of data for service to the public. The development process required that the POCA and PLS-PT data be converted and loaded into the newly generated Geodatabase for testing and for the final load to the production Cadastre_GDB. For this process, tiled ArcLibrarian is appended and data converted using Arc Macro Language (AML) and INFO programs. New items are added to the coverage, and new values are converted using the business rules. In ArcInfo coverage form, the data is reprojected from Washington State Plane Coordinate System (WSPCS) North American Datum of 1927 (NAD 27) South Zone US Feet to WSPCS NAD 83 High Accuracy Reference Network (HARN) South Zone US Feet. Then the data is loaded into the Geodatabase using the SDE administration tools cov2sde and tbl2sde. Final changes to the data are made using Structured Query Language (SQL) scripts in Oracle sqlplus sessions.
Once data is loaded into the Geodatabase, the data is backfilled to the legacy coverages POCA and PLS-PT. The update cycle for the legacy coverages had been once per week, and the backfill process would take place on the same schedule. Before the backfill, the Land Surveyor responsible for the Cadastre_GDB would merge any versions that were back into the default production version and compress the GDB. If the merge and compress were not completed before the scheduled backfill then only the current production version would be backfilled to the legacy coverages. The backfill process is scheduled and run by AppWorx batch control software. The first step to backfill the data is to copy the Geodatabase over to a staging area using the SDE administration tools sdeexport and sdeimport. Then legacy columns are added to the feature classes, and legacy values are converted using the business rules. The feature classes are converted using the SDE administration tool sde2cov into multiple polygon coverages for POCA and a single point coverage for PLS-PT. The polygon coverages are then condensed back into the single legacy POCA coverage using the ArcInfo union command. The coverages are projected from WSPCS NAD 83 HARN South Zone US Feet back into WSPCS NAD 27 South Zone US Feet. Dropping the union identifiers, consolidating the data values, and dropping the items used for consolidating the original legacy values using AML and INFO scripts cleans up the coverages. Finally, the coverages are broken back out into their ArcLibrarian tiles and are made available for general consumption by operational workstation ArcInfo applications and users.
During the testing phase of the project, another process to check the backfilled coverages against the original production coverages was developed and used to red flag the differences between them. In this process, the original production coverages were copied to check coverages POCACHECK and PLS-PTCHECK. The check coverages' attribute items are duplicated and an additional Boolean item is added for each attribute item to record the differences (deltas), for example, dx-coord, dy-coord, and ddate. The delta items are zeroed out to initialize them. The check coverages are related to the backfilled coverages and backfill attribute values copied into the check coverages. Then the backfill and original attribute values are compared and the differences are flagged by setting the delta item value to 1. The Land Surveyors then review the differences for systematic and other unexpected errors. This process has identified many problems with the original data that needed to be cleaned up as well as the conversion problems that were expected.
The data served to the public for the Washington Cadastral Framework was at first a subset of the production data. However, as the maintenance load accumulated, the decision to make the data identical was made. This process was fairly simple, just an SDE export and import to move the data between the production Geodatabase and the framework database. Work is continuing in this area as direct access to the production database from the Arc IMS is being developed.
The transition from workstation ArcInfo to ArcGIS has been long and difficult. The DNR has looked at ArcStorm, SDE, and finally the Geodatabase as a solution to transactional processing of spatial data. However, as the DNR has become more familiar with the ArcGIS and SDE tools, and Esri has continued to improve those tools the transition path is becoming more traveled. Procedures have been developed to make the transition process more manageable, and DNR technical staff are getting training about administering SDE and the GDB. The business users (Land Surveyors) have become very excited about the Geodatabase and the custom software that Esri Northwest has developed to facilitate editing, and are looking forward to beta-testing topology after seeing several demonstrations. Other state agencies with mandates to manage land are also interested in adopting the Geodatabase design and the cadastral editing application. The DNR has been impressed that Esri Northwest has been able to get many of the custom tools incorporated into 8.2 and 8.3 Releases of ArcGIS. This enthusiasm has gotten other business users interested in the technology for their projects within the DNR. As the DNR's first production implementation of ArcGIS comes into general usage, users will have an expectation of greater success moving their coverage data into the Geodatabase and solving their complex business problems more effectively.
DNR team: Dave Steele, PLS, Frank Fischer, PLS, Gwen Roy, Bob Wright, Carrie Wolfe, Byrtle Filyaw, Chris Loveland, Tim Young, Kevin Kozak, Barbara Putnam, Steve Kimsey, Travis Butcher, Barbara Blubaugh. Esri Northwest team: Marty Balikov, Scot McQueen, Jeff Barrette, Chris Buhi.
Environmental Systems Research Institute (2002). Modeling Geodatabases Using CASE Tools. Esri, Redlands, CA.
Environmental Systems Research Institute (2001). ArcSDE Administration for Oracle. Esri, Redlands, CA.
Environmental Systems Research Institute (2001). Geodatabase Design Concepts. Esri, Redlands, CA.
Environmental Systems Research Institute (1999). Building a Geodatabase. Esri, Redlands, CA.
Federal Geographic Data Committee (1997). Framework: Introduction and Guide. Somers-St. Claire, Fairfax, VA. http://www.fgdc.gov/framework/frameworkintroguide/.
Federal Geographic Data Committee, Cadastral Subcommittee (1996, December). Cadastral Data Content Standard. FGDC, Reston, VA. ftp://ftp.fairview-industries.com/pub/cad-stand-1-1.pdf.
Federal Geographic Data Committee (1995, March 24). Content Standards for Digital Geospatial Metadata Workbook. FGDC, Reston, VA.
Tudor, G.S. (2000 May 30). Washington Cadastral Framework Project, Phase 2, 1998-1999 Framework Demonstration Project, Final Report. http://framework.dnr.state.wa.us/cadastre/public/communications/report/fgdc20000530finalreport.htm.
Tudor, G.S. (1999). Understanding and Implementing the Federal Geographic Data Committee's Cadastral Data Content Standard for the National Spatial Data Infrastructure. Esri Conference Proceedings, Redlands, CA, http://gis.Esri.com/library/userconf/proc99/proceed/papers/pap955/p955.htm.
Tudor, G.S.; Wolfe, C. (1999). Washington State Cadastral Framework Project: Implementing the FGDC Cadastral Data Content Standard and Integrating Data from Multiple Sources. Journal of American Congress on Surveying and Mapping, Bethesda, MD.
Washington Department of Natural Resources (2002 June 20). Washington State POCA (Public Land Survey, Ownership, County, and Administration) FGDC Metadata. Federal Geospatial Data Clearinghouse, Washington State Geospatial Clearinghouse Node http://agdc.usgs.gov/AGDCgateway.html, via Search: POCA.
Wolfe, C. (1998 December). The Washington State Cadastral Framework Demonstration Project - Phase 1, Final Report. http://framework.dnr.state.wa.us/cadastre/public/communications/report/finalfgdcrpt1298.htm.
G.S. Tudor, PLS
Assistant Division Manager
/Spatial Database Administrator
Data Resource Management Section
Information Technology Division
Washington State Department of Natural Resources
PO Box 47020
Olympia, WA 98504-7020
E-mail: greg.tudor@wadnr.gov
Voice: 360-902-1542
Fax: 360-902-1790
Web: http://www.wa.gov/dnr/