King County, Washington's, Coordinated Geographic Information System (KCGIS) Program was created when the pending merger of King County and the Municipality of Metropolitan Services (Metro) presented a unique opportunity to consolidate GIS efforts within nine different agencies. This consolidated effort eliminates redundancies, gains efficiencies and results in a system which better serves as a regional resource in a cost-effective manner.
This presentation details the infrastructure designed and the implementation of a centralized core data library system, an OSF/1 based ArcInfo server. This system has been sized and configured to accommodate all maintenance of the core data. It also allows for query access and copying of the core data as needed by KCGIS users on the network. Data storage supports a combined spatial and attribute database of 15 gigabytes, application software requirements and space for growth.
This presentation also details the implementation of some of the GIS sites participating in the Program. For example, in Metro, the ArcInfo server has been designed and implemented to support up to 15 concurrent ArcInfo sessions. Also, for query access, a Novell based server supporting up to 45 ARC/View users has been designed and installed for PC users in the Transit and Water Pollution and Control Departments (WPCD).
In addition, also detailed are the challenges of implementing a Frame-Relay Wide Area Network (WAN) for the nine agencies participating in the Program and accessing the central server with their existing hardware systems which include Sun Microsystems, Inc. (Sun), Hewlett Packard (HP) and Digital Equipment Corporation (Digital) products.
This Program is a major step towards developing a regional geographic resource which serves both incorporated and unincorporated King County. It builds on work done independently within King County to avoid duplication of effort. Data and projects from several agencies will be "fit" together under the Coordinated Program to build an integrated, regional GIS held together by common database standards and design, shared data networks and a common user interface.
Many King County, Washington agencies have been independent Esri ArcInfo software users for a number of years. The King County Coordinated Geographic Information System Program (KCGIS) is focused on providing KCGIS Departmental sites with a centralized core database, and the hardware and network infrastructure with which to access the data. It was decided to implement the infrastructure over a nine month period during the planning and specification phase of the data conversion effort. This tight time-frame highly influenced the implementation approach taken.
The Infrastructure Project is comprised of three very distinct efforts focused on:
It is difficult to talk about one effort without addressing the others because of the interdependencies, so this paper describes all three of these efforts.
Prior to deployment, scoping documents and requirements specifications were reviewed to ensure the infrastructure design met KCGIS agency expectations. The fundamental business issue identified was to provide a cost effective and reliable infrastructure to support the KCGIS core data. The fundamental technical issue identified was to provide sufficient network, CPU, storage and support resources to enable users to productively share a GIS. Ease of maintenance and reliability have been favored in the infrastructure design in recognition that cost of ownership is the most expensive portion of the system costs.
An initial stated goal allowed for the function and position of equipment to depend on where development, maintenance and query access is performed. In deciding on a centralized database, three reasons were cited for keeping the number of copies of the KCGIS core data to a minimum:
The initial design concept was based on a central processor (ArcInfo server) and a high speed WAN with a minimum number of workstations accessing the ArcInfo data. Maintenance and query functions were to be provided via X-terminals or X-emulation over the WAN.
Through initial testing of this design, the following changes in the overall design concept occurred:
Based on our time-frame for implementation, we adopted a general principal of prototyping solution ideas prior to full implementation. In prototyping systems we learned about integration issues and this approach gave us the flexibility to change direction and/or technology very quickly without significant cost investment.
For example, Metro’s plans included implementing a solution very similar to what was scoped for the KCGIS systems. By actively participating in the design, implementation and testing of the various solution components for Metro, we identified the technical limitations of a central processor approach for the KCGIS solution. Using this approach, we could not expect remote users to achieve the same performance goals achieved for local users, without significant cost. It was decided that the majority of the work needed to be performed at the site location using the WAN to transmit and receive core data. We managed the design so that these remote users are not always going across expensive WAN links to get work done.
We also made a conscious decision to postpone hardware purchases for as long as possible. Technology is changing rapidly so we were able to deploy solutions with the latest technology taking advantage of the price performance curve. We could also base our decisions on real world experience with the technology chosen by implementing only what we needed at that time. We also made the decision to have the implementation team purchase all equipment centrally. By using this approach to purchasing, we were able to establish blanket contracts with all key vendors to obtain maximum discount levels for new purchases as well as take advantage of volume discounts for new and existing maintenance service contracts.
The following criteria were researched and analyzed to determine the final central and site systems design:
The general design principal for the KCGIS core data library is to implement systems which support building and maintaining this data. Very little analysis of the data would be performed at the KCGIS site. The production environment is designed as a centralized core data library server, supporting a limited number of ArcInfo users, generally performing local maintenance of production data; and a separate development environment, supporting development activities. KCGIS users access the core data server and retrieve core data to their site locations for analysis.
The KCGIS core data library server design supports the following key requirements:
A major design decision was to not use NFS as a data sharing technology in the WAN environment. NFS is supported and implemented very successfully in the LAN environment, where it supports data sharing and work group integration. The decision to not extend this to the WAN was based on a desire to minimize network traffic as a component of application execution and the technical concerns about supporting NFS in a multi-pathed routed WAN environment.
From this technical decision it followed that the model for implementation would be a high capacity central system focused on data storage and management with several distributed data and application development work groups deployed at various organization sites. This could be characterized as distributed maintenance and use of the data, with centralized "publishing" of officially released data. The core data server is also configured to support whatever local data maintenance is required, with access for X -Windows devices across the WAN.
The basic sizing of the system components was based on the requirements analysis, application capacity requirements supplied by the vendors, and performance analysis. Performance testing was done based on a test suite developed at Washington State Department of Natural Resources (DNR) and modified to fit the requirements of KCGIS. The testing was not done as a benchmark, but as a proof of concept capacity test. The requirements for storage derived from the analysis resulted in 30 GB of disk. Memory was set at 680 MB, and a dual processor Digital Alpha Server 2100 was chosen as the processor. All components of the system allow for future expansion. The standards implemented on the server, mainly TCP/IP and the X-Window system, have permitted excellent integration with the rest of the systems in the KCGIS Program.
KCGIS development systems are designed to be used primarily for the development of user tools in accessing, querying and transferring data. The separate development environment gives us maximum flexibility in supporting the testing of new technologies as well. We can test here, prior to general release of a new supported baseline.
KCGIS sites are basically independent GIS shops. The sites are responsible for deciding the design of their GIS to support their business analysis requirements. Since several county agencies already had existing systems, their designs included utilizing what the users already had and complementing their solution with the necessary query access systems, peripherals and additional data analysis workstations to perform their business functions. High-performance UNIX based workstations were required to support the GIS analysis work being done at the sites.
KCGIS sites, with existing GIS equipment, already had in place UNIX based workstations. This open systems approach, with the flexibility of the Esri software to support multiple platforms, allowed the various agencies to use equipment already supported within their departments. The overall site design did not restrict users from their vendor of choice, though it was cheaper, easier and faster to purchase new systems from the key vendors we had blanket contracts with.
The need for a lower cost user access device also resulted in including in the design an Intel based Personal Computer (PC) running an X-emulation software package. After an extensive evaluation, we standardized on PCX-Ware from NCD as the X-emulation package.
A separate King County Wide Area Network (KCWAN) Design Team, conducted an extensive WAN design effort to identify a WAN architecture for connection between County buildings. In general, the design team favored the implementation of a combination of flexible technologies, platforms that could support sites over the long term, and technologies that work together effectively. Since it was known that several projects required a WAN earlier than the projected KCWAN implementation, the KCWAN design team presented implementation guidelines to the KCGIS Program.
The KCGIS WAN, as designed, uses Frame Relay services. For data-only service requirements at 1.5 Mb/s or less, the Frame Relay WAN offers us improved bandwidth efficiency, reduced capital requirements and reduced network delay when compared to other options such as X.25, private lines, SMDS and ATM (based on today’s pricing). A stated goal of the KCWAN project is to implement ATM technology for several of our county agencies, including several of the KCGIS sites. Implementation of Frame Relay by the KCGIS team, provided a lower cost solution during this interim phase with a direct migration path to ATM during the KCWAN project roll-out.
As we describe our implementation efforts, we will pass on many of the lessons that we learned. We accomplished our goals in a relatively short time frame. We also established some good baselines against which we can go back and review our design assumptions at the time. This has proven very valuable already as we moved from site to site and especially so in the actual implementation of the core data library server.
The team members for the infrastructure implementation were selected for their high level of expertise and extensive experience. This helped greatly with the rapid deployment we were about to undertake. Two teams were established for the network and computing hardware implementations. A team of four, for six months, focused on the roll-out of the KCGIS and Public Works WAN. A team of two, for nine months, focused on the implementation of the central and site systems with support from local resources at the agency sites.
Throughout the effort we contracted, as needed, for additional part-time resources to perform specific implementation tasks, such as upgrading PCs, installation services, and network product testing and training.
A centralized core data library, where KCGIS agencies could make available and/or access KCGIS agency data has been implemented for the KCGIS program. Figure 1: KCGIS Diagram, KCGIS Central, documents the systems implemented.
Please note that on each site diagram presented herein, the WAN products and network connections are also shown. Frame Relay uses a technology called Permanent Virtual Circuits (PVC) within it’s structure to designate a direct 'hop' between sites. PVCs are established between sites based on performance requirements. The number of PVC's, and who they are directly connected to, are shown in each diagram and described in the KCGIS WAN Implementation Section.
The central system provides a structured, controlled production environment where large volumes of data are maintained and made easily accessible on the network. Used primarily for the development and maintenance of core data, several technologies were required to support the requirements outlined. The central data server must support data integrity for the production environment. This means that multiple gigabytes of data are managed and maintained on a high availability basis. The server is also a multi-processor system which provides high compute performance with excellent scalability and high I/O bandwidth. The use of RAID storage technology and Digital's Advanced File System provide high availability, high performance and excellent configuration flexibility.
The central server also employs high performance hardware and software to backup, restore and archive the large volumes of data involved. The software used for backups is Networker from Legato Systems. The hardware is a Digital Linear Tape (DLT) cartridge tape stacker which supports seven 20 GB cartridges. The "clone set" functionality of the Advanced File System (AFS) permits consistent backups of data even when the file system is in active use.
In addition, AFS allows flexibility in configuration set-up (dynamically add and remove space), logical disk capability (group physical disks into logical volumes), faster boot up time (NO fsck), file striping, and on-line defragmenting.
The KCGIS server has several network connections with FDDI as it primary interface.
The server configuration, as shown in Figure 1, is expandable up to four CPUs, memory to 1 GB, and additional I/O controllers which allows growth to support a higher number of users.
The development environment for KCGIS is implemented to be a more open environment since it is used to test new applications before production release, to run Beta versions of software, and to permit easy cooperative interaction among the development staff. The environment uses NIS and NFS to provide a consistent setup for users on any of the systems.
This implementation provides general connectivity to the centralized core data while keeping local network traffic isolated to the local development environment. We extended this concept through the implementation of an Ethernet switch for the development environment. The switch provides very high bandwidth in the local development environment and provides another layer of filtering to keep traffic isolated to the local segment.
The development environment consists of a combination of systems, including UNIX workstations, general purpose PCs and X-Terminal devices. We established baseline configurations for each of these and have implemented the site specific equipment as identified in the site diagrams.
Data maintainers and application developers require a high performance workstation or high performance PCs to perform the work assigned. The minimum developer workstation specification is:
UNIX Workstations: PC Workstations: Sun / HP / Digital Alpha Workstations Intel Based Pentium 90 Processor 32 MB Memory, minimum 32 MB Memory, minimum 2 GB Disk Storage 2 GB Disk Storage 20” Monitor 20" Monitor Tape Back-up System 2 MB High Performance Video Card Solaris, X, OSF/1 MS/DOS, 6.0 or greater Esri ArcInfo MS Windows, 3.1 or greater Other Esri applications as required NCD PCX-Ware, 2.0 or greater Esri Applications as required
For user access, we also established two baselines based on Esri’s recommendations and our own testing. The minimum user access specification is:
PC Workstations: X-Terminal Workstations: Intel based 486/66 personal computer NCD, HP, SUN X-Terminal 16 MB Memory 16 MB Memory 500 MB Disk Storage High Performance Video Card, 1 MB minimum. With new systems we ordered 2 MB cards 17” monitors for ArcView only users 20” monitors for ArcView and ArcInfo users MS/DOS 6.0, or greater MS Windows 3.1, or greater Esri ArcView Windows 2.0, or greater NCD PCX-Ware, 2.0 or greater
To give some idea of the general site systems implemented, three different site configurations will be presented. Figure 2: KC GIS Diagram, Metro, represents the implementation at Metro of Digital systems. One of the more challenging implementation issues, revolved around providing 45 users query access, via ArcView, to Metro’s data. While several implementation strategies were identified and tested, we implemented a Novell based ArcView Server to support this for Transit and WPCD users.
While initial design strategies involved copying a sub-set of Metro’s core data to the ArcView server for query access, through testing we implemented a different model for the short term so we could further evaluate testing results. Our implementation consists of performing an NFS mount of the data residing on the Metro’s UNIX server with the Compaq server acting as an NFS Gateway to the Novell desktops. The Compaq server is the application server for ArcView and PCX-Ware users, as well. Since we have FDDI connections between the two Metro servers and most users are working on the same network backbone, data access, so far, has met performance requirements.
The other interesting challenge of this task was how to ensure that the 45 users selected for participation, had PCs appropriately configured for running ArcView, and in some instances, also running PCX-Ware. User access presented opportunities to explore the most optimum configuration for accessing the data both locally and remotely. Use of existing desktop devices was a critical factor during implementation and we used the baseline PC configuration established for user access as the model for these users.
Since the majority of users already had existing PCs, we conducted an evaluation to determine what was currently residing on the desktop and what type of work that individual would be performing during the work day. We were quickly able to identify which systems required replacement and which could possibly be upgraded. New systems would replace any 386 model or 486 model which required a CPU upgrade. Our upgrade strategy was to only upgrade memory and/or add a 420 MB disk to any systems that did not have the 500 MB minimum requirement. While this appeared to be the best approach to this issue, upgrades were difficult and sometimes more labor intensive than we expected. We did not receive, in all instances, the latest accurate information and we were sometimes quite surprised by some of the boxes we opened.
Figure 3: KCGIS Diagram, Surface Water Management (SWM), represents the implementation at SWM of Sun systems.
Figure 4: KCGIS Diagram, Department of Development and Environmental Services (DDES), represents the implementation at DDES of HP systems.
We learned several lessons as we went along and improved our own implementation efforts along the way. Some of the things we would change or add if we had to do it all over again are:
Network connectivity is one of the basic assumptions underlying the entire KCGIS program and is vital to its success. The design implemented provides general connectivity while keeping local network traffic isolated to the local environment. We implemented a Frame Relay Network as defined by the KCWAN design team. Using a combination of CISCO routers and U.S. West’s Frame Relay service, we have WAN access available to all KCGIS user sites.
The Frame Relay Service give us the benefits of a high-speed digital private line with the cost effectiveness of shared network resources. Frame Relay Service works by establishing software-defined logical connections, called Permanent Virtual Circuits (PVCs). These circuits act essentially as a dynamic or virtual private line on a public platform. Figure 5: KCGIS Wide Area Network, is a representation of the Frame Network installed.
The KCGIS WAN implementation was a multi-step approach. We followed standard network implementation procedures, such as; network requirements planning, product testing, building wiring, ordering, staging, installation, training and placing sites into production. During a couple of these steps, we learned some significant lessons that we would like to share. The protocol implemented includes primarily TCP/IP, however support for DECNet, Novell, and Appletalk is also available as defined in the requirements.
We chose a 'real' site, which was planned for implementation, as the test site. This 'real' test site gave us challenges that we would not have experienced in a test lab situation. This testing saved us significant time during the roll-out because we knew what issues we would be faced with and we were ready to troubleshoot issues a lot more quickly. The test not only gave us experience with the technology, it also provided insight to the vendor's methods of doing business. We were much better prepared for vendor management during the full roll-out.
The lessons learned during this pilot test were so significant that the time we planned for this was well worth the time spent. Always plan sufficient time for testing and create a test plan which includes all aspects of what you plan to do. We expected testing to take no more than ten work days. Testing went close to four weeks because of unexpected issues encountered.
Network implementations are always very complex. There are building wiring issues, network product configuration issues, protocol issues and now we had frame relay circuit issues. We used the following rules when problem-solving:
Anytime we were unable to connect, we knew which components needed further evaluation. Seems like an easy thing to remember, however, when you are troubleshooting symptoms, it is real easy to just keep going on and on, never really finding out what the real problem is.
Develop installation acceptance test scripts. Our goal was to ensure that the network was not only in place, but that each GIS device could successfully access the WAN. We tested these scripts during the Pilot Test and re-wrote based on lessons learned.
We did not plan or conduct any formal training on frame-relay technology even though this technology was new to the entire team. The network team learned as they went along by spending lots of hours reading manuals and trying new configuration commands during down time. In hindsight, like the 'real' pilot test, taking the time for formal training would have saved some time in the long run.
For all sites participating in the KCGIS Program, we conducted site visits, filling out a pre-defined survey form identifying where the router was going to be placed; where the GIS equipment would be located; as well as any local area networking issues that needed to be resolved prior to our implementation. Throughout the visits, we verified the wiring standard in place. What we learned was that there were two standards in place for building wiring. Some of the buildings used EIA/TIA 568A wiring and others used EIA/TIA 568B. We learned the hard way not to mix two specifications within a site.
Our plan included for the network team to visit the site once for installation. We configured the routers in our test lab, and delivered them once the circuits were installed at the site. We wanted to make sure that all implementations tasks were completed and that everything we needed to complete the install was ready for the site visit.
We learned that we needed to plan well in advance for ordering the Frame Relay circuits. From the time of order to installation, plan at least thirty days into the schedule even if the vendor tells you otherwise. We learned to request a due date, one to two weeks earlier than needed, especially at critical sites.
Insist on frame relay vendor participation at weekly status meetings. The service provider is a key member of the team so ensure the local vendor resources feel they are a part of the success. It is amazing to see how much faster the doors open when critical issues arise, when they feel they are part of the team.
Evaluate your test equipment needs. Buy or lease your own network testing equipment, as determined, for the duration. Don't compete for this equipment with internal production network support staff. Production needs will always have priority over development efforts.
We decided on at least a two person team per site install. When issues came up, it was easier to discuss with the team member potential solutions, and issues seemed to get worked more quickly this way. Plan the teams to have different expertise, if possible. Someone with wiring skills with someone strong skills on router implementations ensures you have all bases covered at the sites during the install.
We worked very closely with our site contacts. Because this WAN was specific to GIS users, we had identified a resource at the site who arranged for access to the site, computing equipment and the LAN environment or LAN Administrator when needed. We kept these people in the loop at all times. Dissemination of information to the sites was through this resource.
Placing sites into Production was a formal milestone task. One recommendation is to define what 'done' and/or 'in-production' means. We established that it meant that the end-user could access GIS data from their approved workstation of choice over the WAN. Until this could be tested, we were not 'done' or 'in-production'.
Implementation can be one of the most exciting parts of the project. Generally, people like to get new products and technology that will help them do their job easier. By putting in the WAN early, GIS technicians now send data to one another, continue analysis and then pass it on to the next agency for work. It can finally end up on a plotter that is located across town and no one had to get into their car and deliver a tape and/or diskette to get it there. The infrastructure is in place and the agencies can share system resources because of the standards followed.
We have not completed the data conversion effort yet, so full access to the core data library is not available, however, people are utilizing the WAN to make other aspects of their job easier and pulling whatever information is available now. We are also working out the procedures for data access and this approach gives us the opportunity to try these procedures out prior to full implementation.
The functional model of an Enterprise-Wide GIS is reflected in the structure of the underlying infrastructure. the KCGIS is being deployed as a central data storage and management system with applications for departmental business functions distributed at various sites.
The roll-out of the systems and network has gone very smoothly in spite of the many lessons learned, including:
Hopefully the information presented will help you as you plan your GIS environment.
King County Coordinated Geographic Information System Scoping Document, August 1993. Prepared by the Municipality of Metropolitan Seattle in consultation with the following: King county Computer and Communications Services Division; King County Council Staff; Department of Public Works; Department of Assessments; Department of Parks, Planning and Resources; King County Office of Financial Management; Department of Development and Environmental Services.
Prepared by - KC CCS/Metro and Interacif Communications Solutions:
Site Enumeration and Classification, KCWAN Design Project - Phase 1; February 24, 1994.
Architectural Alternatives, KCWAN Design Project - Phase 2; March 28, 1994.
Architectural Recommendations, KCWAN Design Project - Phase 3; April 28, 1994.
Implementation Strategies, KCWAN Design Project - Phase 4; April 29, 1994.
Ralph M. Seifert