As a local government The City of Kissimmee had to ask the age-old question "Do we go with the unproven new technology or stay with old reliable for a little longer?" This paper presents the advantages and disadvantages of the migration from the old coverage and shapefile model to the new geodatabase model. The presentation also looks into the specific successes and challenges that the City of Kissimmee has had with the migration. The presentation is focused on the challenges, benefits, and hardships of migrating to the new technology, starting with the use of personal geodatabase and finally moving on to ArcSDE and SQL Server as the geodatabase.
The City of Kissimmee is a city of around 50,000 people in Central Florida about 15 minutes from Disney and 30 minutes from Orlando. Our population can double in size during peak tourist season. We have been using GIS since 1991 but over the past 3 years our GIS has reached a crucial maturity. It is used in almost all the departments in some form or another. Community Development uses it for things like zoning, land use and code enforcement, public works uses it for things like storm water, sanitation routes and road projects, our water utility uses it for asset management of water, sewer and reuse infrastructure, our IT department uses it for data and phone port mapping of city hall and fiber optic routes, and the City Commission uses it to control and direct our aggressive annexation strategy. Every department has asked for a map at one point or another. Our main GIS staff is about 15 people with 4 being officially GIS positions. They are responsible for creating and maintaining datasets. We also have numerous other end users that look-at, query, and make maps from the data. Other then AutoCAD we are a total Esri shop. We use ArcInfo, ArcEditor, ArcView 8.x, ArcView 3.x, and ArcIMS, ArcExplorer and MapObject based Applications.
The City of Kissimmee is always at the leading edge of technology and we wanted to be no different with our GIS system. We have slowly been migrating over to ArcGIS since 8.0.x came about 2 years ago and are continuing to migrate. Most of our major GIS users (about 15) are already over to the ArcGIS. A lot of people still use ArcView 3.x and probably will for a while. We are also starting to phase out 3.x for the very basic user by developing department specific ArcIMS applications.
As I started writing this paper I realized my topic had to be expanded from talking about data formats to talking about software packages and the data formats they use. To fully understand why we decided to switch, from shapefiles and coverages on a file server with ArcView 3.x and ArcInfo 7.x to ArcGIS 8.x and the geodatabase, I had to discuss both data and software considerations. The ArcView shapefile and ArcInfo coverage models have been around for a long time and have served the GIS community well. The users know the structure, advantages and disadvantages of these data types and have learned to use them productively and successfully.
As you all know already ArcInfo Workstation is a very complex program that is mostly command line driven and used an Esri proprietary language called AML to customize and automate. ArcInfo workstation has enormous capabilities and power. It can do everything from annotation to data creation to sophisticated analysis, but is to difficult to learn for an end user. ArcInfo used a data format called the coverage. The coverage’s format consisted of 2 directories; the coverage name directory, which contained all the point and line work and an INFO directory, which contained all the tabular data. This is not an ideal situation. If you moved or deleted a folder outside of ArcInfo without the other one the geographic data was useless. You either had to really know what you were doing or be sure to only move or delete items from within ArcInfo. Also INFO is a very difficult flat file database program to use. It was one of the main database programs in the past but it soon become outdated with the arrival of RDBMS such as Access, SQL Server, Oracle, etc. ArcInfo was for the very high end user. Command line ArcInfo had a very steep learning curve.
A few years after ArcInfo was released; ArcView and the shapefile entered the market. ArcView was a simple and quite intuitive data viewer that could read the coverage and create simple maps. This initially was useful but wasn’t widely used because it didn’t have much functionality. Subsequent releases added functionality and ArcView has become the most widely used GIS program in the world. It can do a lot of things but there are still limitations in what it could do. With the ease comes the loss of functionality. Topology, annotation, complex analysis, and high level data conversion were no longer available. ArcView’s main data format is the shapefile. Even though you lost some functionality, ArcView with its user friendly GUI and the shapefile made GIS available to more people. The same problem that existed with the coverage also existed with the shapefile. There are multiple files associated with one shapefile. To use the shape file you need at least 3 files (shp, shx, and dbf). There are also many others that can be associated with that file. All this means is that to share data you had to send all three of the files or the geographic data was unusable (you could still view the tabular data with other programs such as Access or Excel).
There had to be something created that could combine the functionality of the coverage, the ease of use of the shapefile and have the ability to be customized for future needs so Esri developed the geodatabase. The geodatabase is a complex format stored in Microsoft Access or any other RDMS out there such as Oracle, Microsoft SQL Server, and Informix etc. The beauty of the Geodatabase is the user doesn’t need to know the underlying table structure to use it. The user just adds, creates or modifies the data like they used to. This makes the transition from ArcView very easy. The transition is easy because the similar GUI and also the addition of common windows functions such as the “right click”. All the while an administrator can do all the tweaking and background work. To transfer a Geodatabase all you have to do is give a single Access database file to another user. There is no need to zip multiple files and folders any more. All your GIS data can be stored in one database (there is a size limitation in a personal geodatabase). Switching to the geodatabase is allowing us to add in relationships, behaviors and properties. These are true advantages; the one file is a mere product of the data structure. Finally the geodatabase is an object unlike the coverage and shapefile which are flat files. This will help us to model real-world things better.
There are many advantages and disadvantages to consider when thinking about migrating over to the ArcGIS geodatabase model but before you even think about that in a municipal setting you have to start thinking about the big picture how will this affect the city and how far into the future will this product take you before you have to start all over. We also realized that there were not many alternatives. We knew ArcView was going to be phased out so we decided it would be better to switch sooner rather then later. The major advantages of switching are improved data integrity, improved performance, and easier training. Other reasons for switching are the ability to create feature-linked annotation and better editing tools. There are many other advantages that will be listed below. Some of the disadvantages are data conversion, cost, newness and user reaction. Also listed below are other disadvantages. Other then the few major disadvantages there have been a few complications and learning experiences that have arisen with the switch.
One of the biggest advantages for switching is the availability of domains. Domains help with the major problems data integrity. The domain allows us to set up codes or ranges that prevent the user from adding incorrect values. They can only add what is in the domain. For example, if a user is adding in a building on Main street, when trying to add attributes, the user can only select Main from the drop down list, not Maine or some other variation. This prevents typographical errors and keeps the administrators from correcting numerous mistakes.
Secondly, ArcGIS is making it easier for us to train users in one common program. Before ArcGIS we had to train in both ArcInfo and ArcView 3.x. With ArcGIS we can teach the basics to everyone at the same time. As others need more functionality they don’t need other software, just further training.
Thirdly, the geodatabase shows a large performance increase over the shapefile and coverage when being loaded into ArcMap. I have run a few tests at different times of the day over our network. You can tell that there is a performance increase during the middle of the day because of SQL Servers internal processes such as caching. This performance increase is only noticeable when using a multi-user geodatabase not with an Access personal geodatabase. A personal geodatabase does load faster then a shapefile when it is loaded locally and about the same when it is loaded by multiple people over a network.
Fourthly, another strong reason for switching to the geodatabase as our main data creation/editing tool is the ability to create annotation. Yes, we all know that it is not the greatest in its ease of use and it missing some very basic functionality but it is functional and is a lot easier to use than coverage annotation. The one feature that is missing that is causing us the most problems is you cannot change the color of the annotation without editing.
Finally, the editing tools are far superior in ArcGIS than in both ArcInfo and ArcView. There is a shared edit tool, CAD-like drafting tools, and numerous options on a right click, such as the ability to type in distances and angles. With the amount of data the City has to update and maintain these new features were extremely valuable. Even though all of the data formats can be edited in ArcGIS the geodatabase works the best and has the best performance. Also, it is very simple and almost seamless to convert from a feature class to a shapefile for the 3.x users and for data transfer. The shapefile for the time being is still the standard for file sharing.
Some other advantages to think about when decide to switch are the availability of transparent rasters, the ability to use 3rd party tools and applications that use the geodatabase, and its easier to view CAD drawings in ArcGIS. Also there is no need to duplicate data for ArcIMS, you can project on the fly as long as there is a projection file, and easier data backup due to the use of SQL Server.
One of the biggest disadvantages of switching to the geodatabase model was the need for conversion of all of our datasets. This conversion issue turned out not to be a major problem. We realized that we didn’t have to convert everything all at once because ArcGIS can read everything. We have only converted things that we update in house. Everything that comes from an external source is mostly left in its original format and then is added as time allows.
Another major disadvantage is cost. Switching over to the ArcGIS platform is a lot more expensive at the beginning. You have to purchase the upgrades to 8.x, buy SDE, SQL Server, and initial training. You either have to have a DBA on staff or contract out one for initial set up and maintenance. And finally you have to have a maintenance contract on the software for the upgrades since the product is upgraded frequently.
The last major disadvantage is newness. First it is going to be harder to find experienced new hires and users. There are a lot of known bugs and functionality issues that are constantly being fixed and there new ones found frequently. There are not a lot of local user groups to get advice/help from. Finally it is unproven there is no way to really tell if this is going to work out in the long run because not many entities have shown this to actually work. So there will be a lot of trial and error and kinks to work out.
One other thing to consider is user reaction. There are going to be people that don’t like change. They have been using the old technology for a long time and know what they are doing with it. They do not want to be bothered with learning something new. The biggest difference users will see is the change in terminology. Also switching platforms frustrates users when trying to spread the use of GIS especially if someone has just been starting out.
There have not been any major issues with switching over to the geodatabase just a few learning experiences. This includes what is the best way to set up a geodatabase and if it should be customized or out of the box. Also before going to a multi-user geodatabase there were personal geodatabase issues. Finally custom scripting is a lot more difficult then it was in ArcView 3.x and ArcInfo 7.x
The first issue is what is the best way to set up a geodatabase. Should the data be in a stand alone feature class or in a feature dataset? Should we use subtypes or domains? Should we do any customizations or should we use the default setup. The first database I created was for our Planning Department. I created a database for zoning, but misused subtypes and it caused major problems when trying to update. I was under the impression that a subtype was a coded value using numbers instead of text codes. This is untrue. I have learned the proper way to use subtypes is to have a feature class (theme) and divide them up by type. Example: You have a feature class with valves and there are 10 different types of valves with different attributes associated with them. You would use a subtype on that feature to symbolize and attribute each of the valve types. Even though subtypes did basically what I needed it did not work well for updating. If I wanted to do any of the geoprocesses which create new feature classes I lost the subtype and had to recreate it every time. So I went out to look for a better way to do this. For my purpose I should have used a coded value domain. With a coded value domain the user can only add an attribute that is in the domain. This helps with data base integrity. For example, our City’s zoning has 31 different codes. The coded value allows us to change the zoning by choosing from a list rather than typing in a value. The user cannot mistakenly type RA1 instead of RA-1 which would prevent incorrect query results.
Secondly, another issue we have been dealing with is the customization of geodatabases. For one of our applications, storm water, we want the date that a feature is added and the date that it is modified. I found a dll in the ArcObject Developers Samples that allows us to do this. The only problem is every time the users get a new machine or a new version of ArcGIS installed, the dll is unregistered and the geodatabase is useless until it is reregistered. The easiest way to fix this for me was put the dll on a network drive and register it from there that way when the system is changed it is always in the same place and easy to reregister rather then going to all the users’ machines.
Thirdly, another issue we are currently having is the single user geodatabase is not very good if many people are trying to view it. There is a significant performance hit. Also like with the shapefile if one person is editing it no one else can use it. We are currently in the process of switching to a city wide ArcSDE with SQL Server multi-user geodatabase with versioning to alleviate this problem. I figure that after the major switch to a multi-user geodatabase there will be other issues and complications to deal with and conquer. We have just recently started using the multi-user geodatabase and have not had much time to find many issues yet. Over the next few months we hope to see that everything we have done will be beneficial and major problem free.
Finally, the only other issue is with the complexity of the Object model. There are many more customization possibilities using VB/VBA, but if you are not a full programmer it is very difficult to learn and understand. Unlike with Avenue and AML, which are top down programming languages, VB/VBA are object oriented and it takes a lot more time to learn and create projects. To get through this there is a lot more help out there than for Avenue or AML. There is a very good instructor led Intro to VBA class. It helped a lot with the basics. The 2 volume set Exploring ArcObjects is a very good reference manual, and ArcObjects on line and the developer’s samples are great tools to get you started.
Overall, the switch from old to new has been interesting and beneficial. The users enjoy the new technology because it makes their job easier and more productive. The users like GUI, the “Right-Click” functionality, and cartographic tools. I, as the GIS Specialist and tester of the new technology, look forward to moving on with the ongoing transfer from old to new and am sure it will benefit the City because of all the available and yet to come features of this amazing new data format. Based on the advantages and disadvantages listed above and the alternative to not migrating (which is being left in the dust) we are satisfied that it was the appropriate decision to make.