Internet Street Map Editing Maintenance Using ArcIMS and ArcSDE

Geocoding and Location Base Services are becoming increasingly important applications that utilize comprehensive street centerline databases. Examples of these applications are cell phone 911 routing and in-car navigation. These applications require highly accurate data to support them in terms of geographic accuracy, full attribution and overall completeness. The geographic accuracy and the completeness of street attribution for GDT’s street centerline database is improving rapidly and there will come a time very soon, when the bulk of GDT’s maintenance efforts will consist of adding new data that simply did not exist before (i.e. new subdivision, 911 related re-addressing, etc). It will become increasingly important to acquire and disseminate this new information as rapidly as possible because, as in the case of 911, lives depend on it. Likewise, commerce depends on it. The delivery person has to know where your new house is to deliver your new washing machine. The insurance agent needs to know where you live so he/she can quote you a rate.

Our traditional methods of resource gathering (primarily phone based solicitation) and data dissemination (primarily CDs) will become obsolete very soon, because we simply won’t be able to get the data, our clients need, to them fast enough.

We have built an Internet editing and data downloading site that is the first installment of a distributed data sharing plan. The system consists largely of commercial applications which have been customized and integrated together to facilitate the sharing of digital street data. The site consists of:

• Online editing utilizing map images generation by Esri’s ArcIMS™ and ArcSDE™ as the data source
• Data download capabilities using Safe Software’s Spatial Direct™ and FME™
• Sophisticated customer management and security
• Client feedback concerning specific recommended edits
• Upload capabilities for data incoming to GDT to facilitate acquisition of digital data resources

In the future we intend to provide the following:

• Online transactional updates to GIS systems built on Relational Database Management Systems
• Change management logging (allowing customers to maintain relationships with our data to private data sets.
• Hosting and maintaining private attribution associated with GDT’s street centerline data (i.e. rating territories, delivery areas)
• Handheld device (PDA and Laptop) map editing

Building and maintaining a street centerline database that covers entire countries and yet keeps pace with rapid development is an ominous task. GDT has embarked on a strategy of data sharing with companies and governmental entities where GDT acquires information from these local experts that we then incorporate into our countrywide databases. GDT in-turn provides these partners with regional data in a consistent format, that has undergone rigorous quality control. GDT also provides a means to disseminate the shared data to partners in virtually any GIS format.

GDT has two Internet data sharing programs which facilitate data sharing and collaboration:

• Community Update – a public/private partnership with local and regional governmental agencies. GDT provides our comprehensive street, transportation and boundary layers to these agencies via internet download in virtually any GIS format and map projection. In return, the agencies share their local knowledge by uploading datasets and/or identification of required additions or alterations necessary to our database via online editing.
• Online Enhancement – an application for our commercial customers who’s application require extremely up-to-date information. It is, however, more economical for them to utilized GDT to compile and format the database rather than maintaining it in house and for this reason, they are willing to assist us with information acquisition for optimal performance of their applications.

The main components of GDT internet data sharing applications are:

• Internet map editing
• GIS data downloading to partners
• Application user management
• Customer feedback
• Data upload to GDT

These are described below. These functions are handled by integrating a variety of commercial applications and custom software.

GDT maintains an Esri ArcSDE™ database consisting of our complete street centerline database for the United States and Canada to be used for Internet editing applications. Other layers consist of political boundaries, water, rail and points of interest. The street centerline database contains a complete suite of attributes necessary for both geocoding and location based services applications. The data is presented to the user for editing purposes utilizing ArcIMS™ (the image server) with the Cold Fusion (a internet server which enhances HTML with a variety of custom tags for database applications). Incorporated with map displays generated by ArcIMS™ are a proprietary editing system written with Cold Fusion and Javascript which allows users to make suggested edits to the street layer. We use our internal Matchmaker SDK™ for geocoding.

The general data problem is for all other categories of edits. It requires clicking a point on the map. This is followed by a questionerre requesting specific information about the problem and provides space for a narrative relative to the problem.

Once a series of edits are performed, the user can then save their work, which is stored in a separate SDE layer. The edit layer is visible to all users so they can be kept informed of pending changes to the database.

Primarily for our Community Update, subscribers we have a data download facility which allows all of GDT's data for their local areas to be downloaded in virtually any GIS format. The user selects from a form(Figure 4), desired layers to download, the GIS format and the projection. The area to download can be selected from an ArcIMS generated index map. This information is transmitted to our Spatial Direct™ server (Safe Software) where data from GDT's SDE™ database is parsed and format converted (using FME™ from Safe Software) and compressed. Once the process is complete the user is automatically e-mailed the location of the compressed file for download via Internet browser utilities.

Many of our data partners have very sophisticated GIS capabilities and may have attribution and/or geography that already exists in their databases that does not exist in ours. We use a complex conflation process (that is beyond scope of this paper) to assimilate this information. However, to acquire this information we have an online utility that walks our partners through a metadata entry form (Figure 5) that spells out the information we need before we can make use of their data. Generic instructions for using File Transfer Protocol (FTP) are provided. After weeks of experimentation, FTP was concluded to be the most robust exchange mechanism for receiving data. It provides an indication that transfer is actually taking place, it can handle larger files and is simply more robust and universally used than other automated file transfer mechanisms used on the Internet.

Some of our Online Enhancement or Community Update partners that have large organizations or affiliates require the capability to manager multiple users. The software provides a mechanism that allows these client managers to create and maintain accounts for their affiliates. These managers can provide client users selected users privledges equivalent to their own or selectively restricted. They can also provide their users with access to a geographic area equivalent to their own or any subset within their bounds. For example a county government could provide a city within its bounds, access to data for that city only.

As was stated earlier, many of our clients require information they provide to GDT, be turned around and incorporated into the core working database as rapidly as possible. So that clients can monitor the progress of this process, part of Community Update and the Online Enhancement Request System is the ability to track the status of edits made in the online editor. User can query GDT's tracking database based on an edit id or simply list the status of all edits that have been made. Status indicators let end users know whether their edits are under review; going through the quality control process; or, in some cases, if edits were rejected; and, finally when edits have been incorporated into the core database.

The three main challenges for this system are:

• Managing customer expectations: Maintaining acceptable turnaround from entry of edits to incorporation into the database
• Maintaining data currency
• Quality control: Assimilation of widely disparaging data into a coherent and accurate database

GDT is in the business of selling a street centerline database. If a customer is willing to take the time to tell us how to improve our data, then they logically expect to be rewarded with seeing the information they provide incorporated into the database in a timely fashion. Fulfilling that need in all instances is difficult: 1) due to the wide range of customer applications and what is required for the app; 2) due to the wide range of quality in information we receive; and 3) our scheduling of finite human resources to perform the editing.

Customer applications, and therefore elements of data that are important to them, vary tremendously. For example, one call centers (utility dispatchers) want the street database as inclusive as possible, right down to streets that have not even been constructed yet. They are in the business of routing installers to construction sites. Cellular 911 responders however, require positional accuracy above all else, since a GPS point must fall on a road for calls made from a car.

We receive information from our clients that encompass everything from highly accurate detail plat maps of new subdivisions, to a simple notes stating that data is missing or incorrect, but with no indication of what is missing or wrong. Because we have a finite staff, we rank the incoming information based on:

• customer priority - is the customer paying us to priortize their request?
• accuracy and completeness of incoming information;
• our ability to assimilate the information (usually related to the format of incoming data)
• location within focus areas: Does the data lie in areas that GDT has targeted for immediate attention?

Entities that participate in GDT data sharing programs, run the gambit from private individual who are being helpful to commercial client who develop 911 applications and need the information they provide back in their hands as part of a database ASAP. Any information that comes in is assigned a rating based on the above factors. We intially verify all incoming information with independent sources. Once a client shows that they provide consistent, high quality data, then we assign them a higher rating and can assimilate the information with fewer quality control measures.

Customers do pay us to update our database for their needs. In these instances the customer has determined that it is more economical for GDT to maintain up-to-date databases for them rather than attempting to do it all themselves. Many telecommunications and utility companies participate in the OnlineEnhancement Request Program.

At the other end of the spectrum, where data coming in has sparse information , we may wait to process an area, until the volume is sufficient to warrant working it or until we come around to that area in our standard rotation. In these instances we will need to acquire further map resources to upgrade the area in question. It is far more efficient to systematically work an area than going through individual requests at random.

Overall quality of our database (i.e. positional accuracy and completeness of attribution) is continually improving. There will come a point when most of our editing staff will devote their time to managing change and incoming request for changes will always be handled immediately

GDT uses propietary software for editing internally and likewise the database is stored with a unique layout for optimal use with our software. Consequently, it is necessary to continually sync the data exposed to our data sharing partners with the data that is actively being edited internally at GDT. This requires that we extract data from our internal editing database and format it appropriately for SDE™. Since we make many ten-of-thousands edits every week, the closer these two databases are in sync the better off we are. In many instances, our partners will notice a problem in their own application dataset, however they will observe that a correction has already been made in the online data set due to its more recent currency. By exposing the most recent data, our customers will be editing what our internal editors see, and will not be sending edits based on obsolete information.

Currently we are refreshing the data once a month, however, we intend to start doing this on a daily basis in the future.

As alluded to above, when clients start sharing data with us, we are very vigilant with QC measures to make sure that the information coming in is valid and more current than what resides in the core database. Initially no suggested change is actually input into our database until we can verify the change independently through another source. Once we have established a working relationship, this rule may no longer apply. In many cases there simply is no other way to validate the information because it is simply too new and the authority we are dealing with either was responsible for example, building the new road, or got the information first hand from the responsible authority. Remember participants in our various programs NEED data to be as up-to-date and accurate as possible and have a vested interest in seeing the best data possible.

GDT is in the business of selling data. Because we maintain a single comprehensive database, our data sharing partners all sign agreements that state that the information sent to GDT either belongs to them or they have permission to redistribute it for commercial use.

In the opinion of the author, GDT is transitioning from a street centerline data PROVIDER to a data broker, where our value-add will be in assimilation of local data sets into a coherent nationwide data set that rigidly adheres to published standards. In essence we already are data brokers, however, the data that we assimilate still comes in largely on paper and requires much handwork to incorporate into our database. This will change when in-essence the GDT database will be a superset of our client's and partner's data with identical currency.

In order to get the latest data in our data sharing partners hands, in the future we will:

• Transfers data to/from GDT to clients will happen by electronic means (over the internet) and will be much more frequent than current once a month, once a quarter or once a year.
• Transfer will be eliminated altogether as client will use and work on a database housed on internet servers at GDT.

What will change in the future is that:

• Data will adhere to strict metadata standards and database rules. For example a street segment will be clearly defined for things such as when and where the segment can begin and end (i.e. ALL STREETS BEGIN AND END AT INTERSECTIONS). Metadata will be specific to both geometry and individual attributes so that a client will know precisely the currency and source of the particular piece of information including a history of precursor information.

GDT has embarked on an Internet data sharing partnership with both government and private industry. The intent is to provide value to all partners: Government and Industry will be provided with data that adheres to rigid nationwide standards and quality control and that is the most up-to-date possible. GDT will also facilitate data access, distribution and storage via the Internet. GDT will gain access to map information from the sources that create the information with can then be incorporated into nationwide commercial products. Long term success of this endeavor hinges on the ability to rapidly transfer and synchronize information between GDT and it partners.