Geographic Data Technology, Inc. (GDT) is transitioning the way we gather information for our street-centerline database. We are moving away from traditional phone solicitation for map resources to data sharing arrangements with local data producers who have a vested interest in maintaining up-to-date street centerline database and associated attribution. The Internet is the obvious vehicle for sharing data between local agencies \ companies and Geographic Data Technology. Currently we have built an Internet mapping application which allows users to view our data on-line and to inform GDT where that data needs improvement or additions. The site also has a download capability so that the users can download the latest available data anytime in virtually any format. The system utilizes ArcSDE™, ArcIMS™, and Safe Software's Spatial Direct™ as the backbone. This paper describes this system and the direction that we are taking it.
Geocoding and Location Base Services are becoming increasingly important applications that utilize comprehensive street centerline databases. Examples of these applications are cell phone 911 routing and in-car navigation. These applications require highly accurate data to support them in terms of geographic accuracy, full attribution and overall completeness. The geographic accuracy and the completeness of street attribution for GDT’s street centerline database is improving rapidly and there will come a time very soon, when the bulk of GDT’s maintenance efforts will consist of adding new data that simply did not exist before (i.e. new subdivision, 911 related re-addressing, etc). It will become increasingly important to acquire and disseminate this new information as rapidly as possible because, as in the case of 911, lives depend on it. Likewise, commerce depends on it. The delivery person has to know where your new house is to deliver your new washing machine. The insurance agent needs to know where you live so he/she can quote you a rate.
Our traditional methods of resource gathering (primarily phone based solicitation) and data dissemination (primarily CDs) will become obsolete very soon, because we simply won’t be able to get the data, our clients need, to them fast enough.
We have built an Internet editing and data downloading site that is the first installment of a distributed data sharing plan. The system consists largely of commercial applications which have been customized and integrated together to facilitate the sharing of digital street data. The site consists of:
In the future we intend to provide the following:
Data Sharing Benefits and Model
Building and maintaining a street centerline database that covers entire countries and yet keeps pace with rapid development is an ominous task. GDT has embarked on a strategy of data sharing with companies and governmental entities where GDT acquires information from these local experts that we then incorporate into our countrywide databases. GDT in-turn provides these partners with regional data in a consistent format, that has undergone rigorous quality control. GDT also provides a means to disseminate the shared data to partners in virtually any GIS format.
GDT has two Internet data sharing programs which facilitate data sharing and collaboration:
Figure 1. Community Update homepage: http:// www.communityupdate.com.
Figure 2. Online Enhancement Request System Homepage: //126.96.36.199/onlineER/index.cfm.
To date we have over 700 government and private industry groups participating in either Community Update or the Online Enhancement Request system. We receive in excess of one thousand online edit suggestions per week in addition to numerous conflatable datasources. Partners have made over 1200 downloads of data for their areas of interest since the inception of the Community Update program.
Components of Data Sharing
The main components of GDT internet data sharing applications are:
These are described below. These functions are handled by integrating a variety of commercial applications and custom software.
Internet Map Editing
Figure 3. Online Editing Page for Community Update.
Editors are allowed to input suggested edits (shown in purple in Figure 3) in the form of Adds, Delete, Update and General Problems. The adds, deletes and updates are specific to the street layer. For adds, a user can submit multiple mouse clicks to digitize new streets. Once done, a window pops up with a form for filling in appropriate attribution and metadata. For deletes and updates, the user selects a street segment that then is highlighted (by rendering in a transparent layer facilitated by ArcIMS™). Attribution pops up in a separate window. In the case of a delete, the user simply verifies that the correct segment has been selected. For modifies, the user is presented with the attribution which can then be modified.
The general data problem is for all other categories of edits. It requires clicking a point on the map. This is followed by a questionerre requesting specific information about the problem and provides space for a narrative relative to the problem.
Once a series of edits are performed, the user can then save their work, which is stored in a separate SDE layer. The edit layer is visible to all users so they can be kept informed of pending changes to the database.
Primarily for our Community Update, subscribers we have a data download facility which allows all of GDT's data for their local areas to be downloaded in virtually any GIS format. The user selects from a form(Figure 4), desired layers to download, the GIS format and the projection. The area to download can be selected from an ArcIMS generated index map. This information is transmitted to our Spatial Direct™ server (Safe Software) where data from GDT's SDE™ database is parsed and format converted (using FME™ from Safe Software) and compressed. Once the process is complete the user is automatically e-mailed the location of the compressed file for download via Internet browser utilities.
Figure 4. Download Page for Community Update.
Upload Capabilities for Acquisition of Digital Data Resources
Many of our data partners have very sophisticated GIS capabilities and may have attribution and/or geography that already exists in their databases that does not exist in ours. We use a complex conflation process (that is beyond scope of this paper) to assimilate this information. However, to acquire this information we have an online utility that walks our partners through a metadata entry form (Figure 5) that spells out the information we need before we can make use of their data. Generic instructions for using File Transfer Protocol (FTP) are provided. After weeks of experimentation, FTP was concluded to be the most robust exchange mechanism for receiving data. It provides an indication that transfer is actually taking place, it can handle larger files and is simply more robust and universally used than other automated file transfer mechanisms used on the Internet.
Figure 5. MetaData Page for Uploaded Data.
We also have partners that, for various reasons, choose to edit off-line. In this case, they download data in whatever GIS format they choose, edit in an environment they are comfortable with, then they upload either the changes they have made, or the entire edited data set. If the entire data set is sent, we determine the additions, deletions and changes they have made to the data by comparing it to an archive copy of the data they had previously downloaded. This comparison occurs in the same format that the data was edited and is facilitated by FME™ by Safe Software.
Some of our Online Enhancement or Community Update partners that have large organizations or affiliates require the capability to manager multiple users. The software provides a mechanism that allows these client managers to create and maintain accounts for their affiliates. These managers can provide client users selected users privledges equivalent to their own or selectively restricted. They can also provide their users with access to a geographic area equivalent to their own or any subset within their bounds. For example a county government could provide a city within its bounds, access to data for that city only.
Figure 6. Managing Users in Community Update.
User Feedback Loop
As was stated earlier, many of our clients require information they provide to GDT, be turned around and incorporated into the core working database as rapidly as possible. So that clients can monitor the progress of this process, part of Community Update and the Online Enhancement Request System is the ability to track the status of edits made in the online editor. User can query GDT's tracking database based on an edit id or simply list the status of all edits that have been made. Status indicators let end users know whether their edits are under review; going through the quality control process; or, in some cases, if edits were rejected; and, finally when edits have been incorporated into the core database.
Figure 7. User Feedback Page.
The three main challenges for this system are:
Managing Customer Expectations
GDT is in the business of selling a street centerline database. If a customer is willing to take the time to tell us how to improve our data, then they logically expect to be rewarded with seeing the information they provide incorporated into the database in a timely fashion. Fulfilling that need in all instances is difficult: 1) due to the wide range of customer applications and what is required for the app; 2) due to the wide range of quality in information we receive; and 3) our scheduling of finite human resources to perform the editing.
Customer applications, and therefore elements of data that are important to them, vary tremendously. For example, one call centers (utility dispatchers) want the street database as inclusive as possible, right down to streets that have not even been constructed yet. They are in the business of routing installers to construction sites. Cellular 911 responders however, require positional accuracy above all else, since a GPS point must fall on a road for calls made from a car.
We receive information from our clients that encompass everything from highly accurate detail plat maps of new subdivisions, to a simple notes stating that data is missing or incorrect, but with no indication of what is missing or wrong. Because we have a finite staff, we rank the incoming information based on:
Entities that participate in GDT data sharing programs, run the gambit from private individual who are being helpful to commercial client who develop 911 applications and need the information they provide back in their hands as part of a database ASAP. Any information that comes in is assigned a rating based on the above factors. We intially verify all incoming information with independent sources. Once a client shows that they provide consistent, high quality data, then we assign them a higher rating and can assimilate the information with fewer quality control measures.
Customers do pay us to update our database for their needs. In these instances the customer has determined that it is more economical for GDT to maintain up-to-date databases for them rather than attempting to do it all themselves. Many telecommunications and utility companies participate in the OnlineEnhancement Request Program.
At the other end of the spectrum, where data coming in has sparse information , we may wait to process an area, until the volume is sufficient to warrant working it or until we come around to that area in our standard rotation. In these instances we will need to acquire further map resources to upgrade the area in question. It is far more efficient to systematically work an area than going through individual requests at random.
Overall quality of our database (i.e. positional accuracy and completeness of attribution) is continually improving. There will come a point when most of our editing staff will devote their time to managing change and incoming request for changes will always be handled immediately
GDT uses propietary software for editing internally and likewise the database is stored with a unique layout for optimal use with our software. Consequently, it is necessary to continually sync the data exposed to our data sharing partners with the data that is actively being edited internally at GDT. This requires that we extract data from our internal editing database and format it appropriately for SDE™. Since we make many ten-of-thousands edits every week, the closer these two databases are in sync the better off we are. In many instances, our partners will notice a problem in their own application dataset, however they will observe that a correction has already been made in the online data set due to its more recent currency. By exposing the most recent data, our customers will be editing what our internal editors see, and will not be sending edits based on obsolete information.
Currently we are refreshing the data once a month, however, we intend to start doing this on a daily basis in the future.
As alluded to above, when clients start sharing data with us, we are very vigilant with QC measures to make sure that the information coming in is valid and more current than what resides in the core database. Initially no suggested change is actually input into our database until we can verify the change independently through another source. Once we have established a working relationship, this rule may no longer apply. In many cases there simply is no other way to validate the information because it is simply too new and the authority we are dealing with either was responsible for example, building the new road, or got the information first hand from the responsible authority. Remember participants in our various programs NEED data to be as up-to-date and accurate as possible and have a vested interest in seeing the best data possible.
GDT is in the business of selling data. Because we maintain a single comprehensive database, our data sharing partners all sign agreements that state that the information sent to GDT either belongs to them or they have permission to redistribute it for commercial use.
In the opinion of the author, GDT is transitioning from a street centerline data PROVIDER to a data broker, where our value-add will be in assimilation of local data sets into a coherent nationwide data set that rigidly adheres to published standards. In essence we already are data brokers, however, the data that we assimilate still comes in largely on paper and requires much handwork to incorporate into our database. This will change when in-essence the GDT database will be a superset of our client's and partner's data with identical currency.
In order to get the latest data in our data sharing partners hands, in the future we will:
What will change in the future is that:
GDT has embarked on an Internet data sharing partnership with both government and private industry. The intent is to provide value to all partners: Government and Industry will be provided with data that adheres to rigid nationwide standards and quality control and that is the most up-to-date possible. GDT will also facilitate data access, distribution and storage via the Internet. GDT will gain access to map information from the sources that create the information with can then be incorporated into nationwide commercial products. Long term success of this endeavor hinges on the ability to rapidly transfer and synchronize information between GDT and it partners.
About the AuthorClayton Morlock has been a Senior Software Engineer and Project Manager for Geographic Technology, Inc. for the last 4 years. He has worked with Geographic Information systems for the last 15 years and has been building Internet mapping applications for 3 years. He has experience in Digital Road Networks, Environmental, Oil, Mining and Defense GIS applications. He holds a BS and MS in Geotechnical Engineering.