Thomas E. Lonski, SchlumbergerSema
Robert Parsons, SchlumbergerSema
Besides the size of an enterprise-wide Esri implementation, another aspect that sets these implementations apart is the level of systems integration with other customer computer systems. This combination of systems integration and large numbers of users creates special data management issues. While implementing a large Arc8 project, SchlumbergerSema delved deeply into Esri's transaction management and versioning implementation. This paper will present the data management challenges encountered by SchlumbergerSema and the solutions developed to meet these challenges.
The characteristics that set enterprise-wide Esri implementations apart from the typical Esri customer are the large number of users, the amount of data, the volume of data updates, and the integration of Esri with other computer systems. SchlumbergerSema has implemented, or is in the course of implementing, several projects with these characteristics. Our larger projects have from 200 to 800 non-Web Esri users (i.e., ArcGIS, ArcView, and MapObjects), 2 to 80 GB of vector ArcSDE data, 50 to 200 edit users, and 2 to 10 system interfaces. The system interfaces add to the volume of updates generated by the edit users. Updates in external systems are usually propagated to Esri within minutes and each external system can have up to 100 updates per day.
The experiences cited in this paper are primarily from two projects, one for a city government and another for a gas and electric utility. This paper is also based on SchlumbergerSema’s experience with long transaction management and versioning from other GIS vendors. The city government project has over 700 non-Web Esri users, 3 GB of vector ArcSDE data, 65 edit users, and 2 system interfaces. One of the system interfaces is to an asset management/work management system used by the sewer and water departments. The other system interface is to a permitting and addressing system. The utility project has 400 non-Web Esri users, 80 GB of vector ArcSDE data, 90 edit users and seven system interfaces. The seven system interfaces are to asset registry, customer information system, master street index, work management, mobile workforce management, outage management, and system planning.
In the course of implementing these projects, SchlumbergerSema encountered three transaction management and versioning challenges. The first of these challenges is in ArcSDE’s implementation of conflict detection for long transaction management. Figure 1 uses Unified Modeling Language (UML) sequence diagram notation to provide an overview of the process users follow while working with versioned data. The sequence diagram shows both the process steps and the relative timeline. The key steps are that separate transactions (related groups of edits) are performed in separate versions, which have to be reconciled before the data can be posted. If conflicts are detected, the conflicts have to be resolved before the transaction can be posted. Accurate conflict detection is fundamental to an industrial-strength long transaction management solution that can support enterprise-wide implementations.
Figure 1: Versioning Work Process
SchlumbergerSema extensively tested ArcSDE’s implementation of long transaction management. All testing was done using a versioned geodatabase on Oracle 8.1.7 using ArcGIS 8.1.2 and ArcSDE 8.1.2. The test cases are listed below in Table 1. The columns “Edit 1” and “Edit 2” show the kind of edit that was performed on either the same feature or related features as indicated in the “Relationship” column. The “Within” relationship means the edits were performed on the same feature. The “Connected” relationship means the features edited in “Edit 1” and “Edit 2” are in the same geometric network and are connected to one another. The result “CD” means a conflict was detected and “NCD” means that no conflict was detected.
As indicated by the comments in Table 1, SchlumbergerSema found six instances of interesting conflict detection behavior.The first four cases occurred during testing and data conversion with geometric networks. In these four cases, the shape or attribute edits were made to edge features while junction objects were added or deleted. The shape edits to the edge features did not move the end of the edge where the junction object was being added or deleted. For the utility project, these errors occurred when the gas main network was loaded and posted to default. At this point, two versions were created and gas valves were loaded into one version and gas fittings were loaded into another version. While the first version posted without any errors, the second version generated hundreds of conflicts.
Table 1: Conflict Detection Test Cases
|
Edit 1 |
Edit 2 |
Relationship |
Result |
Comment |
|
Shape |
New object |
Connected |
CD |
Misreported as a shape conflict. |
|
Attribute |
CD |
|
||
|
Shape |
Delete object |
Connected |
CD |
Misreported as a shape conflict. |
|
Attribute |
CD |
Misreported as a shape conflict. |
||
|
Shape |
Attribute |
Within |
CD |
This should not be a conflict. |
|
Connected |
NCD |
|||
|
1:1 |
NCD |
|||
|
1:M |
NCD |
|||
|
M:N |
NCD |
|||
|
Shape |
Shape |
Connected |
NCD |
|
|
Attribute |
Attribute |
Within |
CD |
This should not be a conflict when different attributes are edited. |
|
Connected |
NCD |
|||
|
1:1 |
NCD |
|||
|
1:M |
NCD |
|||
|
M:N |
NCD |
|||
|
Shape |
Relationship Class Attribute |
M:N |
NCD |
|
|
Attribute |
Relationship Class Attribute |
M:N |
NCD |
|
|
Shape |
Annotation |
Annotation Class |
CD |
OK |
|
Attribute |
Annotation |
Annotation Class |
NCD |
Part of the difficulty is that the Conflicts window does not list a connectivity property so it cannot accurately report conflicts detected in connectivity. In some cases, the Conflicts window indicated the conflict was with the shape. In other cases, there was no indication of which property caused the conflict as shown in Figure 2. Esri reports that these conflict detection bugs have been fixed in ArcSDE 8.2. SchlumbergerSema has not yet repeated these tests with the newer version of ArcSDE. This example is included in this paper to show that wrinkles are still being worked out in ArcSDE’s conflict detection that may cause false conflict reports. These false conflict reports can be difficult for users to understand.
Figure 2 Conflict Window bug
SchlumbergerSema finds the next two conflict detection cases more interesting because Esri considers this behavior correct and as designed. Since ArcSDE tests for conflicts at the feature level instead of at the property level, it always flags edits on the same feature as conflicts. However, as long as the feature’s identifier (either primary or alternate) is unchanged, edits on different properties in a feature are never conflicts. For example, if a system planner updates the “Summer Load” on a transformer and a maintenance user updates the “Last Inspection Date”, both of these changes are correct and need to be kept. If the feature’s identifier is changed, then what the feature represents in the real world has changed and there could be a conflict. For example, when a designer replaces a transformer (and thus it has a new serial number or other identifier) while a draftsman is updating the transformer’s position from field redlines. In this case, the designer and draftsman need to be aware of each other’s changes. In SchlumbergerSema’s experience however, identifiers are rarely modified once a feature is created. The vast majority of edits on a feature are changes to the feature’s other attributes rather than the identifier. Therefore, most edits to the same feature should not cause a conflict. When there is no change to the identifier, the only time that separate edits on the same feature should be reported as a conflict is if multiple versions are editing the same attribute simultaneously or editing the shape simultaneously.
This becomes more than a theoretical issue for large Esri implementations. The large number of users means that users in different organizations are constantly editing different aspects of the same features. Likewise, the system interfaces are continuously updating features at the same time users are editing other aspects of those features. Some examples of these interactions are:
These interactions combined with ArcSDE’s implementation of conflict detection will make conflicts a common occurrence. Further, the majority of these conflicts will not actually be errors. The changes in the conflict version will be separate from the changes in the edit version and the user will want to merge these changes. Therefore resolving conflicts and reconciling data become even more important to the system’s ease of use and, ultimately, its success.
The next challenge SchlumbergerSema faced in implementation is ArcGIS’ conflict resolution functionality. Figure 3 shows the conflict resolution workflow. It is organized into four columns. The left column is followed when there is no conflict. The middle column is followed when a conflict is accurately reported. In this case, only one of the versions (Edit, Conflict, Pre-edit) is correct, and the changes in the other versions need to be discarded. Based on the discussion in the previous section, most conflicts will actually fall in the right two columns. As the user views the differences between the Edit version and the Conflict version, he will (hopefully) realize that both versions are correct and he needs to merge the changes. At this point the user picks one version and then copies the changes from the other.
Unfortunately, the out-of-the-box capabilities of ArcGIS do not help the user merge changes. The user has two options:
1. Switch between versions to cut and paste changes from one version to the version being posted. This involves a great deal of graphical user interface navigation and mouse clicking. This approach is not used for practical reasons.
2. Write down the information from one version and re-enter the data in the version being posted.
As a result, conflict resolution becomes a laborious process, especially when the user has to recreate complex shapes. Users will frequently encounter mistaken reports of conflicts and will be forced to duplicate the data entry work in order to post their version. This extra conflict resolution work will increase as the number of editors increases and as the number of system interfaces updating ArcSDE increases.
One simple solution to this problem is to implement a smart clipboard that allows the user to copy an entire feature. Once the feature is copied to the clipboard, the user can manipulate the feature or any of its properties, including shape. This will allow the user to cut and paste an entire feature or pick selected properties to cut and paste. Ideally, the user could use both the smart clipboard and the feature editors from within ArcGIS’ Conflicts window. This would allow the user to merge versions without have to exit the Conflicts window and re-reconcile the version being posted.
Figure 3: Conflict resolution workflow
A more sophisticated solution is shown in Figure 4. In this case, the Conflicts window has been modified to add radio buttons on the conflicting properties. The user can directly select to keep desired property values in each version and then use the pop-up menu to merge her selections into the Edit version that she is attempting to post.
Figure 4: Modified Conflicts Window
Reconciliation is the third challenge SchlumbergerSema encountered during implementation. Figure 5 illustrates ArcSDE’s reconciliation behavior when many versions are being posted simultaneously to the database. If another version is posted after a user has started her reconciliation but before posting her version, the user will be forced to re-run reconciliation before she will be allowed to post. While this behavior is completely correct, it becomes problematic in implementations with large numbers of editors and/or system interfaces that are frequently updating ArcSDE. Further, the issues discussed in the previous two sections add to the reconciliation problem. ArcSDE’s conflict detection behavior means that conflicts are more likely to be reported during reconciliation. The laborious conflict resolution process means that it will take the user longer to resolve conflicts. In addition, SchlumbergerSema has found reconciliation to be quite slow on large, complex ArcSDE geodatabases. All of these factors combine to create a significant delay between the time when a user starts her reconcile and starts her post. As a result, users can get caught in the situation where she is unable to reconcile and post fast enough and is forced to reconcile her version repeatedly. SchlumbergerSema has implemented other GIS products that also support long transaction management, where it was impossible for users to post her versions because of this same reconciliation problem.
Figure 5: Conflicts introduced by reconciliation
A solution to this problem is to implement a Batch Reconcile and Post (BRP) server. The BRP server is an application service implemented on top of ArcGIS on a standalone server. The reconcile and post functions in the user installations of ArcGIS are replaced with custom functionality. When a user performs a reconcile or post, his client workstation submits a request to the BRP server rather than run the reconcile/post on the user’s PC. The BRP server runs the reconcile/post and notifies the user of the results. If there are conflicts, the user resolves the conflicts and re-submits the reconcile/post request.
The BRP server solves the problem described above because it serializes the reconciles and posts being performed by all the users and all the system interfaces. The user can request the BRP server to perform both a reconcile and a post. If there are no conflicts during the reconcile, the BRP server immediately performs the post. Since the BRP server is handling all of the reconcile/post requests one at a time, it is no longer possible for another version to be posted during this operation and cause repeated reconciles.
The BRP server also produces a productivity advantage. If the user runs the reconcile/post locally on his PC, he is unable to perform any other Esri work until the reconcile/post completes. This becomes an increasingly larger problem the longer it takes for the reconcile/post to execute. With the BRP server, the user can submit his request and immediately begin working on a new task (presumably in another version). Additionally, the BRP server can be configured to have more computing power than is available to users on desktop PCs. Since reconcile/post can be very compute and Input/Output (I/O)-intensive, the BRP server can execute reconcile/post requests faster than the user could perform them on his desktop PC. When the work queue is short, or empty, the user can get reconcile/post results more quickly using the BRP server.
The three key points in this paper are:
All of the issues presented here have been discussed with Esri and reported as bugs or enhancement requests. Until Esri is able to update ArcSDE to address these issues, users must carefully consider these issues and ensure that they are addressed for successful enterprise-wide implementations.
Thomas E. Lonski
Director
tlonski@slb.com
Robert Parsons
Senior Software Engineer
rparsons@slb.com
SchlumbergerSema
6399 S. Fiddler’s
Green Circle, Suite 600
Greenwood Village, CO 80111
Phone: (303) 741-8400
Fax: (303) 741-8401