Author: Bart Guetti
Abstract
Many organizations are currently in
the process of developing fully integrated geographic information
systems. These systems, when completed, streamline the management,
update and analysis of geographic information. The more complex
systems consist of GIS, RDBMS and modeling components which are
utilized by many departments for a variety of purposes. Applications
to enable agencies to access and maintain their maps and related
attribute information are an important component of these systems
and must be developed. Once developed, they must be implemented
in a production environment.
The road from application requirements
analysis to implementation is not an easy one, however, with challenges
presented by refinement of user functional requirements; operating
system, platform and software upgrades and evolution of the graphic
and attribute databases. This paper will present the process followed
while implementing Chester County, Pennsylvania's Property Update
Application (PUA). It will discuss the requirements of the PUA,
application development models, how these models were applied
in developing the PUA, the various discoveries that occurred,
and finish with a series of recommendations on how to increase
the chances of successfully completing an application on time
and within budget.
PUA Requirements
Due to the rapid land development in
the County and the many agencies that are involved in managing
and accessing land records information, the Chester County Department
of Assessment required a land records system (LRS) that would
integrate the processes of mapping, recordation and assessment,
enabling them to process transactions in a coordinated and time
effective manner. It was for these reasons that a GIS in conjunction
with an RDBMS were chosen as the principal components of the CHESCO
LRS. Furthermore, an application that would allow the County to
simultaneously update their maps and tabular records was urgently
needed. For this reason the Property Update Application (PUA)
was developed.
A requirements analysis for the PUA was completed in 1992 (Kevany, et. al. 1992). Several models were produced as part of that analysis including data flow diagrams, Figure 1. This diagram represents the data flow associated with processing an application for a subdivision.
Figure 1
Also produced was a process model, Figure
2. This model represents the functions that the PUA must perform
to process an application for a subdivision.
Figure 2
In addition to the models, requirements
definitions and specifications were produced. The following is
a summary list of the major requirements defined.
Uniform Parcel Identifier (UPI) -
As required by Pennsylvania Law Act
1988-1 SB131, each land parcel in Pennsylvania must be assigned
a geographically based parcel identification number. The UPI must
be unique and it must be provided to any individual who requests
one within 24 hours of applying for one. Therefore, the PUA had
the functional requirement of generating this number and the non-functional
requirement of generating it within the specified time period.
Graphical User Interface (GUI) -
Considering the time requirements for generating the UPI and the
complexity of GIS software, the entire PUA needed to be accessible
through a menu driven interface. This included the ability to
select an application to process, check the parcel and planimetric
features out of the database, make the necessary edits, check
the features back in to the database and the ability to release
a transaction or retract a checked in edit.
Graphical editing
- The County required the tools to update the parcel and planimetric
layers in the database. The graphical editing functions needed,
included: 1) the ability to add and edit points, nodes, labels,
arcs, polygons and regions; 2) integration of digital orthophotos
into the editing environment; 3) adding and editing of annotation;
4) entry of data using mouse, keyboard, digitizer and coordinate
geometry.
Oracle Interface - The
County utilizes Oracle to store its deed, assessment and appraisal
information. The PUA was required to obtain information from Oracle
regarding the status of applications for property transactions.
The PUA also needed to store information in Oracle about the status
of these property transfer applications, as well as information
about the physical characteristics of the property such as acreage,
lot number, subdivision phase number and history.
Database Management
- An overall management system that would protect the integrity
of the spatial and non-spatial data and facilitate its maintenance.
Due to the large number of users of and therefore transactions
on the database, the system required feature locking, coordinated
commits and history tracking.
Multiple Transaction Types - The PUA is required to handle
six different types of property transactions, including subdivisions,
combines, area and boundary adjustments, construction plans, and
condominium declarations. Each transaction type required a unique
combination of the functions listed above, Figure 3.
Functional Requirement
Transaction Type UPI GUI Editing Oracle Database Management
Subdivision X X X X X
Combine X X X X X
Area Adjustment X X X
Boundary Adjustment X X X
Construction Plan X X X X X
Condominium Declaration X X X X
Figure 3
Application Development Models
Application development is a multiple
phase process consisting of the the general steps of specification,
development, validation, and evolution (Sommerville, 1996). Many
different process models exist to accomplish the above steps.
However there are currently four major process models that have
gained popularity: waterfall or life cycle model, evolutionary
development, formal transformation and assembly from reusable
components (Sommerville). The nature of the application, and the
organization developing the application, determine the model that
is best suited for the situation. Also, a single model may not
be appropriate for the entire application. For instance, the GUI
may be developed using evolutionary development whereas a more
critical, safety oriented portion of the application may be developed
utilizing the formal transformation method.
A definition of each of these models
includes:
Waterfall
- represents the above listed phases of specification, development,
validation and maintenance as separate process phases such as
requirements definition, software design, implementation, testing,
etc. After each phase is defined it is "signed off"
and development goes on to the following stage. This model is
most effective where the clients needs are known and well documented.
Waterfall Model
Evolutionary development - the phases of specification, development, validation and maintenance overlap and iterate. The client is active in this refinement process which utilizes exploratory programming and throw away prototyping to assist in this refinement. This model is usually applied where the clients needs are either unknown or unclear.
Evolutionary Model
Formal transformation
- utilizes formal mathematical methods to produce a specification
and transform these specifications into a program. These are "correctness
preserving" and assure that the application meets its specification.
This model is used in applications where safety is a concern.
Component re-use
- focuses on utilization of existing components to develop the
application. This model is employed where components exist that
can be readily integrated into the application.
Model Characteristics
Important application model characteristics
include:
Understandable - how easy is it to understand
the process definition?
Visible - do the process activities
result in visible products?
Supportable - can the process be supported
by CASE tools?
Acceptable - is the process acceptable
to and usable by the developers?
Reliable - are errors avoided or trapped
before they become problems?
Robust - can the process continue in
spite of problems?
Maintainable - can the process evolve
with organizational requirements?
Rapid - can the process deliver a system
quickly?
These characteristics are also the criteria
for selecting which process model to utilize on a project. No
CASE tools were available for the project, therefore supportablity
was not determined to be an important characteristic for this
project. All of the models were acceptable to the development
team, therefore that criteria was dropped from the evaluation
process.
To determine the best model to use for
a particular application, the process models must be evaluated
for their characteristics, Figure 4.
Development Model Characteristics
Model Understandable Visible Reliable Robust Maintain Rapid
Waterfall X X
Evolutionary X X X
Transformation X X X X
Re-usable X X X X
Figure 4
Correspondingly, the clients requirements
also need to be evaluated to determine which characteristics best
meet their requirements, Figure 5.
Model Characteristics
Functional Requirement
Understand Visible Reliable Robust Maintain Rapid
Multiple Transactions 1 2
UPI 1
GUI 1 2
Graphical Editing 2 1
Oracle Interface 1 2
Database Management 1
Figure 5
Once these evaluations are completed,
a model(s) can be selected which best meets their needs.
Model Selection
The evaluations above were used to determine
which model to use to develop each portion of the application,
Figure 6. Those requirements that translated into a specific module
were more amenable to this evaluation than those that did not.
Also, some requirements were developed using more than one method.
Development Model
Functional Requirement Waterfall Evolution Transformation Reusable
Multiple Transactions 1 2
UPI 1
GUI 1 2
Graphical editing 2 1
Oracle interface 1 2
Database Management 1
Figure 6
1-Primary method 2-Secondary method
Application Development
The PUA was developed utilizing ArcInfo
version 7.0.3, Oracle version 7.2.2 and ArcStorm version 7.0.3.
The primary module of ArcInfo used was ArcEdit with occasional
escapes to ArcPlot and Arc.
UPI - Development
of the UPI module of the PUA was accomplished using the waterfall
model. This model was employed as 1) requirements of this portion
of the application were well known and documented; 2) the legal
mandate requiring the UPI, required that progress on this portion
of the application be visible.
The UPI is required by law to be unique
statewide and is defined to be based on the coordinates of the
geometric center of the parcel. It consists of the two digit municipality
code - DISTRICT, the 200 scale tile number - TILE_2400, the hundreds
place of the x and y coordinates - X-COORD and Y-COORD, and a
four digit condominium number - CONDO, if applicable.
18 545300 152971 0001
DISTRICT TILE_2400 XY COORD CONDO
The DISTRICT, and TILE_2400 were determined
by using ArcPlot's RESELECT OVERLAP command on the MUNIBND and
TAXGRD coverages to determine the appropriate polygon that the
centroid fell in. The X-COORD and Y-COORD were generated using
the Arc ADDXY command, while the condo number was obtained from
Oracle by adding one to the highest condominium number for this
particular UPI.
GUI - The
GUI was developed using a combination of evolutionary development
and component re-use methods. Given the fluid nature of the workflow
involved, evolutionary development was used in developing the
menus and forms. Also, considering the extensive collection of
ArcTool icons and menus available, much of the GUI was developed
with component re-use. Where available and suitable, ArcTools
icons and menus were employed. This included most of the arc,
node, label, polygon and annotation editing menus. Many new icons
were required for the ArcStorm commands and were developed using
BitMap, an AIX 3.2.5 utility used for developing bit mapped icons.
Custom menus were required for entering attributes into the Oracle
tables and were developed using ArcInfo's FormEdit utility. Menu
bars were developed using vi, Unix's text editing command. Each
different transaction type required a different main menu, as
only certain functions were required for each transaction type.
To prevent users from accidentally performing an illegal operation
for a particular transaction type, only those operations allowed
were present on the main editing menu for that transaction type.
Graphical Editing -
Most of the graphical editing functionality required was available
in Esri's EditTools, a library of AML's and menus for editing
ArcInfo coverages. Because of the robustness of these tools and
the ease of their re-use, component re-use was selected as the
primary development approach for this portion of the application.
Some modifications to the EditTools were required however to merge
the edit menus for arcs, polygons, nodes and labels into a single
menu. Modifications were also necessary to exclude inappropriate
edits. Due to the non-functional requirement of being able to
produce UPI's within 24 hours, label creation and centering functions
were added to some of the main editing menus to speed up the process.
Additionally a button was added to allow the user to quickly switch
edit coverages, as was a linking and snapping sub menu. The ability
to add annotation from related tables was also added.
Oracle Interface - The
database that PUA would be interacting with had already been designed
and developed, Figure 7, and because of this, the requirements
for the data to be handled by this interface were well documented.
The database also contained several integrity constraint checks,
that were also well documented. For these reasons, use of the
waterfall model was possible.
Figure 7
Also, given the critical function that
this interface would serve, visible progress on its development
was essential. This further strengthened the case for use of the
waterfall approach for the interface to Oracle.
One major portion of the interface was
not as well documented, or at least understood, however, and that
was the capabilities and operation of ArcInfo's Database Integrator.
Such issues as execution of stored procedures and the advantages/disadvantages
of using cursor vs. SQL*PLUS statements were either unclear or
totally missing from the DBI's documentation. For this reason,
the evolutionary approach also was employed to explore the use
of the DBI in developing the Oracle interface.
Database Management - Given
the large number of users that would be updating both the ArcInfo
and Oracle databases, ArcStorm, Esri's spatial database management
software, was selected to manage the database and was therefore
incorporated in the PUA. It would provide several of the requirements
surfaced during the requirements analysis, including feature locking,
history tracking, and coordinated commits to the database. However
it was a new product which lacked extensive documentation in its
use. For this reason, it was necessary to develop these portions
of the PUA utilizing the evolutionary development model.
Utilization of ArcStorm proved to be
the most challenging portion of the PUA development effort and
required the use of both exploratory programming and throw away
prototyping to develop the database management portion of the
PUA. Development of the PUA would have been prohibitively expensive
using any of the other models. The major drawback to the evolutionary
approach, that being poor visibility of progress, was substantiated
as an issue in the development of the PUA.
Findings
Many discoveries were made in the process
of developing the PUA. Some were due to the undocumented behavior
of the software modules being utilized. Some were due to the refinement
of needs defined in the requirements analysis. These refined needs
are sometimes referred to as emergent requirements, or those requirements
that emerge as the customer's understanding of the system develops
during the system development (Sommerville). Several discoveries
were also made when moving the PUA from a development environment
to a test environment and as a result of the upgrade of the computing
environment. The following is a list of the discoveries made.
Multiple Transactions -
The exact requirements for each transaction type were still evolving
as the application development process began. It required several
prototypes to fully develop the requirements for each application
type. The major areas requiring clarification were which fields
in the Oracle database required updating and which annotation
subclasses would need to be edited for each application type.
UPI - Development
of the UPI generation portion of the PUA was the least difficult
task of the application. This was primarily due to the well documented
requirements for the UPI and the developers familiarity with the
ArcEdit and ArcPlot modules. There were some unexpected requirements
that evolved, however. A question arose over which UPI to use
to generate new UPI's for condominiums whose parcels were subdivided
or combined. A second challenge was presented by determining which
municipality code to use for parcels that spanned several municipalities.
A decision was reached to use the code of the municipality that
contained the largest portion of the parcel. Although not a major
problem, it did present an extra bit of programming effort.
Another discovery made when migrating
the application from the development environment to the test environment,
was the difference in speed in accessing other layers. While developing
the PUA, the municipal boundaries and tax map grid coverages were
accessed as single seamless coverages with acceptable performance.
Once PUA was moved into the test environment these were accessed
as LIBRARIAN layers, with totally unacceptable access times. Access
was much quicker in coverage format, and therefore the AML's had
to be revised to access them in this format.
Revisions to the UPI generation AML
was required when it was discovered that ArcView needs a simple
item to relate to. This was discovered when the parcel database
began to be accessed by the public using ArcView as the public
access platform. A revision was necessitated as ArcView needs
a simple item to relate to the tables. The parcel layer, however,
contained a redefined UPI item. The solution was to add a new
item to the PARCEL.PAT that was a fully defined UPI. This did
force some minor changes to be made to the PUA to populate this
new field.
GUI -
The use of existing ArcTools AML's and menus made development
of the GUI much easier than starting from scratch. However, there
were some unexpected developments with the development of this
portion of the PUA. Some of the ArcTools AML's required the initialization
of key variables. Certain tools would not run without the execution
of other AML's. Another discovery resulted when trying to run
certain AML's from a menu. AML's with certain names would not
run, but when the AML name was changed, they would. RELEASE.AML
and SAVE.AML would not run until renamed. Apparently these commands
are ArcInfo atools with the same name, which created the confusion.
Graphical Editing -
The majority of the graphical editing required for the PUA was
standard ArcEdit capabilities. ArcTools has separate menus for
each feature type. To simplify the editing process, the individual
editing menus were combined into a single menu for the PUA. Another
discovery was the limitation on the size of the numbers that could
be calculated in ArcEdit. One field in the SUBDIVISION.PAT is
APPN_NUM, which is the property transfer application number. This
is a 12 digit integer field which INFO calculations can handle.
However the CALCULATE command in ArcEdit, has a size limit of
11 digits and therefore calculation of the application number
had to be accomplished with an AML using CURSORS. Until this was
discovered attempts to calculate the APPN_NUM resulted in an empty
item, without any warning of a problem.
Annotation in ArcEdit can be entered
from the keyboard or from the database via a relate. A problem
occurs however when there are many related records to choose from.
The only relate type available when relating to an Oracle table
is FIRST. This means that the data from the first related row
is all that is accessible. For an AREA ADJUSTMENT transaction
type there will be multiple rows for a parcel, one with the original
acreage and the new one. To enable PUA to access the most recent
acreage a routine had to be developed to NEXT to the most recent
acreage row for the desired parcel. Once this was solved, another
problem was presented in that the value of that row had to be
moved to the appropriate EditTools AML variable so that it would
be presented in the annotation editing form.
Oracle Interface - The
original design called for the interface to Oracle to access stored
procedures for updating the database. Stored procedures are PL/SQL
programs used for defining and accessing the database. It was
discovered that the DBI only allows the execution of procedures
that do not alter the database. Therefore, all of the stored procedures
that were to be developed in PL/SQL had to be developed in AML.
Another DBI limitation discovered was AML's inability to handle
NULL values. When attempts were made to insert NULL values into
the Oracle database using the DBMSEXECUTE command, AML would return
a "missing expression" error. The solution to this was
to use the QUOTE function with variables that may contain NULLs.
The two major options for updating Oracle
with ArcInfo are the Data Base Integrator commands, DBMSCURSOR
and DBMSEXECUTE. When inserting new rows in a small database,
both methods execute in approximately the same speed. However,
it was discovered that when dealing with a large database, DBMSEXECUTE
is much faster at inserting new rows. Therefore, DBMSEXECUTE was
the Oracle update method selected for inserting of new rows into
the database. DBMSCURSOR, however was used to update individual
rows as it provided acceptable performance and allowed users to
update individual fields.
A decision was made early in the design
stage that no records would be deleted from the Oracle database.
Instead, they would be "retired" through the use of
time stamps. This decision combined with the databases primary
key requirements, meant that many tables required conversion to
a compound key. Through the addition of sequence numbers and other
foreign keys, the tables were modified to handle multiple instances
of the same primary key. However, the need for this was not discovered
until some of the modules had been developed. Therefore the AML's
that updated those tables needed to be modified to use these new
fields. These changes presented an additional application development
challenge, that of keeping the AML's, database and data dictionary
synchronized.
Attempts to upgrade Oracle to v 7.3
presented a problem for the Oracle interface part of the PUA.
A problem occurred when PUA attempted to connect to Oracle multiple
times. The DBI would fail to connect on subsequent tries, issuing
an Oracle RPC error. This problem occurred as Oracle 7.3 was not
supported by ArcInfo v7.0.3. The only solution available was
to reinstall Oracle v7.2.2.
Database Management -
Given the relative newness and complexity of ArcStorm, this portion
of the application development process had the steepest learning
curve and correspondingly the largest number of unanticipated
requirements. Overall, interactions with ArcStorm required that
"all instruments be in tune". It requires that sufficient
computing resources be available, that the network and all servers
be performing well, that all features attempting to be checked
out are unlocked, and all integrity constraints are met upon checking
in. If all the conditions are correct, ArcStorm is a powerful,
but demanding, technology. If the conditions are not optimum,
and the transaction fails, there is an extensive amount of cleanup
work involved. See Guetti, 1996 for more details on ArcStorm related
discoveries.
Upgrades to Oracle v7.3 presented a
problem as described above. The upgrade of AIX from 3.2.5 to 4.1.4
also presented a problem to ArcStorm. The Automounter software
for mounting NFS file systems, concatenated a tmp_mnt directory
onto the remotely mounted file system path names. This presented
a problem to the ArcStorm server as it no longer recognized the
file system name and therefore could not find the resources it
contained. The workaround for this was to keep all the necessary
files on locally or permanently mounted filesystems.
Results
The PUA was finally completed and moved
from a development environment to a test environment in March,
1997. It is currently being utilized by the Chester County Bureau
of Land Records (BLR) on a daily basis for the entry of new property
transfers and those transfers that have accumulated since the
data conversion freeze date. Overall the satisfaction with the
product is high and it is playing a critical role in the BLR workflow.
Requests for enhancements are already being made and with the
software engineering techniques applied in PUA's development,
the effort required to complete these enhancements and others
should be acceptable.
Recommendations
As can be evidenced from the findings
above, the road from application requirements analysis to full
implementation, can be a slippery one. Even with the best development
models, there is plenty of room for surprises and detours. Based
upon the experiences gained in development of the PUA, a list
of recommendation on how developers can avoid problems has been
developed.
Re-use components. Software components
are becoming commodities. Where possible, re-use these components
to develop applications. Development times will decrease while
software quality will increase.
Prototype. Where custom software does
have to be developed and requirements are uncertain, use prototypes
to develop initial versions of the application. These can be
used to obtain more refined requirements and possibly used as
a template for the final application.
Document, Document, Document!! Develop
written requirements definitions and specifications of excruciating
detail. This has two benefits. First, it stands a greater chance
of giving the developer clear guidelines on what functions the
application needs to perform. Second, it forces the user to think
through how the application will be used which reduces the chances
of missing important requirements.
Document the programs. This will assist
the interpretation process for the principle developer, any new
developers, as well as users. It should also assist in tracking
changes to programs and relate those changes to changes in the
functional specifications.
Develop a data dictionary. This will
greatly simplify application development as the dictionary can
be referred to for variable existence and spelling rather than
searching to find the variable in another program. This searching
can consume a large amount of time. If the original variable can
not be found in another program, then there is a risk of creating
a second variable for the same purpose.
Develop standards for naming programs.
Establish an order such as verb_noun, e.g. select_features.aml.
Make sure that names clearly describe what the program does and
that the verbs and nouns are used consistently throughout the
application.
Practice high cohesion in designing
components. Cohesion is a measure of the relatedness between components.
A component should implement a single logical function. Components
that contain unrelated operations possess low cohesion.
Practice loose coupling in designing
a component. Coupling is a measure of the interconnections between
components in a system. Loosely coupled systems consist of components
which are almost independent of one another. Loose coupling is
obtained by ensuring that the data representation are held within
a component and that the interface to other components is through
a parameter list.
Don't mix functional with non-functional
requirements. Keep the functions that the system must possess
separate from the hardware and performance requirements.
Don't try to beat your prototype into
a final version. If it needs to be thrown away, do so.
Test solutions on a full database prior
to deploying them throughout the entire application.
Consider rapid application development
(RAD) techniques for fast prototyping.
Develop test cases. For instance for
the PUA, the actions that were required for each of the property
transaction types. Work through these preferably during the design
phase, or at a minimum during the prototyping phase
Develop viewpoints (different views
of the system) by all stakeholders. In the case of the development
of the LRS, viewpoint analysis would have surfaced the different
requirements of ArcView vs. ArcInfo, for the UPI.
Just say NO! What appear to be minor
requests for application enhancements, may turn out to be significant
requests, if they affect other AML's. Don't forget to also factor
in the testing associated with the requested change. Beware of
feature creep.
Version management. Create a new version
each time a new set of requirements are defined or refined. There
should be a new version for each iteration of the development
cycle.
TEST! Test all components of the application,
with real cases as much as possible. Static testing is much less
expensive but much less successful at detecting problems.
Conclusion
Two major challenges existed in developing
the PUA. First, the behavior and requirements of critical modules
of the software being utilized. ArcStorm and the Data Base Integrator
are complex systems which have many requirements and features
that were undocumented. The second major challenge in developing
the PUA, was the undocumented and unknown requirements of the
application. Specifics on COGO entry, condominiums, and parcel
history were being refined through the development of application
prototypes.
These unknowns forced much of the PUA
development to be conducted using evolutionary development with
an emphasis on prototyping, which proved to be an effective method
of refining requirements. However, when developing an application
consisting of such resource intensive components as ArcStorm,
DBI and Oracle, prototyping can be a very time consuming process.
Also, use of the evolutionary method can make it hard to control
the scope of the application. Many of the features in the final
version of the PUA were not in the original requirements document.
Nonetheless, development of the PUA without the use of evolutionary
development techniques would have been unacceptably expensive
and most likely resulted in an incomplete product.
Developing the PUA was an exciting and
rewarding experience. By utilizing certain software engineering
techniques, development of the PUA was simplified and a higher
quality application was produced. Observation of these techniques
and the recommendations put forth in this paper will give future
applications a much greater chance of being completed on time
and within budget.
Footnote
Implementation
The separation of application development
phase from implementation phase is a difficult process and which
the literature provides little clarification. The literature categorizes
implementation either in the testing or maintenance phases. For
purposes of this discussion, I used the migration of the application
from a test environment to a production environment where the
application is used with the actual database by end users as my
definition of the implementation phase.
References
Guetti, Bart (1996), Developing a
Land Records System using Oracle and ArcStorm, Proceedings
of the 1996 Esri User's Conference.
Kevany, Mike (1992), Chester County
Land Records System Master Plan Update, PlanGraphics Publication
Number 451R
Sommerville (1996), Software Engineering,
5th edition, Reading MA: Addison-Wesley
Bart Guetti
PlanGraphics, Inc.
1300 Spring Street
Suite 306
Silver Spring, MD 20910
bguetti@plangraphics.com