The City of San Diego is constructing yet another GIS data layer--Storm Drains--to add to its ArcInfo based Regional Urban Information System (RUIS). An estimated 60,000 storm drain components are being captured from the City's existing hard copy map series. Resulting mapping applications either being constructed or being planned for include a Host ArcInfo ArcTools-based storm drain maintenance system, a water runoff and three dimensional modelling system, and a robust ArcView 2-based environment involving display and query with other integrated data layers. This paper highlights the significant factors with which the project's design personnel were faced, including mapping needs, and preparing a two dimensional mapping system to provide the necessary information to a three dimensional modelling product. This paper also wrestles with modern day software engineering concepts, and the difficult decision of customizing Esri's off-the-shelf product, "ArcTools".
This paper is divided into two distinct sections. The first section discusses the storm drain project as it relates to project planners, and those who might also be considering mapping a storm drain system in ArcInfo. High level application and database design concepts are presented, along with some purely technical programming points. The project manager may browse a few of the technical portions, should they become "tedious." The mid-range or advanced programmer is encouraged to "think along."
A major goal of this paper is to embrace the positive aspects of ArcTools. The concept of customization is contemplated, but only as clear cost benefits are seen for a specific tool.
This paper is semantically divided into the following sections:
The mapping needs related to San Diego's storm drains are diversified. A number of the City's agencies involved in planning need to view some aspect of the City's storm drain system.
The steward of the system is City Engineering. Personnel there are responsible for the inventory and upkeep of the storm drains on a daily basis. Engineering Design Services (a separate department) is responsible for the modelling of new storm drain systems that are to be installed in new commercial or housing developments. They are also responsible for potentially remodelling existing storm drain systems impacted by new inflows.
Although not directly related in storm drain development, a number of city agencies exist which must incorporate storm drain criteria into their development planning processes.
The digital storm drain maps will be stored and managed in the Host ArcInfo environment on a UNIX server. ArcStorm will be used primarily to permit multi-user, concurrent access of the attribute portion of the data stored in INFO. A custom ArcTools application is being developed for graphic and tabular editing and for production/maintenance plotting. The second portion of this paper is dedicated to the customization techniques used to refine ArcTools for specific engineering maintenance needs.
ArcView 2 will play a significant role in the City's overall Storm Drain System. Engineering Design Services has invested in four high-end personal computers for ArcView 2 use. They will use PC/NFS to access the data on the UNIX file server, therefore accessing the data in its most current maintenance state.
As for background display of related data, such as legal lots and surface water, a quarterly snapshot of the data will be downloaded from the City's data repository to Engineering's local server. This is strictly due to the apparent performance restrictions using PC/NFS. PC/NFS works well in a local area network (LAN), but very poorly over a wide area network (WAN). WAN/PC/NFS Arcview 2 test results revealed clear infeasibility of remote data access.
Ad hoc generation of field work area prints will also be generated for field crews using the ArcView 2 platform. Lap top computers may also be considered, which allows field personnel to reduce their paper trail and also access the most current of information in the system (as opposed to outdated paper maps).
ArcView 2 will also be used by various planning agencies to provide services at the customer level. Robust query, ad hoc mapping, and sophisticated overlay analyses are being designed using ArcView 2's intuitive tools. Several city data layers will be used, such as soils, digital orthophotos, zoning, and addresses. A typical user function in a building permit scenario will be that of a user supplying a site address of a land owner, and receiving as an answer all known as-built storm drains and flowage easements associated with the land owner's lot.
Unlike some infrastructure mapping styles, which focus on simple automated mapping and facilities management (AM/FM), the City intends to make full use of ArcInfo's spatial GIS component. Cost benefits from the GIS spatial component are at first glance remarkably extensive.
The design phase of this project had to carefully balance its long term goals with its immediate mapping needs and available financial resources. For San Diego, any data layer creation effort undertaken generates monumental costs, as compared to that which smaller cities experience. To illustrate, the City's current map library is divided into over one thousand tiles. For layers including the County of San Diego, 3,500 tiles must be mapped. City GIS layer projects are therefore staged, strategically distributing data capture over phases.
With respect to the storm drain system in the city proper, the components number 60,000. The amount of data entries associated with this total require a multiplication factor of 15. More importantly, the research needed to compile the actual database attributes easily consumed more labor than the straightforward process of data entry.
The existing storm drain maps promised a positional accuracy of +/- 10 feet, which raised the difficult question of attempting to improve the accuracy through extensive resurveying. Two factors became clear.
First, the City's existing land base accuracy is only marginally better, having been purchased from a utility company with less stringent accuracy needs. Secondly, it was decided that relative accuracy was sufficient. If features were situated well with respect to right-of-ways and street center lines, most mapping applications would be satisfied. There remained the need for absolute accuracy for water runoff modelling, the answer to which is discussed later in this paper). In the future, it was decided that the absolute positional accuracy of the storm drain system would dovetail the City's planned landbase rectification effort.
This section attempts to highlight significant aspects of the data model, in hopes that it will provide some experiential advice to cities undertaking a similar project.
The storm drain linear features (arcs) include: regular drains, forced, and encased drains, brow ditches, channels, underdrains and long drainage boxes. No polygons are planned for in this layer. Storm drain features chosen to be represented as points include: headwalls, inlets, catch basins, desilting basins, lugs, manholes, outlets, pump stations, tide gates, and underdrain outlets.
From a non-gis perspective, the linear and point structures in a storm drain system are obviously closely related in terms of water flow. They are connected and must be modelled with no distinct separation. For this reason, project designers chose to use the Arc/Node model (nodes representing the point features), and to attach key attributes to each type of feature.
The advantage to using the Arc/Node model is the close topological relationship that ArcInfo automatically maintains between the arc and the node. Any node feature, such as an inlet, immediately knows its connecting linear drains, and vice versa.
No known Arc/Node model problems have presented themselves in ArcInfo release 7.02. However, one minor problem was encountered in the use of ArcView 2.
While for cartographic purposes the node features were symbolized and rotated for proper orientation, nodes cannot be rotated in ArcView 2. As a consequence, any descriptive annotation that was placed close to a node point feature while it was rotated may be cosmetically overwritten if the node is displayed at a zero degree angle. An enhancement request has been made to the developers of ArcView 2, with hopes of a timely response. In the meantime, there exists a programmable alternative in ArcView 2 using Avenue. This may be undertaken if quality map prints are required in Engineering Design Services. Plotting in Host ArcInfo is problem-free.
Almost all annotation is derived from the attributes attached to the arc and node features. Text exists for: original drawing number, feature elevation, size, material, actual length, installation date, and work order number.
Careful consideration was made for the possibility of maintaining a feature link between the annotation and its parent arc/node attribute record. This mechanism would allow any changes made in a tabular fashion (independent of the graphics) to be programmatically updated in the map annotation. Three factors led project designers to choose against it.
First, significant overhead is incurred. Additional disk space is needed for maintenance of the attribute and for building a text attribute table for each annotation feature class. In the storm drain model, nine annotation feature classes exist. Already, the annotation has been the major contributor to the more than 2.5 gigabytes of digital storm drain data. Nine additional tables were not desired.
Secondly, attributes for the storm drains are more often static than changing. The benefits of automatic updates would be minimal.
Thirdly, because of the density and resulting close inter-placement of the annotation, automatically updating the text strings with new values could potentially cause unsightly overstrikes that would not immediately be detected if this were done programmatically.
The result is to rely on quasi-automatic annotation generation tools during graphic maintenance, since most attribute updates will usually be initiated by graphic selection. After successfully attributing a given feature, the user simply requests automatic new text generation.
For linear and point features, 3 classes of information are being attached. Physical characteristics include type of feature, dimensions, and materials used to construct the feature. Administrative attributes include operational status, work order number, installation date, original design document number, and the owner responsible for maintenance.
These attributes involve information that must be provided to add-on software packages to perform three-dimensional modelling and water flow analysis.
Some of the physical characteristics already mentioned are used, in additional to three elevation measurements. Due to the less than ideal absolute accuracy of the features, true characteristics, such as "true length" and "true surveyed location" are maintained. Currently, much of this information is unknown, but will be gathered on an as-needed basis. This patience in data completeness allowed for immediate comprehensive mapping of the entire city, with information gaps only occurring in the modelling domain. For these cases, the modelers, who will work only in small areas at a time, will gather "on demand" the specific data needed to complete their model. In reality, many individual storm drain systems that are already in place will go years unmodified.
Some drain attributes are not entered, but are calculated from other existing attributes such as elevation and diameter. These are: slope, capacity, discharge, basin area, hydraulic grade line, and water inflow rates.
The choice of attributes was based on a sewer modelling pilot performed for the Water Utilities Department of San Diego. A software product named "INSEWER" was used. The pilot revealed accuracy constraints and some ideal factors for which it is not feasible to maintain attributes. In terms of accuracy, point features must have a positional accuracy of +/- 0.5 feet, and an elevation accuracy of +/- 0.1 feet. This ultimately requires accurate site surveys.
An example of an ideal factor is to know the true shape of the manhole. Whether it is conical, cylindrical, or rectangular will influence the modelling results slightly. Project designers opted not to seek such a fine attribute in light of the other accuracy standards.
A number of programming issues were considered in the design of the storm drain management system. This section is intended for system designers who are not necessarily programmers but are interested in the programming concepts that influence the success of a user application. This section is also intended for the interested AML application programmer.
Inevitably, application development is evolving into a new form. Both hardware and software are rapidly being improved or revolutionized, with previous versions having as short a life span as a year. With significant strides being made in the GIS industry, one avoids embracing a current project data model for too long a time. Both the financial supporters and designers of projects are limiting their ambitions to projects with application life spans of just a few years or less. Long term project development aimed at long term cost benefits can now be expected to fall short of their goals because they soon need to be overhauled.
The industry mood is for "rapid application development". Quickly designed and programmed applications are designed to be short-lived; either soon replaced or easily modified to embrace changing technology. One answer to this trend is to purchase inexpensive off-the-shelf products which offer significant functionality that can then be tailored to a user's specific needs. In some cases, there may not be a need to tailor at all. Integration of add-on applications with base software systems is becoming more and more a simple task. This approach can prove to be more cost effective than to design and build your own system from the ground up.
San Diego Data Processing Corporation, like other large application developers, is charged with developing several user applications each year. All of these applications share a common base functionality in display, query, plotting, reporting, and so on. Esri's newest release of ArcTools provides complete base functionality in every one of these areas. The ArcTools system is friendly both to the user and to the programmer. It is ideal for a programmer interested in either "cannibalizing" the programming code behind the tools, or customizing its functionality for a specific user.
Arctools is a product of object oriented design, which, among many things, provides encapsulation and reusability of code, independence of user functions, and few or no "ripple-effects" caused by minor system changes. Maintenance is straightforward and reasonable.
Significant programming constraints were placed on the storm drain project. The project had a limited budget, but a wide variety of base requirements. An even more limited budget is being projected for post implementation program maintenance and enhancement. In short, the project budget had to be competitive. but there was no luxury of building an application from scratch. In addition, programmers at SDDPC are requested to follow the philosophy of writing applications which can be reused by similar projects with different clients. All factors pointed toward the use of ArcTools.
However, it is quite clear that ArcTools, by its generic nature, is not necessarily, in its delivered state, the final application that users in a production environment require. It is tailored to be friendly and complete, but by its generic nature (servicing all potential types of users), it is not always the most efficient. Each user must consider the design of ArcTools and decide if it is stand alone, or needs some customization. Esri should be complemented on their well designed product, because they built into their design the capability of changing it for particular needs.
Production of large, maintenance-oriented projects necessitates customization. Users must be able to "cut corners", while managers must build in project-specific quality control.
In the case of the storm drain project, the engineering clients have strict efficiency and data integrity requirements. Four designated technicians will be charged with maintaining 60,000 arc and point features, 300,000 pieces of annotation, and 2,100,000 attribute entries.
The remainder of this paper is to discuss some choices of customization in ArcTools for the storm drain project, and to compare the generic versus the customized. It is an effort to show the complementary relationship of ArcTools and its admitted need to be customized for certain large scale projects.
An admitted point in the customization of ArcTools will be the inevitable need to update the storm drain application upon Esri's next significant release of ArcTools. In response, this potential "cost" is balanced with the cost benefits of customizing. Every specific tool added to or revised for storm drains means potential "revamping" in the future. Customization is performed only when its cost benefits outweigh the cost of revamping in the future. If customization has little benefit to the user, it is dismissed. The following section is an example of "customization versus use of an existing generic ArcTools tool."
Due to the fact that one cannot predict a particular user's screen size and resolution capabilities, ArcTools elects to pick a general layout. The graphics canvas occupies the right portion of the screen, the main menu the upper left portion, with subsequent menus occupying the left portion of the screen below the main menu. Additional sub menus are generally placed beside their calling menus, or in the center of the screen. The unfortunate, but understandable result is a smaller than desirable graphics canvas, with blocky generic menus often intruding on the canvas. The user's remedy is to manually move menus side to side as needed to see any graphics that are covered.
With only a minimal effort, project designers designed a screen layout, and edited the menus to conform to the layout. Figure 1 shows the general screen layout for the storm drain project.
Figure 1. Screen Layout
The screen thus has "dedicated" spaces. A thin, horizontal wedge on top is dedicated to "permanent" panning and zooming functionality. The upper left portion remains the location of the main menu. The mid left position is reserved for graphic selection tools, along with popular combined "select and perform an action" tools.
The lower left portion of the screen is dedicated to optimized "current feature" editing tools. Any subsequent sub menus will typically base their placement relative to the lower left portion of the screen.
The actual graphics canvas is increased significantly in size due to the tighter control of the menu positions.
The decision to design a new screen has advantages and disadvantages. Screen usage is optimized, but is specifically designed for maintenance workstations using motif on 19 inch monitors.
Synopsis. The user's basic needs for arc feature attribution are to be able to input data attributes for each arc selected. Arctools does provide this capability, even allowing the user to access related table items, ArcTools' table editor gathers the information it needs on the current attribute table, and creates a standard input form menu that will reliably accept user input.
Nevertheless, attribute entry carries with it several issues with respect to a system's database. There are concerns for quality control, both interactive and "post session." Project leaders of large database efforts require efficiency because of the overwhelming amount of data to input. Moreover, technicians under pressure to perform repetitive work will naturally require the easiest, most routine way of entering data. Figure 2 shows a specific storm drain input tool that was added to the base ArcTools application.
Figure 2. Arc Coding Menu
Given so many specific needs, a custom menu was developed for the most efficient and reliable entry possible. This represents an additional programming investment, but it shows clear cost benefits.
Close scrutiny of this concise menu will reveal customization in 1) user oriented attributes, 2) item domain control, 3) commit/rollback options, and 4) record specific and global coding possibilities.
In contrast with ArcTools, all input fields are labeled in plain english, rather than in cryptic item terms. For those items such as CODE, STATUS, and OWNER (marked with an asterisk), the known possible values are hidden but "on line accessible" with the click of a button. The user is not able to type in values for these types of fields, because this may introduce error. The user must choose from a pop up scrolling list such as that seen for Material. While this feature is not feasibly included in ArcTools, it provides benefits in quality controlled data entry and allows minimal input from the user. In addition, the user is given choices in "english", and the actual code stored in the database is abbreviated(for disk space efficiency).
ArcTools' method of dynamically creating attribute input menus is quite powerful, but allows only simplistic data entry. As soon as a user has made an entry into a field, the value is directly committed to the associated record. If the user changes his/her mind immediately afterward, he/she must remember the old value and re-enter it.
Figure 2 shows a button labeled "M"("Memory") beside each field. Data entry is not committed to the record after each carriage return. The entry is not committed until the user is satisfied with the entire record entry, and "applies" the changes. Before the apply, the information is maintained in a temporary "variable" environment. Should the user need to return to the original value of a particular item, the input field's adjoining "M" button provides a recall of the original value as it still exists in the INFO record.
The same menu in Figure 3 provides the possibility of both individual record coding and global calculations on a selected group of elements. The reason for this dual capability is that certain attributes may be common for a set of storm drain features in a common area. Examples are "Work Order Number" and "Operational Status". To the right of the "memory" buttons is a vertical column of check box widgets headed by the letter "G", for "GLOBAL". For common attributes, the user makes the data entry just once, and implies "global calculation" by checking the corresponding global check-box.
A common strategy of coding a given set of newly digitized features is to make a "first pass" of global attribute coding, then visiting each record for the remaining specific attributes. The menu enables both type of attribution.
The goals of ArcTools appear to be quite clear and successful. The ArcTools application accompanies the ArcInfo software in order to provide complete and friendly menu-driven functionality for any user for any type of work. The application is worthy of praise because its internal programming was not casually written simply because it had "general" goals. It was purposely written with a sound, object oriented methodology which does not simply allow customization, but encourages it.
Given the rapidity at which base software is changing, an application designer/programmer is no longer afforded the luxury of "building from scratch". Streamlined software developers and third party vendors are delivering products that already cater to a large percentage of a specific user's needs. It is nonsensical to think that hundreds of ArcInfo pan/zoom menus exist in the GIS world today that probably share at least 80 percent common functionality.
Nevertheless, robust users of ArcInfo have specific needs in efficiency, user friendliness, productivity, and client database- specific quality control/quality assurance. The application designer must thoroughly justify the investment in customization, and plan for this customization to become obsolete in just a few years or less. The cost benefits must be close to "staggering."
Several key people at SDDPC have made major contributions to both the design and execution of the storm drain project. These include Lisa Stapleton, David Main, Thomas Richard, and Karina Erazo. Our Engineering clients were equally instrumental, including Lisa Adams, Bill Roubinek, and Steve Bertsch. The software engineering concepts mentioned in this paper are based on the valued thoughts of SDDPC's software engineering team, including Robert Hartman, Kent Wright, Juan Segovia, and others. My special thanks to this exceptional group of individuals.
Environmental Systems Research Institute, Inc. ArcTools and the ArcInfo ARCDOC Documentation File. Redlands: Esri, 1995.