Mark Johnson

The Government has successfully contracted for the production of vector maps of various types since 1988. Each map fulfilled an essential government need and when finally accepted contained minimal errors. Error detection and correction, however, carried a heavy price. Recently, we have applied rigorous project management techniques to control errors before they start and to bring greater discipline to the management of each project. We place a heavy emphasis on a detailed interpretation and digitizing plan for each and every feature to be captured. This paper summarizes our approach and our experience during the first year using this new technique.

This paper presents a process for managing digital mapping projects. The specific technical skills employed in creating ARC/INFO vector map data shall be left to others to describe. One key presumption of this paper is that all production and most support services are contracted out. The Government role is generally limited to managerial and technical oversight, with the exception of some critical technical supporting roles involved with source acquisition. The contracting process and contractual relationships are governed by United States Government Federal Acquisition Regulations (FAR). Description of the FAR and a discussion of its impacts on Government digital mapping projects is beyond the scope of this paper.

The purpose of this paper is to share with the Geographic Information System (GIS) community information about our experiences and management methods in creating digital vector maps. First, I shall propose a typical generic process that any digital mapping project might use. Next, I describe pitfalls we have identified from the implementation of specific digital mapping projects that followed the generic process. Thirdly, I present a rigorous new management process, which we apply to urban maps in an attempt to mitigate those pitfalls. The new project is the Enhanced Urban Vector Map (EUVMAP) project and I will report on the results of our experience over the first year of the project.

Figure 1 illustrates the typical process described here. The typical digitizing project begins with identification of a mapping need and definition of user, project, and system requirements stemming from that need. Assuming Government resources are not available to create a map to satisfy the need, the Government must contract out to satisfy the need. As a general rule the Government must use competition to the maximum extent possible in seeking contractual suppliers of its products and services. Therefore, once the need is identified and the requirements defined, the Government would solicit for services or fixed-price products to satisfy the need. The Government receives proposals and selects the lowest cost or best value proposal to provide the required digital map.

The Government most often provides to the selected producer the sources and specifications to be used in creating the digital map. Upon receipt of sources and specifications the successful production contractor would create the digital map and present it to the Government for acceptance. To be sure it is receiving what it has contracted for the Government must inspect the delivered data. When inspection reveals that the producer has met the terms and conditions of the contract the Government pays the producer.

The above process description is, of course, very abbreviated and simplistic, but serves to illustrate the typical process. A few key supporting activities bear mentioning, as the degree to which they are executed well, or not well, greatly impacts the success of the overall project.

First, whether procuring the needed map under a firm-fixed price or a cost reimbursable contract, the Government should have a good idea of the reasonable cost of the map-the "should cost" estimate. This estimate, more formally called the Independent Government Cost Estimate (IGCE), is described in the appendix. The sample IGCE provided in the appendix assumes a cost reimbursable contract and a Government directed Work Breakdown Schedule (WBS). The Government should create the IGCE prior to issuing its request for proposal (RFP). The IGCE is used in judging proposals and in managing the future contract.

The IGCE is essential in judging the cost proposal for each city work order and in managing city project costs. If the IGCE lacks rigor and detail then it will be less useful in guiding project management. Ultimately, the award fee is based upon how well the contractor meets the proposed cost, which was negotiated from the basis of the IGCE. This entire process serves to incentivize the contractor to know their business and attendant costs in great detail. Moreover, the EUVMAP process can penalize over-achievement. In other words, the Government customer determines how much error they are willing to live with. The Government project office then tells the producer the required error rate. If the producer spends too much time and money on making a layer perfect that doesn't need to be perfect they may be penalized.

Also, regardless of the contract type, the Government usually must have assistance in defining requirements, compiling the source & specifications package, inspecting data, and performing other ancillary duties, which the Government is not well-prepared to perform. Performing this support role is a contractor having a special set of skills-the System Integrator (SI). The SI may assist the Government by performing the bulk of the laborious tasks shown in figure 1 as Government Processes. The SI must not only have special skills in the specific technology area-digital cartography-but must also be expert at gathering and documenting user requirements; analyzing and tracking technical, schedule, labor, and business aspects of the project; and understanding the unique needs of Government users. The SI should be independent of the producer.

Other Government agencies often provide special support to digitizing projects, depending upon the particular project requirements. The National Imagery and Mapping Agency may provide geodetic, cartographic, and imagery support. Other intelligence agencies may provide specialized information and expertise.

From 1993 through 1997 MSC contracted with three companies for the firm-fixed price production of Urban Vector Maps, based upon a Defense Mapping Agency (now National Imagery and Mapping Agency) specification. While this initial UVMAP project was successful in building a library of over 30 vector maps of selected foreign cities for intelligence users, the project became somewhat costly and drawn out, especially in checking and correcting all errors in the delivered data. The Government also determined that the firm fixed price negotiated for some cities might have been significantly higher or lower than the actual costs incurred by the companies.

Although MSC/producer communications and relations were open and amicable, conduct of the project followed the traditional "Government/supplier" polarized model, rather than an integrated team model with combined activities and goals. Analysis of the UVMAP experience and other digitizing projects before and after UVMAP revealed the following problems:

• Time delays to specify, compete, and procure contracts for each individual project.
• Time delays in inspecting delivered data and making corrections.
• For firm-fixed price procurements, inadequate responsiveness in correcting errors.
• Frequent questions or clarifications on coding individual features.
• Miscommunication or misunderstanding of coding specifications.
• Lack of financial incentives beyond the negotiated cost or fixed price
• At times adversarial "Government vs. contractor" project environment.
• Quality assurance responsibility and control not clearly defined and therefore, shifted to Government rather than Producer.

In formulating a follow-on to the UVMap project, MSC wished to combine lessons learned with refined product requirements and project management techniques. MSC's analysis of past experience and upcoming customer desires goals identified seven principal goals:

1. Apply Agency-mandated PM philosophy & techniques.
2. Deliberately develop each city as a self-contained project.
3. Form Government/contractor integrated development team.
4. Offer monetary incentives rewarding successful adherence to time and quality goals.
5. The Government pays for the actual cost of the service, no more/no less, plus a reasonable profit based on good/bad performance.
6. Establish project control gates to build trusted producers and guard against bad performance.
7. Know exactly and in detail, what and how every feature and attribute is to be captured prior to beginning any digitizing.

Figure 2 illustrates the new EUVMAP project process, as modified from the typical generic process (see figure 1), formed to implement the above seven goals. The reader will note four main differences between this and the earlier generic process. First, Government/contractor interaction begins early and is continuous throughout the project. Second, the procurement step is removed from each individual production project process by conducting an up-front competition that results in one contractor who will conduct all mapping projects for the duration of the program. Third, great emphasis is placed on data inspection to be conducted by both parties, but with primary responsibility for error detection and repair on the producer. Finally, the reader will note the addition of a Government design phase that would be conducted for each individual project. This diagram does not address the management control gates mentioned in goal six-these are described later in the sub-section titled "Phased Approach & Control Gates."

Each city comprises a new project within the overarching EUVMAP project, or alternatively, program. Figure 3 schematically depicts the total EUVMAP Project cycle. The overarching EUVMAP project begins on the far left of the figure with a Requirements Definition phase. After project requirements are defined, the project enters the Concept Definition phase when the Government begins to define how the project will be implemented in order to meet the previously defined requirements. Next, performance requirements for the conceptual production activities are defined and the project enters the Acquisition Planning and Source Selection phases. It is in these phases that the Government will plan and execute its effort to receive bids and select a contractor (the "source") from as many as 5 to 10 prospective companies.

The EUVMAP project type may best be described as "iterative." Other common project types are evolutionary, incremental, or spiral. See Forsberg, et al for more information about project types. The EUVMAP project stems from common systems engineering practice-the same deliberate processes established in the engineering community to build aircraft, transportation systems, housing units, even computer operating systems. Our project is perhaps not nearly as complex as these examples, but equally benefits from a similarly disciplined approach. The EUVMAP project cycle is tailored from a possible total of hundreds of project milestones and control gates in a universal project management template, but still encompasses the same 8 project phases prescribed for all projects (note: a ninth, deactivation phase is not included in EUVMAP) (ibid.). The basic EUVMAP project cycle is shown in figure 3. Figure 4 presents a template for each individual city project.

One key to successful implementation of a project is the clear definition and segregation of roles within the project. Following are the clear-cut major roles for each EUVMAP project. Blurring of the roles can result in ambiguous responsibilities and in risk that tasks will not be properly executed.

Customer: The Government office with the mission need or that is paying for the production. The customer may or may not be the same as the user.

User: The persons ultimately employing the data to satisfy some part of their mission. Users might or might not be customers.

Government: Organizationally, this is the Map Services Center. Specifically, the contracting officer, the project manager, and contracting officer's technical representative act for the Government. In larger projects this subset of people would comprise the Government Program Office.

System Integrator (SI): Helps the Government perform technical tasks such as gathering user requirements, specifying designs, coordinating & measuring production progress, configuration management, testing, deployment, etc.

Producer: Designs, develops, and delivers the product.

Systems engineering practice advocates three periods and nine phases for any project cycle regardless of the end product. The three overarching periods are:

1. Study period

2. Acquisition period

3. Operations period

The three periods are further broken into a total of nine phases:

1. User Requirements Definition phase
2. Concept Definition phase
3. System Specification Definition phase
4. Acquisition Planning phase
5. Source Selection phase
6. System Implementation phase
7. Deployment phase
8. Operations and Maintenance phase
9. Deactivation phase

While the EUVMAP project cycle contains these periods and phases, I find it much easier and more meaningful (certainly much more memorable) to think in terms of five key steps or activities common to all digital map production projects:

It is from these five basic steps that the control gates for each EUVMAP project are defined. The Center for Systems Management defines the control gate.

Control Gate. "A management event in the Project Cycle that is of sufficient importance to be identified, defined and included in the Project Schedule by Executive Management or the Project Office in Government and Industry. A Control Gate requires formal examination to evaluate Project Status and to obtain approval to proceed to the next management event according to the Project Plan."

The following table shows the required control gates for each EUVMAP project in relation to the activities they precede or follow.

With some exceptions, a control gate is a meeting at which the contractor presents evidence that it has completed the required work and is ready to move on to the next activities. The control gate is a checkpoint at which all parties have agreed to stop and consider whether to proceed to the next stage of the project. It is at the control gate that the Government typically reaches one of four possible conclusions:

1. the review is acceptable and the contractor may proceed with the next activity
2. the review is accepted with limited authorization to proceed
3. the review is not acceptable and must be repeated
4. The review is seriously flawed and unacceptable. The Government and contractor should initiate discussions to correct the problem or terminate the contract.

Figure 4 presents, in the form of a notional schedule, the major activities, products, meetings, and control gates for each production project-the project plan. Each EUVMAP project begins when the Government identifies a need and a customer commits funding and other resources to the project. The project plan provides a uniform template reminding the project team of all major project items and events.

The project plan is read in rough chronological order from left to right, vertical phase by vertical phase. For example, starting on the left, the User Requirements Definition phase is the initial phase of the project and begins with a control gate, the Project Initiation Review (PIR). The project manager calls a Project Initiation Review (PIR) at the earliest possible convenience of all parties in order to rapidly communicate the known details of the upcoming effort and to allow all parties to make appropriate preparations. The major activities of the User Requirements Definition phase include gathering and verifying user requirements. The major product resulting from these activities will be the User Requirements Document (URD). A major meeting will be held to "walkthru" the URD and finally, the phase will conclude with a User Requirements Review (URR) where the users validate their requirements as documented in the URD.

Each phase represents a new project focus and progressively better-defined production designs and specifications. Although the phases occur in rough chronological order, they need not be sequentially executed. For instance, the Concept Definition Phase activity to "Identify, Acquire, and Examine Source Materials" may begin immediately after PIR, which actually kicks off the User Requirements Definition Phase.

Continuing with the example project plan, after the URD is approved the project enters the Concept Definition Phase. At this point many activities are occurring simultaneously in an effort to conceptually define the data to be produced and quickly move toward a detailed specification. The Government and SI are rounding up sources for the digitizing effort, studying special requirements for expertise, such as linguistics or geology, and refining early estimates in order to assign sufficient funds. During this phase production design documents are drafted and, if the project is deemed sufficiently complex, the project manager will call a Project Concept Review (PCR) before moving into the Specification Definition Phase. PCR is the only optional Control Gate and is held in order to review and decide on key competing project and production concepts before moving on to fully define specifications.

The City Study Period continues as the Government and SI write the production specification and further instructions to the producer. The later are given in the form of a Production Design Document (PDD). For the EUVMAP project the Government has adopted and enhanced NIMA's Military Specification for Urban Vector Maps. MSC maintains configuration management control and may therefore rapidly modify the specification to meet each project's unique requirements. The modification process, however, is designed to follow standard NIMA feature coding schemas in order to maintain compatibility. Think of the EUVMAP specification as an addition to, and not a subtraction from, the UVMap Mil-Spec.

When the Government has thoroughly defined exactly what it wants it presents its specifications, source material, and other instructions to the contractor at the Project Specification Review (PSR). Successful completion of this review concludes the Project Study Period and begins the Implementation Period. At this point the involvement of the production contractor becomes more than occasional. The producer receives the Government specifications and material and builds their implementation plan. The Implementation Plan contains all the detailed schedules, resource plans, tooling, coding tables, etc. needed in order to build the required product. The producer presents the Implementation Plan to the Government at the Critical Design Review (CDR).

The CDR is the last control gate prior to authorizing the producer to begin production. It is an important control gate, because once production begins changes become more expensive to implement. Following CDR approval, the production contractor executes its Implementation Plan. During the course of production the producer reports status to the Government on a monthly basis. Status includes schedule, cost, technical, and business status. The Government also periodically conducts informal in-process reviews of production technique and progress.

Typically the producer delivers individual sections of the total project area as each is completed. This staged delivery works better for the Government's SI, which has to inspect the delivered data. Prior to accepting any data for full inspection, the Government, SI, and producer jointly determine if the data meets minimum standards for inspectability. This joint examination and decision comprise a Data Test Readiness Review (DTRR). A DTRR is conducted for each and every data delivery. DTRR usually does not require face-to-face meetings and therefore is conducted telephonically.

After the data is deemed "inspect-able" the SI inspects the data and reports all discrepancies to the Government. The Government decides how to resolve discrepancies (errors)-whether to return the error to the producer for repair or have the SI fix it. The resolution depends upon the extent of the problem, although in general the producer is responsible for repairing all errors.

The project cycle concludes after the SI has inspected all data deliveries and the Government has accepted the completed data set. After accepting all data from the production contractor, MSC invites the customer to participate in a formal Data Acceptance Review during which the customer and users decide whether the data MSC has now acquired meets their original needs. Upon successful completion of DAR, the EUVMAP team's responsibility ends. Further maintenance, modification, and use of the data becomes the responsibility of other elements. The city project concludes with a discussion of lessons learned at a Final Project Review.

A detailed description of EUVMAP quality assurance and control procedures is beyond the scope of this paper, but deserves mention. Quality assurance plays an integral and visible part throughout each EUVMAP project cycle. The Government requires that the producer designate a Quality Assurance Coordinator (QAC) and use Independent Validation and Verification (IV&V) procedures. The QAC must be completely independent of the "production line" and is not subject to performance evaluations by the production manager. The QAC establishes a quality assurance program for the production effort and periodically examines quality and reports findings and recommendations to the production manager.

Both producer and SI use the Government-provided EUVMAP Inspection Process (IP) and software tools (IP Tools). The Government's philosophy equates data acceptance to a test. The Government wants the producer to pass the test and, therefore, openly provides "copies" of the test and the "answers" to the producer. If the data passes the inspection of the producer using the Government's IP tools, then it should pass the Government's inspection using the same tools. Random selection of inspected areas and features keeps the producer's data fixes honest.

Quality Assurance is thorough and complex. Table 2 depicts the table of contents for the IP Manual for any city. The Inspection Process leaves no quality aspect of the data unexamined. Table 3 illustrates how quality standards are established at the beginning of the project according to what the user is willing to live with. Subsequent inspections and acceptance decisions are based upon these standards.

During this first year we have nearly completed two EUVMAP projects and have begun yet a third. The three projects represent a total effort of in excess of 20,000 labor hours between the Government, the SI, and the producer. All projects have been closely coordinated with the US Government's primary producer of military maps of foreign areas-the National Imagery and Mapping Agency (NIMA). In fact, NIMA personnel directly participate in each project.

We have found that the EUVMAP project process meets detailed customer needs in a timely manner. During the former UVMap project (1993-1997) many producers required over 12 months to deliver individual cities, although no UVMAP city was as detailed and complex as any of the EUVMAP cities begun so far. The first EUVMAP project took eight months, start to finish. The second, a much larger effort on the order of 5,000 to 10,000 production labor hours, began its study phase in July 1998, started production in October, and is now projected to finish in August 1999.

The EUVMAP project has so far met all its goals. We have succeeded in implementing a new project management philosophy and technique. Each city is a self-contained project implemented by an integrated Government/Industry team. Each project uses disciplined management methods, such as control gates, and thoroughly defined specifications.

The project has attained the goals to pay actual cost plus a reasonable profit and to incentivize the producer with award of variable fees (profits) based upon performance. The first EUVMAP city was completed on time and at roughly 15% less cost than negotiated.

Although the Government has not conducted a thorough analysis, preliminary indications are that the new project techniques have achieved significantly improved quality. Error rate, compared to the previous UVMAP project, has been low. Error rates as high as 4,000 or more per city were not unheard of during the UVMAP program. In the first section of the first EUVMAP city the SI found 1,272 errors, primarily feature attribution problems. The first section consisted of a total of 24,365 features. Because more than one error may apply to one feature, an exact error rate cannot be calculated, but we can say that the error rate was found to be no worse than 5%. By comparison, error rates of up to 35% were common on the older UVMAP project.

This appendix provides a method for calculating the expected cost of an EUVMAP project. Included are a detailed cost estimate (1) and a description of the basis for that estimate (2).

Following is a rough breakout of the top level WBS tasks into subtasks for cost estimating purposes.

General comment: The IGCE assumes no specific EUVMAP experience on the part of the producer (in other words, they may have produced other similar products, but are on the bottom of a steep learning curve for this particular product). Future projects would benefit from their experience gained in this first EUVMAP project and should then require significantly less effort than estimated in the IGCE. The IGCE assumes careful, deliberate planning and execution by the producer. The IGCE represents approximately the largest reasonable estimate of hours and cost, given liberal, yet reasonable management practices. The IGCE does not take into account any extra Government-imposed cost control instructions.