Mickey A. Fain

Quality Assurance: How to Build QA into the Conversion Process

Abstract

Data is a critical link to the usefulness of a GIS and the applications that are created based on GISs. Whether data conversion is done in-house or by an outside vendor, the quality assurance process is key to the long term reliability of the resulting database. The database created for a GIS is large, generally containing hundreds of thousands of pieces of information. This information is spatially-oriented and contains attributes in related databases. Any quality assurance process must address both the spatial data and the attribute data. To verify quality within these broad parameters is a monumental task. Total Quality Management and Statistical Quality Control techniques provide a foundation upon which processes that can tame the quality assurance issue can be built. A Zero-Defect approach that allows organizations to actually build quality into the conversion process is explained in this paper. Also discussed are the ways in which this approach to data conversion can ultimately reduce both conversion costs and implementation times.

To build quality into the data conversion process, it is first necessary to define what constitutes a "quality" product. This is true whether the conversion process is taking place in-house, or is being done by an outside vendor. This definition of quality helps provide a better understanding of the specific target, making that target easier to achieve.

To be meaningful within the world of GIS, the term quality requires a measurable definition. It is not sufficient to simply equate quality to accuracy. Accuracy is defined as "In exact conformance to fact: Errorless." With respect to data, we typically equate accuracy to line following, or capture of attributes. However, "quality" frequently involves subjective expectations that are not explicitly defined.

Quality in the context of an outside vendor means meeting the user's reasonable expectations expressed in quantifiable terms (usually as project specifications) the first time around. Thus, quality can incorporate not just accuracy as regards data, but scheduling, project reportingevery interaction and communication within the conversion process, including the data. This definition of quality reflects the true way in which vendors are evaluated in an industry in which meeting deadlines is nearly as important as meeting specifications. Quality in this context does not mean delivering a Mercedes Benz instead of a Yugo. In other words, it does not mean delivering greater accuracy than required, or exceeding specifications. Instead, it means delivering data which precisely meets the specifications of the user.

Quality for those who are converting data in-house means setting and meeting reasonable expectations. The same standards that one would set for an outside vendor should be established internally, and these should incorporate accuracy, as well as cost and scheduling.

Typical QA Processes

After the specifications and expectations are defined, the conversion process begins. Quality assurance steps are usually separate phases in the process, taking place after a map has been converted.

Quality assurance processes have typically involved inspecting 100 percent of the work completed. When errors are found, some percentage of the data has been sent back to previous steps to be re-worked. The data has then looped around again to the QA process, where another 100 percent inspection has been done.

If this process is taking place with an outside vendor, yet another QA process is generally initiated by the user upon receipt of the data. Frequently, the customer initiates another 100 percent inspection process. Errors are often found during this inspection, and some data is returned to the vendor for re-work. This data must again pass through the QA process, both on the vendor side and on the user side. If all goes well, the loop finally closes when fewer than "X" number of errors are foundor when everyone is exhausted. The standard error ratio in the GIS production industry is five errors for every 1,000 bits of information processed.

So What's the Problem?

The amount of time the typical QA process takes is the most obvious problem. However, the real problem is that it does not work. If 100 percent inspection really worked, it would not have to be done again and again. And we know it does not work because errors are frequently found in data that has already been inspected and found to be "error-free."

Why does this occur? First of all, inspecting spatial data is no easy task. Compare it to "finding Waldo" in a tri-county area, or proofreading the entire Encyclopedia Britannica without a spell-checker. Someone will always find another mistake, no matter how assiduous one is. Thus, if the goal is accurate data, this is not a good methodology.

Another problem with this approach is that all errors must be communicated to the vendor or the in-house technician. This usually takes the form of lists such as "should be a 6 instead of a 7," and so on. Errors can easily be made both while compiling these lists and while making the corrections. Each item that requires re-work implies time, and thus expense. And, since conversion technicians are re-doing work they have already done, rather than moving on to the next data to be dealt with, it is easy to see how this can also impact the schedule: one week to convert, two days to QA, one week to re-work, one day to inspect, one week for user inspection, one week for re-work, two days to inspect, and so on.

But the greatest cost of all cannot be measured in days, or weeks, or dollars. What about all the errors that were not caught? And what of the cost of the decisions based on those errors? Finally, consider the legal liability this implies, not to mention the lack of confidence such errors can generate regarding the quality of the entire database.

How Can This Process Be Improved?

Very simply. Do it right the first time. A "Zero-Defect" approach to the conversion process accomplishes this. The only way to achieve quality data is to design a process that ensures that it is created accurately the first time and does not rely on a 100 percent inspection process. Using this process management approach, SMARTSCAN has managed to reduce the average error rate from 5 per 1,000 bits of information processed to just 5 per 500,000 bits. To explain why this works, let's look at a simple example. The manufacture of red paint provides many useful analogies. Using this analogy, it will be easy translate this approach to the more complex world of spatial data creation.

One method of manufacturing red paint would be to hire ten people, give them all the ingredients and send them off to produce paint to match a color card. The quality control would take place at the end of the process when each can of paint would be compared to a master color card. Is each can close enough? Some cans would meet the standard and some would not. This is obviously a very haphazard process and requires a 100 percent inspection process.

The customer purchasing eight gallons of red paint would also need to inspect each one to make sure they all appeared to match, perhaps opening as many as fifteen cans before finding eight that he or she considered close enough. This is because each of the ten individuals hired to make the paint used a slightly different process. Most customers would soon doubt the reliability of the cans of paints matching and probably decide not to use the product.

To correct this problem, a precise method must be defined. The order in which each ingredient is added must be spelled out, the lighting in the plant must be standardized, the temperature must be specified, how long each can is stirred must be statedevery variable which might affect the color of the paint must be taken into consideration as part of a complete process.

The inspection procedure could change dramatically once the process of creating the paint is under control. Once there is reasonable confidence in the process, it would be possible to inspect every other can. Then perhaps only every fifth can would need to be opened. Then every tenth can. Statistical sampling formulas dictate the percentage of cans that need to be inspected.

What if there are still errors?

Once the process has been controlled, the paint might still change color slightly. There can only be two reasons for this and both are easily remedied.

The first and most typical way in which this might happen is if a variable within the process was missed in the original design of the process. For example, perhaps over time, dirt has collected under the rims of the cans. The solution is obviously not to send the one bad can of paint back. It's not even sufficient to send the whole batch back. The process itself must be analyzed and changed to include the step "thoroughly wash cans before using," and everyone involved must be educated regarding the change made to the process.

The second possibility is that someone did not follow the process precisely. This can be prevented through training and discipline.

The Zero-Defect Strategy

For process management to work, it is essential that every corporate or administrative level subscribe to a Zero-Defect strategy. Otherwise, it will be easy to say, "Well, one out of ten cans is not bad." Even if the goal of Zero-Defects is never achieved, it is necessary to always strive for this level of quality because it forces continual improvements to the process.

This effectively converts errors into opportunities to improve the process. As though this were not beneficial enough, consider the savings during the final QA process alone. No longer a 100 percent inspection ritual that takes place at the end of the paint manufacturing process, quality assurance is built into the process itself, and statistical process control is limited to some small percentage of the total number of cans. Schedules are met or exceeded, implying further savings for all concerned.

How to Achieve Zero-Defect Data Conversion

This process management approach can be applied within the complex environment of GIS as effectively as it can be used within the simple world of red paint. Moreover, it is equally useful in data conversion, data maintenance and data handlingvirtually any aspect of GIS data management regardless of where the work is performed.

The steps to achieve Zero-Defect are as follows:

Define the specifications and the expectations for the project.
Define the overall process flow required to create the end product.
Divide the data process into phases. This makes it possible to look at smaller and smaller pieces of the process.
Define the expectations for each step. It is helpful to ask "What is expected of this step?" about each step in the process. Each step then defines the specifications which the previous step must meet before delivering the data to it. This is how QA is incorporated into every step of a processby defining the specifications of every single step.
Ask each person in each step of the process, "What are the tasks within each step that, if completed, will produce Zero Defects?" Have them write these tasks down in order and disregard at first how long it takes to complete these tasks.
One of the issues that is sure to come up is, how detailed should this listing and description of tasks be? The answer is simple: Only detailed enough to produce the same result every time, regardless of who is completing the task. Leave out all information that is not necessary to achieve this end. For example, it is not sufficient to state, "Stick the label on the can" (returning briefly to the red paint analogy), but it is not necessary to specify "Pick up label with right hand and..." Perhaps something along the lines of, "Align upper left hand corner 1/2" from the rim and..." will turn out to be sufficient. There is an element of trial and error to this stage in the implementation of process management.
Start up the process. Every time an error is detected in any step of the process, stop the process (at SMARTSCAN, a bell rang throughout the production area). Analyze the error, change the process in such a way that it cannot occur again, document the change, and inform the entire crew of the change.
Slapdash, makeshift changes are not sufficient. The changes made to the process may be simple, but they must address the true flaw in the process that caused the error.

Conclusion

Usually, when someone says, "From now on, we're going to concentrate on quality," the response is, "How can you afford to?" The perception has been that high quality is somehow more costly than poor quality. But if a poor quality product cannot be used and is unacceptable, poor quality actually becomes far more costly. Then it is appropriate to ask, "How can we afford not to concentrate on quality."

By focusing on the process, striving for Zero-Defects and transforming every error into an opportunity to improve the process, it is possible to ensure consistent, across-the-board quality on every aspect of every conversion project all of the time.

Mickey A. Fain is the founder, President and CEO of SMARTSCAN, Inc. in Boulder, Colorado. SMARTSCAN is the leading full services GIS data conversion services company, providing data conversion, data integration, quality assurance and conversion project management services to government agencies, natural resource companies, public utilities and system integrators.