By Avis L. Webster and Kristi Lombard

Abstract. The short life cycles of GIS technology products, compounded by an increased complexity in job roles, heighten the demand for lifelong GIS training. Trends in GIS training include shifts from training to performance and from training to lifelong learning. Increasingly it is essential to demonstrate training outcomes, measure performance outcomes, deliver specialized training, emphasize knowledge management, and rapidly develop and deploy training. Efforts to cut GIS training costs underscore the need for accurate evaluation and measurement methods. This paper discusses the use of evaluation and measurement techniques to promote continuous improvement in the cost and performance effectiveness of GIS learning.

Our paper discusses a topic near and dear to our hearts. As GIS professionals, we (the authors, but you too?) often experience an persistent, overwhelming sense of urgency to re-tool and re-train in order to perform quality work competitively in the matrix of ever-changing technological, methodological and organizational changes. The last ten years have witnessed changes in the GIS Industry on a massive scale. Changes from command-line driven applications to GUI applications which increasingly have to be delivered across the Net using client-server architectures to exploit data from enormously complex data warehouses and Enterprise RDBMSs on new generations of hardware according to rapidly evolving standards. Within this ten-year timeframe, we have been in danger of becoming obsolete end-users of GIS technology. Those of us who continued to make our living in GIS have had to acquire whole new skillsets. In the face of the pressure to maintain our status as valuable employees, we are compelled to continue training and learning. We see no end in sight; hence the notion of lifelong learning in GIS.

In a broader context, we find ourselves in the knowledge-based economy of the post-technological Information Age, where knowledge and skills have become the key factor in competition and where knowledge is the only lasting resource of competitive advantage. As we approach the millennium, concerns about the decline of Western firms, technological change, social change, market drivers, demographic changes, and the increasing globalization of business are fueling a "desperate quest" for new approaches to management and organization. One of the latest outcomes of the search for new paradigms in management is the theory and practice of the learning organization. The learning organization is a key means of adapting to and shaping an increasingly evolving and complex environment because a learning organization mobilizes the learning of all its members in a process of continuous self-transformation. Knowledge management, is the next logical step beyond the learning organization. (Starkey 1998, Demchenko 1999, DfEE 1998). Within this framework, many enterprises are positioning themselves as being Knowledge Based. In order to realize the goal of being knowledge based, these organizations must become learning organizations exercising knowledge management.

Concurrently, the Information Technology industry faces a problem of global proportions: If the IT skills and labor shortages aren't addressed, many CIOs fear national IT sectors will lose their competitive edge, economies will suffer and innovation will slow (Busse and Brandel 1999). So it’s no surprise that training budgets are on the rise. The most significant trend in the marketplace is a shift from training (skills acquisition) to improving performance. Number two is computer skills training and the third most significant trend is a shift from training to learning (ASTD 1997, Ernst & Young LLP 1999). According to Ernst & Young LLP in 1997 companies spent $18 billion worldwide on IT training alone. The same Webpage cites research published by International Data Corp: that number is estimated to reach $27.9 billion by 2001. Training expert Jack J. Phillips (1994) estimates that total spending on training in the U.S. rose 7% in 1993 to about $30 billion. Training for Information Systems professionals is a significant part of that.

The Return on the Investment. Bits and bytes from the Ernst & Young Webpage cite a study entitled Competencies and Competitive Edge by Watson Wyatt Worldwide: Companies that link skill development to business strategy have a 40% higher shareholder return than companies that don’t. The ASTD reports the results of another survey based on a sample of 40 publicly traded firms in a broad range of industries. This study provides preliminary evidence indicating that a relationship between training expenditures per employee and measures of financial performance. (Bassi and McMurrer 1998). During the period of the study, the change in market-to-book ratios showed greater increases for companies that ranked in the top half of training expenditures. Publicly traded firms that spend more money on training tended to have higher net sales and gross profits per employee and were more highly valued on Wall Street. According to the Eric Rolfe Greenberg, Director of the American Management Association (AMA) International Research Reports, companies are 80% more likely to increase employee productivity with training (AMA 1998).

Despite the labor and skills shortage, and preliminary evidence that training is positively correlated with profitability, there is an increasing demand towards the need to measure the return on investment on human resource development (Malaysia HRD Online 1998, AMA 1998). The pressure for measuring ROI in training is increasing for the following reasons:

• Training and HR personnel, both inside and outside Information Systems, are increasingly accountable to show how investments in training can lead directly to business objectives like increased revenue and higher productivity. In 1996 93% of training professionals stated that they are under increased pressure to demonstrate return-on-investment (ROI).
• In an increasingly competitive business environment all functions including public relations, marketing, purchasing, etc. are accountable for profitability.
• Management requires ROI information in a form and language that they understand i.e. what is the return on investment on training and development? The thinking being "What is measured, can be managed."
• Managers are facing a make-versus-buy issue when it comes to retooling their staffs. Managers need to know the ROI for the significant investment in retraining an IT professional in order to evaluate training benefits to the business versus outsourcing or bringing in new people.

In 1959, Donald Kirkpatrick created what is still the most widely used method of evaluating training programs. An ASTD survey indicates that 67% of organizations that conduct training evaluations use the Kirkpatrick method to the degree that "level 1" and "level 2" have become part of the training lexicon. Nearly all organizations perform Level 1 and Level 2 evaluations, and many organizations perform Levels 3 and 4 evaluations. (Stone and Watson 1999). Kirkpatrick's four levels of evaluation are summarized in Table 1.

Table 1. Kirkpatrick’s four levels (Alliger, et al. 1998, Tannebaum 1998)

Recognizing the prevalence and simplicity of Kirkpatrick’s typology, Alliger, et al. (1997) conducted an effort to analyze prior training effectiveness studies in order to provide more specificity about the levels. Their research was conducted with the USAF Armstrong Labs and the University at Albany and was the recipient of the American Society for Training and Development (ASTD) 1997 Research of the Year Award.

Alliger, et al. examined the research literature for studies that reported correlations among more than one measure of training effectiveness. Using "meta-analytic" procedures to combine results across 115 correlations from 34 prior evaluation studies, the researchers developed an augmented framework to provide clearer distinctions within and between frameworks. (See Table 2.)

Table 2. Comparison of Frameworks (From Alliger, et al. 1997)

Results. Due to a limited number of studies that reported Level 4 results, they limited their findings of the Levels 1 to 3.

The research examined reliability and correlational results. "Reliability" refers to the consistency or repeatability of a measure, for example, the authors explain that a scale that is consistently five pounds light is reliable, but not accurate. Low reliability places a ceiling on accuracy (e.g. a scale that reads the same weight differently intermittently can have limited accuracy). The researchers saw stronger correlations between measures when both measures were related to training content.

Based on their results, they present several recommendations for workplace learning and performance professionals:

• When using reaction measures to infer whether people will transfer newly trained skills to the job, include utility type reaction items in the reaction measure. Utility reactions predict transfer as well or better (but not perfectly) than affective reaction measures and most measures of learning.
• Use utility reaction measures closely related to the content of the training when appropriate.
• When using both affective and utility reaction measures to predict transfer, examine the utility reaction measures separately.
• When training is designed to teach behaviors or observable skills, use a test that demonstrates the behaviors or skills rather than a knowledge test. Behavioral demonstrations predict subsequent performance better than knowledge tests.
• For critical, high-visibility programs, use a mix of evaluation measures to obtain a more thorough picture and better diagnostic capability. The authors of this study recommend that you get answers to several questions, such as:
• Did trainees like the course (affective reactions)?
• Did they think they would be able to apply it when they returned to work (utility reactions)?
• Did they know what to do at the conclusion of training? (immediate learning)?
• Did they retain it later (knowledge retention)?
• Could they do it at the conclusion of training (behavior/skill demonstration)?
• Did they perform it on the job (transfer)?

Dr. Jack Phillips (1997) adds a fifth level in his model for evaluating training results – Return on Investment (ROI). ROI is the dollar value of benefits obtained by an organization over a specified time period in return for a given investment in a training program. An ROI analysis is not complete until the measured training results have been converted to monetary values and compared with the cost of the program. This shows the true contribution of training.

It was once considered impossible to measure the ROI of training, but now many organizations are doing so. It is a difficult and complex process that is dependent on a long-term perspective. Engaging in discussions with cost accounting experts can be helpful, and once the process of measuring ROI is begun, the process will improve. This section describes the steps in calculating the ROI, techniques for increasing the ROI and some challenges in applying ROI analysis to measuring training benefit.

The framework shown in Figure 2 serves as a simple and practical tool for developing ROI; from data collection to calculating the actual monetary return. ROI is calculated by determining the net benefit of the training program over the total cost of the training program.

Figure2. Flowchart for calculating ROI (From Malaysia HRD Online 1998)

The following steps outline the process for calculating ROI:

• Collect level-4 evaluation data. The question to ask here is "Did on-the-job application produce measurable results?" (This is addressed in the section above, the following steps are outlined in this section).
• Isolate the effects of training from other factors that may have contributed to the results.
• Convert the results to monetary benefits.
• Total the costs of training.
• Total the benefits of training
• Compare the monetary benefits with the costs.
• The non-monetary benefits can be presented as additional-though intangible-evidence of the program's success.

Training results can be divided into hard data and soft data (Phillips 1997).

Hard data are the traditional measures of organizational performance, such as labor cost, production rates and sales. Hard data can be used to measure output, quality, time and cost. They're objective, easy to measure, and easy to convert to monetary values. Management tends to find hard data highly credible. Hard data is available in most types of organizations, including manufacturing, service, not-for-profit, government, and educational.

Soft data is usually data that is behaviorally driven, such as absenteeism and turnover rates. This data is more difficult to measure, difficult to quantify and more subjective. It is difficult to convert to monetary values. Management tends to find this data less credible as a measure. For training that focuses on developing "soft" skills (e.g., communication skills), soft data is often required as a measurement tool.

The Table 3 provides examples of hard and soft data.

Table 3. Phillips' examples of hard and soft data.

2. Determine the value for each unit. This is easy for hard data and difficult for soft data

The following list of benefits, adapted from Shepard (1999), can be used as a guideline for calculating benefits. Forecasting and measuring benefits – these categories are not mutually exclusive. Care should be taken to avoid including the same basic benefit more than once.

Labor savings – training results in less effort to achieve same levels of output.

• Reduced duplication of effort
• Less time on mistakes
• Faster access to information
• Time savings - each operation, less supervision, fewer reworks
• Labor savings – less overtime, temp help, can downsize

Productivity increases – additional output can be achieved with the same level of effort. (implies that increased output is a goal. If not, then better to express benefit in cost savings)

• Improved methodologies – reduce required effort
• Higher skill, faster work
• Higher motivation, increased effort
• Convert improvements into hard dollar amount.
• Improved productivity – faster work, more units produced

Other cost savings – things other than savings related to labor.

• Fewer machine breakdowns, lower maintenance costs
• Lower staff turnover, lower recruitment and training costs
• Reduction in bad debts
• Better workplace moral – lower turnover rate, which means lowered hiring costs
• Improved health and safety – fewer accidents, reduced legal and insurance costs
• Better morale – less turnover, strikes, absenteeism
• Improved quality - fewer rejects, bigger sales, better products

Other income generation – new income as a direct result of training

• Higher success rate winning contracts, increased sales
• Sales referrals
• New product ideas, resulting in new products

Each training benefit outlined should be converted to monetary value. Two examples follow:

Time savings = number of hours saved * average hourly salary

Production gains = monetary value of extra goods produced

The cost of training includes one-time costs (development of course, equipment), and recurring costs (workbooks, room rental, trainer wages, and productivity loss). The following list of costs, adapted from Shepard, broken down by functional task, can be used as a guideline for calculating training costs. Keep in mind that this list is not all-inclusive and that other factors may apply to your specific organization.

Design and development costs – the costs of creating the training program.

• Internal days of design and development
• Cost of external designers and developers
• Other design and development costs (copywrite, travel, expenses)
• Cost of COTS products and materials

Promotional costs – the costs of promoting or internally ‘advertising’ the training

• Internal days of promo activity
• Cost of external agencies
• Other direct costs (posters, brochures, web pages)

Administration cost – the costs related to the time taken by the training department to administer the training

• Hours of administration required
• Direct administration cost (registration materials, registration fees)

Faculty costs – the costs related to the time taken by the training department to deliver the training

• Number of students who are going through program
• Hours of group training
• Hours of one-on-one training
• Hours of self study training
• Additional faculty hours (preparation time, reviewing homework time)
• Faculty expenses (travel, room and board)

• Cost of training materials
• License cost of COTS products

• Training rooms
• Open labs
• Equipment

• Travel, room and board
• Opportunity cost of work not being done

Evaluation costs – the time spent evaluating the training and performing ROI analysis

ROI is essentially the comparison of the monetary value of the benefits and costs. There is more than one method of comparison.

The simplest comparison is to subtract the costs from the benefits. The difference, then, is the ROI:

Another method, used by Shephard (1999), computes a direct ratio of benefits to costs, expressed as a percentage

The third method, used by Chase (1997), uses a slightly different ratio, calculating the difference in benefits and costs, and normalizing by the training cost

The latter two ROI calculations provide information on the percentage return earned over a specified period as a result of investing in a training program.

Another related measure is the Payback Period (Shephard, 1999). This is a measure of how long it will take (measured in months) before benefits of the training match the costs incurred. This is when the training ‘pays for itself’

If payback period is low (a few months), then management will be more encouraged to make investment. Also, no arbitrary time period is required.

The following example, adapted from Shephard, demonstrates the methods of calculating ROI.

Table 4. Listing of costs and benefits of a training program

Calculations:

There are several things a company can do to maximize the benefits of training. The following suggestions are outlined in Chase (1997).

One of the most effective methods of increasing the effects of training is to prepare employees with a pre-training briefing. This will provide a forum where management can tell trainees the purpose and objectives of their training, why they were selected, and what business need you hope to meet.

The purpose of training is to transfer skills learned back to the job. After training, new skills need to be nurtured until they become habits. This time is the most critical. Techniques learned in training may not be applied because coworkers think ‘they don’t have time’ or managers who don’t cooperate. Management support at this time will have more influence on whether training has any benefit than anything the trainer can do (Chase 1997). It is suggested that management ask employees what they learned, have them develop an action plan to implement what they learned, and follow up on it. Employee rewards also provide incentives. Rewards can be in the form of financial bonuses, certificates, acknowledgement or promotion.

First, decide which programs to evaluate for ROI. For large companies, it may be necessary to develop criterion for what training programs are most important to track. For example, management may wish to track the most expensive or controversial training programs, or those that run most frequently and involve the most people.

As ROI analysis is not an exact science, it is difficult to isolate the effects of training. There are many influences at work. For example, a training class may coincide with a new marketing gimmick, and it is difficult to pinpoint whether increased sales are related to training and which to the marketing. It is also difficult to know when to measure results, as they may accumulate over months or years.

Assuming that benefits continue to accrue some time after training is completed, it is important to specify an appropriate time period for ROI analysis. Choose a period that fits well with the organizations planning cycle (one year, two years). Alternatively, a period that corresponds to the lifetime of the benefit, can be used. This may be more difficult, as you would need to know how long the average student stays in a position where they use the skills being taught.

Some practitioners emphasize that it is not necessary to prove absolute cause and effect. It is sufficient to look for indicators that training has improved performance and productivity.

Lastly, how do you know what is a good ROI? For many companies, a good goal is 25%. For other companies, they believe it is worth it even if they break even. Some management training gets scores of 100% or more, since you can track the manager’s influence on the whole team, not on just one person.

Lifelong learning is increasingly important to the competitiveness of our nation and the success and well being of our individual citizens. Nevertheless, increasing ROI is the bottom line for management. The challenges to justifying investments in training are significant, and a real commitment from the enterprise to connect training and learning to the enterprise mission through evaluation can provide solutions.

American Society for Training and Development (ASTD). 1996. The ASTD Training and Development Handbook: A Guide to Human Resource Development, 4th Edition, R. L. Craig (Ed.), McGraw Hill.

Robinson, D. G. and Robinson, J. C. 1989. Training for Impact: How to Link Training to Business Needs and Measure the Results. Jossey-Bass Publishers, San Francisco.