Measuring+Success

What metrics and analysis tools are being used to prove the value of connected intelligence initiatives to organisations? What tools might be used?

Joe Cothrel & Co's [|Return on Participation] 2000

Ross Mayfield's Power Law of Participation

Kirkpatrick's levels of evaluation: ASSESSING TRAINING EFFECTIVENESS often entails using the four-level model developed by Kirkpatrick (1994). According to this model, evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation. Thus, each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and time-consuming analysis.

Level 1 Evaluation - Reactions Just as the word implies, evaluation at this level measures how participants in a training program react to it. It attempts to answer questions regarding the participants' perceptions - Did they like it? Was the material relevant to their work? This type of evaluation is often called a 'smilesheet'. According to Kirkpatrick, every program should at least be evaluated at this level to provide for the improvement of a training program. In addition, the participants' reactions have important consequences for learning (level two). Although a positive reaction does not guarantee learning, a negative reaction almost certainly reduces its possibility.

Level 2 Evaluation - Learning To assess the amount of learning that has occurred due to a training program, level two evaluations often use tests conducted before training (pre test) and after training (post test).

Assessing at this level moves the evaluation beyond learner satisfaction and attempts to assess the extent students have advanced in skills, knowledge, or attitude. Measurement at this level is more difficult and laborious than level one. Methods range from formal to informal testing to team assessment and self-assessment. If possible, participants take the test or assessment before the training (pretest) and after training (post test) to determine the amount of learning that has occurred.

Level 3 Evaluation - Transfer This level measures the transfer that has occurred in learners' behavior due to the training program. Evaluating at this level attempts to answer the question - Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner? For many trainers this level represents the truest assessment of a program's effectiveness. However, measuring at this level is difficult as it is often impossible to predict when the change in behaviour will occur, and thus requires important decisions in terms of when to evaluate, how often to evaluate, and how to evaluate.

Level 4 Evaluation- Results Level four evaluation attempts to assess training in terms of business results. In this case, sales transactions improved steadily after training for sales staff occurred in April 1997.

Frequently thought of as the bottom line, this level measures the success of the program in terms that managers and executives can understand - increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, and even higher profits or return on investment. From a business and organisational perspective, this is the overall reason for a training program, yet level four results are not typically addressed. Determining results in financial terms is difficult to measure, and is hard to link directly with training.

Methods for Long -Term Evaluation Send post-training surveys Offer ongoing, sequenced training and coaching over a period of time Conduct follow-up needs assessment Check metrics (e.g. scrap, re-work, errors, etc.) to measure if participants achieved training objectives Interview trainees and their managers, or their customer groups (e.g. patients, other departmental staff)

Source: Encyclopedea of Educational Technology as originally submitted to the Forum by K.Sathya Narayanan Source: http://www.isixsigma.com/dictionary/Kirkpatricks_4_Levels_of_Evaluation-626.htm


 * move Source: to References section - what is the need for the repetition of this information here? JM why not link directly???**