Instructional Design, Assessments, Evaluations

When providing training services, Turbo Federal uses the dynamic, flexible ADDIE model – traditionally used by instructional designers and training developers – as a guideline for building effective training and performance support tools. Let’s explore the five phases – Analysis, Design, Development, Implementation, and Evaluation – and what is expected within each phase.

During the Training Analysis phase, Turbo Federal works closely with the stakeholders to define the current training gap by conducting a training needs study. This study answers the most important question, “Is training the answer”? The heart of Instructional Design is the analysis phase. After thoroughly conducting this study, we partner with our clients to choose the best training model based on our findings. Often times, spending your organization’s limited resources on training is not the answer; the gap may be closed by creating new job aids or realigning job tasks.

If training is determined as the best course of action, our team of Instructional System Designers works hand-in-hand with the stakeholder’s subject matter experts to gather additional data. We identify general characteristics of the target audience including prior skills, prior experience, and basic demographics; identify characteristics directly related to the skill to be taught; and conduct analysis of the performance and learning settings. At the end of the Training Analysis phase, we provide you with a complete results report including:

• A goal statement describing a skill, knowledge or abilities (SKA) that a learner will be expected to acquire
• Who the audience is and their characteristics
• Training objectives, the learning environment, and the learner’s existing knowledge and skills
• What a learner must recall and be able to do to perform particular task
• Identified areas and instructional materials that are in need of improvement
• Information about tasks to be completed, how learners will view the content, and the project’s overall goals through conducting workshops, job tasks surveys and/or visits to work center, to conduct job task analysis
• Definition of  new behavioral outcome, the types of learning constraints that exist, and the delivery options using pedagogical considerations
• Timeline for project completion

Turbo Federal, working in collaboration with subject matter experts and using the information gathered from the Training Analysis phase, begins to create the Training Design plan.  Tasks are then identified and broken down into more manageable steps.  We then determine the kind of activities required for the audience in order to meet the goals identified in the Training Analysis phase.

Turbo Federal follows a very systematic and specific method when designing training.  We use a logical, orderly method of identifying, developing, and evaluating a set of planned strategies targeted for attaining the customer’s training goals and executing these strategies with attention to detail.  We accomplish this by:

• Writing performance objectives that consist of a description of the behavior, the condition, and criteria.  The component of an objective that describes the criteria will be used to judge the learner’s performance.
• Documenting the project’s instructional, visual, and technical design strategy.
• Appling instructional strategies according to the intended behavioral outcomes by domain (cognitive, affective, and psychomotor).
• Creating storyboards.
• Designing the user interface and user experience.

The third phase, Training Development, is where the blueprints in the Training Design phase are assembled, where the developers create and assemble the content assets that were identified in the last phase.  Programmers work to develop and/or integrate technologies, and testers perform debugging procedures.  The project is reviewed and revised according to any feedback given.  Some activities include:

• Developing assessment instruments:  understanding the purpose of entry behavior testing, pretesting, post-testing, practice items/practice problems
• Developing instructional strategy:  Pre-instructional activities, content presentation, learner participation, and assessment
• Developing and selecting instructional materials such as lesson and instruction plans

Once Turbo Federal has developed a training program, it can be implemented.  This phase allows our team of Instructional Designers to test all materials to identify that they are functional and appropriate for the intended audience.  During this phase, a procedure for training the facilitators and the learners is developed:

• The facilitators’ training covers course curriculum, learning outcomes, method of delivery, and testing procedures
• Preparation of the learners includes training them on new tools (software or hardware) and student registration

Additionally, Turbo Federal revises instruction and conducts an in-depth analysis to identify poor test items or instruction.  This is also the phase where we ensure that the books, hands-on equipment, tools, CD-ROMs and software are in place, and that the learning application or website is functional.

The final phase is vital for our Instructional Design team because it provides the data used to alter and enhance the design of the courses that we create for our customers.   In this final phase, Turbo Federal ensures the materials achieved the desired goals.  The evaluation phase consists of two parts: formative and summative.

• Formative evaluation is present in each stage of the ADDIE process.  The ADDIE model is an iterative process of instructional design, meaning that at each stage, our designers assess the project’s elements through all phases of the design process and revise them as necessary.  This ensures that all phases of the design project are meeting sound Instructional System practices and our customer’s needs.
• Summative evaluation consists of tests designed for domain specific criterion-related referenced items and provides opportunities for feedback from the users.  Connecting all phases of the model are external and reciprocal revision opportunities.  Aside from the internal Evaluation phase, revisions should and can be made throughout the entire process.

Turbo Federal employs a team of experts who write training surveys following the Kirkpatrick method of training evaluation, and analyze and report training survey results.

Level 1 Evaluation:  Turbo Federal writes surveys that measure evaluation reaction which is how the learners felt and their personal reactions to the training or learning experience; for example:

• Did the trainees like and enjoy the training?
• Did they consider the training relevant?
• Was it a good use of their time?
• Did they like the venue, the style, timing, domestics, etc ?
• Level of participation
• Ease and comfort of experience
• Level of effort required to make the most of the learning
• Perceived practicability and potential for applying the learning

Level 2 Evaluation: When Turbo Federal designs training, we create the best methods of measuring the increase in knowledge or intellectual capability from before to after the learning experience:

• Did the trainees learn what was intended to be taught?
• Did the trainee experience what was intended for them to experience?
• What is the extent of advancement or change in the trainees after the training, in the direction or area that was intended?

Level 3 Evaluation:  Turbo Federal creates survey tools which measure how well the trainees applied the learning and changed their behavior.  This can be immediately and/or several months after the training, depending on the situation:

• Did the trainees put their learning into effect when back on the job?
• Were relevant skills and knowledge used?
• Was there noticeable and measurable change in the activity and performance of the trainees when back in their roles?
• Was the change in behavior and new level of knowledge sustained?
• Would the trainee be able to transfer their learning to another person?
• Is the trainee aware of their change in behavior, knowledge, skill level?

Level 4 Evaluation:  When training has been selected as the course of action, the customer needs to know if there has been a return on investment.   Turbo Federal answers those questions by conducting Level 4 surveys.   The goal of this survey is to ensure that the money or time spent on training has had the desired effect on the business or environment resulting from the improved performance of the trainee.  Measures would typically be business or organizational key performance indicators, such as:

• Volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organizational performance, for instance:
  • Numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc.