The Peer Review Process
Instructional design is inherently complex and at times, messy. Peer review is defined as “the structuring of a process to allow peers to review each other’s professional processes and/or products with the goal of improving such processes or products.” (Woolf & Quinn, 2001). Peer review can help an instructional designer distribute workload, collect and validate institutional knowledge, and create engagement across stakeholders.
Textbook publishers rely on these types of groups extensively, and they do so for a reason. The feedback provided by group members in the early stages of a project can be extremely valuable. As the project matures, these groups can provide great insight into the student and instructor experience.
Creating a Peer Review Group
Peer review groups generally originate in one of two forms. A group can be put together to address a specific training need and serves as the training organization’s resource for the project. Alternatively, a single group can be created to provide feedback on multiple training projects, or even serve as an unofficial “board”. These groups should include people who are affected by the training being provided. A typical group might consist of:
- A member of the training staff, as facilitator
- Two subject matter experts
- Two instructors that have taught the program, or a similar one
- Two students that have recently completed the program, or a similar one
- Two supervisors, responsible for managing people affected by the program
At least two of the individuals should come from outside the training organization if possible. One option might be to include a faculty member from a local college in either a subject matter expert or instructor seat. Involving individuals outside the training department, or even from outside the company can build credibility for the finished product. Also consider the basic viewpoint the training organization will have on the subject matter. Try to bring in a couple of members that have been critical of current programs. Avoid loading the peer review group with people that will simply mimic the views of the training organization. One of the strongest benefits of the group is the ability to have honest, critical discussions.
Membership should be rotated, but not frequently. Some organizations make appointments concurrent with a curriculum update cycle. Others replace a percentage of the members annually so that fresh opinions can be heard. Remember that participants may take time to feel comfortable voicing opinions freely.
Although these groups are extremely valuable, be sure that the training organization retains control of the finished training project. Intellectual property rights are valuable, so consult with legal counsel and set firm guidelines related to the group. Formal agreements may be appropriate in some situations where sensitive information or trade secrets are being discussed.
The group’s purpose should be the free and open exchange of ideas related to the moderator’s agenda, or their assignments.
Peer review groups should meet on a regular schedule. Groups often work by teleconference or web-based meeting. Face-to-face meetings are not critical and can be costly; use technology to keep costs down. Document-sharing sites such as Acrobat.Com, Microsoft Office Live, or Google Docs can allow multiple users to edit a document. That type of flexibility is valuable and can allow everyone to get the most value from the actual meeting. These web applications also allow group members to work on projects at their leisure. For corporate users, be sure the use of such sites conforms to Information Technology (IT) policy, or work with the IT staff to provide an alternative solution.
Review the business or training need that is driving the project and set parameters for the group. In some cases the moderator may choose to make assignments for specific tasks, or simply set an agenda. Ensure there is a timeline for the group’s involvement. Even with the best participants, sometimes meetings can take unexpected turns or have “off” days. Show respect for the group’s time by keeping the meetings on-task and on-schedule.
Members of the group can be internal or external, but should be diverse enough to give honest feedback. Even if there are no active projects, sometimes simply brainstorming and talking about the current state of existing training programs can lead to discussion and positive change. Ask questions, have them review content and make suggestions.
Establishing peer review is beneficial to the instructional designer or training manager at the personal level, because it affects time. Time is a critical resource. Unlike working with subject matter experts one-on-one, the peer review group is capable of taking on individual elements of the instructional design process and bringing back work at a draft level.
The facilitator can take advantage of this in two ways. A good example might be development of student learning outcomes for a new curriculum. Instead of having the group revise a draft list, simply ask members to bring one or two outcomes into the meeting. Collect them and discuss the concepts present, not the grammar or technical elements of the outcome – those are easily fixed by the instructional designer. Repost the list prior to the next meeting, and gather feedback on which outcomes are the most important. In very short order, you’ve identified the critical elements for the curriculum in a minimal amount of design time.
As any development process continues, a second opportunity to distribute work opens up. As with any group, some members will have more time to devote to the process and specific areas of interest or knowledge. Allow members with time and interest to actively develop blocks of content. Keep their contribution process simple – don’t ask for fully developed presentations or notes. Members may also be able to submit relevant photographs or media assets, as long as they have appropriate intellectual property rights for the material submitted.
Peer Review and Institutional Knowledge
Most institutional knowledge resides in the minds and hands of an organizations functional managers and employees. As individuals transition jobs, management changes, or companies reorganize or are acquired, this knowledge can be lost. This knowledge loss is accelerating rapidly due to changes in organizational dynamics. (Ashkenas, 2013).
The peer review process is an excellent method for capturing institutional knowledge. As Ashkenas points out, stopping the loss of institutional knowledge is grounded in three areas: (1) building an explicit strategy that recognizes the role of this knowledge, (2) identifying the key areas of knowledge that needs to be preserved, and (3) using technology to capture the actual information. Using peer review groups actually helps address all three elements of preservation.
Many instructional designers and subject matter experts hear the statement “but that’s not the way it really works” from instructors, supervisors and students after a curriculum is released. The peer review process allows for supervisors and people that actually perform a job to have a role in the creation of training. The insights these individuals offer can add great depth to a training program and help instructors answer the question of “why it’s done this way” in the delivery. They also are a great resource for developing case studies and scenarios.
Peer review groups can be critical at creating engagement and end-user buy in to your training product. Two of the most significant barriers to transfer of training are the lack of management support and lack of peer support. (Noe, 2010). The use of a peer review group ensures that both supervisors and peers have not only been exposed to the curriculum, but have had a voice in it’s creation. A supervisor is more likely to positively reinforce training that he or she has helped develop.
By relying on a number of stakeholders rather than on subject-matter experts only, the content may also be more readily accepted by the students. Peer-reviewed content is often leaner and more focused than traditionally developed content.
Peer Review: In Closing
Peer review is not new to the instructional design process; most textbooks go through a number of similar processes to ensure the content represents accurate information. Peer review serves a valuable purpose to any training organization. The process helps ensure the content is not only representing best practices within the organization, but provides greater opportunities for the successful transfer of training.