Fill out our online application and take an important step toward becoming a Messiah College Falcon!

Four Steps for Doing Program Reviews

The assessment movement in higher education has developed a general review of the process as a feedback loop. One way to conceptualize this process is represented in the following diagram:

Four Steps for Doing Program Review

While it may be appropriate to think of starting at one point in the cycle (e.g., Questions to be Answered), it is important to think of this as a continuous loop with the possibility of these activities taking place on an ongoing basis. For example, the gathering of evidence with regard to accomplishment of goals may happen each year or perhaps even more often rather than on some occasional or ad hoc basis. Likewise, programs may implement change on a more frequent basis than just when significant review activities are occurring.

If the program review is seen as part of this ongoing feedback loop, there is less need to think of it as requiring some sort of heroic efforts every five or ten years.

Nonetheless, in an effort to describe each step of the process, the narrative which follows begins with the top of the diagram (i.e., Questions to be Answered) and moves to each succeeding step in the process.

  1. Identify Questions to be Answered. Identify the significant questions to be answered in each of the three major areas of a unit review: purposes (goals/objectives, processes (environmental variables), and personnel.
    1. What objectives are most important to us? What do we want to find out about in this review process? Many support areas may have more objectives than can practically be assessed. It is better to choose a modest number of objectives which are important to the area and which can be managed successfully. Experts in the area of campus assessment might suggest choosing somewhere from three to five objectives to study.
    2. What do we need to now about the resources and environmental variables that support the work unit? Typically questions in this area will relate to the physical space allocated to the unit, support in the form of budget allotments, and statistics on the persons involved in both the delivery and receipt of program activities. It is important in this area to use comparative data from benchmark institutions and/or national surveys.
    3. What are the characteristics of the various personnel involved in the support area? It is important to take stock of the credentials and expertise of those involved in the area. In some cases it may be possible to use institutional resources to answer questions about personnel. For example, College-wide surveys may reveal information about satisfaction of students with their experiences in interacting with people in the support area. Data from the Student Satisfaction Inventory, for example, may provide useful information in assessing the level of satisfaction with the service provided through the work unit.

  2. Gathering Evidence
    1. How can we connect the areas of purposes, processes and personnel with evidence? It is important to make the step toward what might be “measurable” variables in the areas of purposes, processes, and personnel. If variables are in global or very abstract terms, then some work will need to be done to translate the variables into statements that lend themselves to some sort of measure or observation that is objectively available.
    2. Evidence is broadly defined to cover a wide variety of sources that can be marshaled to answer the questions about purposes, processes, and personnel. There should be multiple sources of evidence. The evidence should include some potential for comparisons to be drawn with other groups and institutions. Hopefully, evidence has been accumulated on a continuous basis and is not being gathered only every five or ten years. If this is the case, then the review will simply provide impetus for compilation and reflection on evidence that has been routinely collected. An important principle to be considered is to use existing sources of information so that the amount of demand-by-evidence gathering may be reduced.

  3. Making Meaning: Interpreting/Studying the Evidence
    1. This stage involves reflecting on the evidence: What does the evidence tell us? It is at this point that preliminary answers to the questions driving the review can be proposed. These reflections and preliminary answers are provided in a written report that will summarize the entire program review process. In general, the report will do the following:
      1. Explicitly connect purposes, processes, and personnel with evidence that has been gathered.
      2. Offer reflections by members of the review team/committee on the results of the review (this should involve a collective process of reflection shared by those responsible for the review as well as remaining members of the program unit)
      3. Statements about both successes and possible failures in accomplishment of purposes
      4. Reflections on which resources seem to be particularly helpful in attaining program goals and which resources may need to be modified or reallocated.
      5. Reflections on the allocation and characteristics of the personnel in the program unit.
    2. It is important to remember that the purpose of the unit review is formative. Honest, thorough analysis and reflection on the evidence is crucial to the success of the educational program review.

  4. Implementation
    1. The entire program review process will fail unless there is an indication that the information gathered in the process is used to implement change. In its charge to Messiah College, the Middle States Commission has insisted that our assessment process provide indications that we are using the information gathered in program reviews and other assessment activities.
    2. Thus, as part of the written report created for each support area or work unit, the program under review must provide a statement of specific action steps to be taken that are based upon the review.
    3. For example, the support area or work unit might:
      1. decide to revisit the mission and purpose of the work unit,
      2. decide to revisit objectives and/or outcome statements with the possibility of revision,
      3. decide to add, delete, or modify existing services,
      4. affirm current practices and explicitly recommit to those practices,
      5. decide to modify selective practices or work processes,
      6. decide to modify allocation of resources,
      7. initiate plans for new programs or initiatives,
      8. decide to discontinue existing programs or practices.
    4. Implied in the above examples is the fact that another important type of action step is the initiation of planning for the next five years (e.g., until the next program review). The report should make certain commitments to activities that will be undertaken in the next five years.
    5. As was the case for the step of reflecting on the evidence, the step of “response” or implementation should also have the support of all the members of the program unit under review. The action steps that are recommended should be owned by the program area. Implied in the submission of the report of the program review is the acceptance of its contents by the entire staff of the program.