9. Evaluation process
Note: This section comes directly from NGTSM06, Volume 3, section 2.20
Steps
9.1 Determine what is to be evaluated, the purpose and audience of the evaluation
9.2 Determine the standards of comparison
9.3 Determine the depth and frequency of post-completion evaluation
9.4 Appoint evaluators
9.5 Collect and assess data, prepare report
9.5 Undertake ex-post CBAs of selected initiatives
9.1 Determine what is to be evaluated
First, determine the subject of the evaluation. It could be a single initiative, a group of initiatives that have something in common, or an entire program of initiatives. It is also important to be clear on the purpose of the evaluation and the audience.
Second, decide on the stage, or stages, of the decision and implementation processes that are to be evaluated. Table 1 lists stages that might be evaluated.
9.2 Determine the standards of comparison
Post-completion evaluations can be applied to the process that led to the outcomes and to the outcomes themselves.
Process reviews look at how the outcomes were achieved. The outcome reviews can point to whether or not a process has worked well. But bear in mind that a faulty process can still produce a good outcome, due to luck.
Outcome reviews involve comparing actual with predicted outcomes. For an outcome review, the standards of comparison are external standards of desirable attributes of outcomes (e.g. correct CBA methodology) and benchmarks or forecasts established during a previous process. To streamline post-completion evaluations, set the benchmark levels during the appraisal and design stages (e.g. construction costs, physical quantities and unit costs of inputs, timing of construction, operating costs, demand levels, revenues, benefits, environmental impacts).
Carry out process and outcome reviews simultaneously for a single stage so that any process-related reasons for successes or failures can be explored. The outcome review at one stage can point to process issues in earlier stages.
Table 1 lists the stages at which process and outcome reviews can be undertaken and shows sources for standards of comparison for outcome reviews.
| Process review | Outcome review | |
|---|---|---|
|
Identification, consideration of options |
Initiatives identified |
Government strategy documents, stakeholder views of options |
|
Appraisal |
CBA, SMT, other analyses, Business Case |
Guidelines, knowledge of correct methodologies, technical publications, government strategy documents |
|
Recommendation |
Recommendation for or against |
Business Case |
|
Planning and design, budget development |
Detailed specification of initiative, scope, cost estimates, risks during implementation stage |
Business Case, design guidelines |
|
Delivery |
Initiative in place, actual costs |
Business Case, plans and designs, budget |
|
Operation |
Performance |
Business Case |
|
Benefit evaluation |
Benefit achievement |
Business Case, Benefits Management Plan |
The timing of the evaluation matters. A stage in Table 1 can be evaluated only after the stage has been completed.
As also shown in Table 1, review and evaluation activities are undertaken both ex-ante (prior to commencing delivery) and ex-post (from delivery to the end of an initiative’s lifecycle). Post-completion evaluations are undertaken on an ex-post basis.
For the initiative operation stage, allow time to establish levels and trends with adequate certainty, and be selective about the variables assessed so as to avoid premature comparisons.
Following the initiative development stage, benefits evaluation can be undertaken to manage the realisation of planned benefits, as part of the post-completion evaluation. A benefit evaluation provides a comprehensive overview of both what an initiative has achieved and how this compares to what was expected (see Appendix D for a comparison of post-completion evaluation and benefits evaluation).
9.3 Determine the depth and frequency of post-completion evaluation
Subject all initiatives to some basic level of post-completion evaluation using benchmarks set during the appraisal and design phases.
To make best use of limited resources available for post-completion evaluation, for detailed evaluations select initiatives that are:
- Large
- Appear to have gone badly or exceptionally well
- A recurring type of initiative
- Particularly risky, including pilot initiatives for testing innovations
- Strategically important
- Long-term, undertaking interim evaluations (such as annually)
- Staged, undertaking evaluations of each stage
- Programs involving a series of smaller initiatives.[1]
9.4 Appoint evaluators
Independent evaluators should be appointed to avoid bias and create objectivity and credibility. For smaller initiatives, it may be reasonable to use internal resources to undertake the evaluation.
Do not allow those responsible for the initiative to evaluate their own work.
9.5 Collect and assess data, prepare report
Having discovered where process or outcomes are good or poor, look for the reasons for the results. Separate consequences of internal management and planning processes from impacts of external factors. The aim is to identify lessons for the future: do not allocate blame for past mistakes or state what should have been done with the benefit of hindsight. Where external factors had an adverse impact, consider whether actions should be taken in the future to mitigate such occurrences.
Be wary of making recommendations for major changes to processes based on a single post- completion evaluation. Keep the recommendations in broad terms, suggesting a direction for change.
9.6 Undertake ex-post CBAs of selected initiatives
A post-completion evaluation may extend to an ex-post CBA. An ex-post CBA compares the actual outcome with a hypothetical base case in which the initiative does not exist. The central question is: ‘With hindsight, how strong was the economic justification for the initiative?’
An ex-post CBA can seek to address the following questions:
- Have the forecast benefits been realised?
- What are the magnitudes of deviations between forecasts and actual outcomes?
- What caused the deviations?
- What lessons can be learned to improve future CBAs and thereby make it a more effective decision-making tool?
The following points are important in undertaking an ex-post CBA:
- Ex-post CBAs are concerned with the overall outcome of the initiative, not individual components.
- In ex-post CBAs, the base case is no longer observable. Challenges remain in constructing a plausible base case. The base case is not the 'state of the world' before the initiative commenced, although pre-implementation information is used for estimating base case characteristics.
- Ex-post CBAs involve analytical effort to estimate benefits and costs. Some elements are already known with complete certainty because they have occurred in the past. Planning, design and construction costs, infrastructure operating costs, demand levels, accident rates and physical levels of externalities are known with certainty up to the present time for the project case of an ex-post CBAs. With the exception of planning, design and construction costs, these all have to be estimated for the base case and projected into the future for both the base and project cases.
The original CBA could be used as the basis for the ex-post analysis by replacing the assumptions (costs, demand forecasts) with actual outcomes or updated forecasts. This type of exercise serves a secondary purpose of providing a review of the original CBA.
An ex-post CBA may be accompanied by an assessment of the broader regional, environmental, social and financial impacts of the initiative.
Findings from ex-post CBAs can be useful in improving the practice of CBAs of future initiatives.
Benefits evaluation can provide an input to ex-post CBAs by enabling access to the necessary information.
Appendix B provides an overview of the methodology for undertaking an ex-post CBA.
9.7 Example evaluations
Appendix A discusses specific evaluation examples drawn from Austroads (2005).
[1] Based on Austroads (2005d, p.4)