Appendix A Example evaluations
Each of the examples cited in this appendix has been developed for a different purpose and, as such, none of them includes all of the elements that ideally would be incorporated into a comprehensive post-completion evaluation report. Nonetheless, the examples highlight the types of variables that are investigated during post-completion evaluations and what lessons can be learned from the process.
A.1 Adelaide’s O-Bahn Busway
The Adelaide O-Bahn Busway (pictured below) opened in 1986. The O-Bahn system involves the fitting of guide rollers to the front wheels of a conventional bus, which can then be driven either normally or on a guide-rail track that bypasses, for example, points of traffic congestion. A total of 12 km of Busway track were constructed, running between the edge of the Adelaide CBD and the north-eastern suburbs.
Figure 3: Adelaide’s O-Bahn Busway
Source: Wayte and Wilson (1989)
The main objectives of the initiative were to:
- Increase accessibility between the north-eastern suburbs and the CBD by significantly reducing travel times and improving the reliability of bus schedules
- Reduce congestion on the existing road network.
The O-Bahn post-completion evaluations as described by Wayte (1991) were focused on the performance of the O-Bahn system and the outcomes of the initiative. In addition to the performance of the O-Bahn system, a series of before and after studies were conducted to assess user responses to the system (Pak-Poy & Kneebone Pty Ltd, 1990; Denis Johnston & Associates Pty Ltd, 1988, as cited in Wayte, 1991). Specifically, the following variables were assessed:
Technical performance of the O-Bahn system. The technical performance of the O-Bahn system was evaluated against design expectations. Events such as stoppages caused by the guidance technology, ride quality and vehicle breakdown were considered.
Safety performance of the O-Bahn system. The number of accidents that occurred per kilometre of travel on the O-Bahn system was compared with the accident rate of conventional buses. This is similar to the ‘comparison of different groups’ method described earlier (Austroads, 2005), whereby conventional buses (a group among which no initiative implementation had taken place) were compared with O-Bahn buses (a group among which the initiative was implemented).
Financial performance of the O-Bahn system. The budget set for the initiative was compared with actual expenditure, and the estimated costs of some elements of a conventional Busway were compared with the costs of the O-Bahn to evaluate the financial performance of the initiative.
Patronage. One of the variables measured in the assessment of patronage impacts was passenger volumes on the Busway corridor prior to its conversion to the O-Bahn system compared with passenger volumes after the conversion. In order to account for exogenous impacts on these figures (such as population growth), changes in bus patronage in Adelaide during the same time period were used as a comparison benchmark.
In defining the focus of attention (purpose) for the evaluation, it was noted that it is important for post-completion evaluations to provide clues as to why measured initiative outcomes have occurred. To investigate why new passengers were attracted to the Busway, a survey of Busway passengers was undertaken.
Reductions in motor vehicle volumes. An objective of the O-Bahn initiative was to reduce congestion on the existing road network. Congestion was not investigated directly, but factors assumed to be related to traffic congestion were investigated. Based upon observed increases in patronage before and after the implementation of the O-Bahn system on the route, and on passenger reports of prior mode of transport, the decrease in the number of vehicles travelling the route was calculated.
Changes in travel time. An objective of the O-Bahn initiative was to reduce the time taken to travel between the north-eastern suburbs and the Adelaide CBD. Bus travel time measurements were taken before and after the implementation of the O-Bahn system so that it could be determined whether this objective had been met.
The post-completion evaluation of the O-Bahn, as described by Wayte (1991) employed comparisons of factors at different points in time (patronage and travel time before and after O-Bahn implementation) and for different groups (accident rates of O-Bahn and conventional buses). An attempt was also made to determine the reasons for some of the changes that took place (opinion survey of O-Bahn passengers) and to address initiative objectives (travel times and traffic congestion).
A.2 Road upgrades at Lauderdale, Margate and Devonport (Tasmania)
The former Tasmanian Department of Infrastructure Energy and Resources (DIER) undertook ‘debriefs’ up to six key initiatives each financial year, a process that involved a range of stakeholders, including initiative managers, contract staff, initiative designers and initiative sponsors. The aim of the DIER debrief process was to seek and communicate the learning opportunities that arose during the implementation of the department's initiatives.
The table below presents a summary of the implementation issues, identified through the debrief process, that were encountered during three road upgrade initiatives undertaken by DIER. Adherence to the predicted timelines, budget and technical specifications were considered key initiative success factors (DIER, 2004).
| Aspect of implementation | Issue encountered | Learning opportunity identified |
|---|---|---|
Initiative definition |
Initiatives are poorly defined at inception |
Initiatives should be well defined at the planning stage |
Financial performance |
Initiative budgets exceeded |
As above |
Design and technical specifications |
Difficulties working with service authorities |
Closer relationships between service authorities, designers and the department are required, and these relationships should be embodied under a memorandum of agreement executed at the planning stages of each initiative |
Management of landowner issues |
No clear process for consultation with landowners |
Define the roles and responsibilities of various team members in relation to consultation with landowners within each phase of an initiative |
Procurement |
Late award of contract date |
Review processes to bring works forward and take advantage of the full construction season |
Maintenance |
Poor handover of initiative from construction to maintenance phases |
Departmental standard design brief to include maintenance requirements/ objectives during the design phase Departmental maintenance personnel to be a part of the initiative team and attend initiative inception meetings |
These post-completion reviews have been used to provide learning opportunities that can guide improvements in the conduct of other initiatives - an important application of post-completion evaluations. For example, DIER - and its successor, the Department of State Growth - have used the learning opportunities provided by these post- completion reviews to improve the programming and delivery processes of all subsequent initiatives.
A.3 Department for Transport trunk road schemes
The United Kingdom Department for Transport (DfT) uses a cost-benefit analysis computer program (COBA) to establish the net present value of proposed trunk road schemes compared with a traffic scenario without the road scheme. The COBA program measures the benefits of road schemes as savings in journey time, vehicle operating costs and the value of reduced accidents. These benefits are assigned a monetary value which is then compared with the capital and operation costs of the scheme (Swift, 2001).
Knight et al. (1996) describe the post-completion evaluation of 11 trunk road schemes during which the actual Net Present Value (NPV) was compared with the NPV predicted using the COBA program, prior to the opening of the schemes. Unlike the two examples presented above, this post-completion evaluation was aimed more at assessing the initiative evaluation technique (that is, the COBA program) than the outcomes of the initiative or the initiative implementation process.
The difference between predicted and actual NPV was calculated by updating original COBA parameters (such as the value of time and of accidents) and substituting predicted traffic data - including traffic volumes, accident numbers and journey times - with measured data. The table below shows the stages involved in comparing actual NPV with the NPV of the initiatives as predicted by COBA prior to their implementation.
| Stage | Description |
|---|---|
COBA replication |
Replicate the original COBA results, as obtained prior to trunk road initiative implementation |
COBA rebasing |
Change the discount rate and manually rebase predicted benefits to the latest Present Value Year (1988) |
COBA updating |
Update COBA economic values relating to time and vehicle operating costs, accidents, annual compound growth rates and maintenance costs |
Updating traffic parameters |
Update accident rates, occupancy rates and vehicle category proportions |
Application of actual traffic data |
Input observed traffic flow, observed vehicle category proportions, observed accident rates and adjust for factors such as road type Adjust COBA speed-flow curves and junction configurations to reflect observed journey time savings |
Source: Knight et al. (1996)
This post-completion evaluation is an example of a comparison between real and predicted situations. Through the steps outlined above, it was possible to determine how some of the actual outcomes of trunk road initiatives, such as reductions in journey times, traffic congestion and accident rates, compared with the outcomes modelled by COBA. The following figure shows the predicted and actual NPV of each of the trunk road initiatives selected for post-completion evaluation as arrived at through the post-completion evaluation process. For example, NPV for Initiative A achieved more than three times the predicted value. The NPV predictions when considered in total were within 6 per cent of the NPV line of best fit. While results were found to be variable across different initiatives, it was found that the main source of inaccuracy came from difficulties in estimating initiative costs.
Figure 4: Actual and predicted weighted Net Present Values
Source: Knight et al. (1996)
A.4 The effectiveness of black spot treatment
The Bureau of Infrastructure Transport and Regional Economics (BITRE) has undertaken three evaluations of Federal Government black spot programs, the last being released in 2012 (BITRE 2012).
The Program's objective is to reduce the social and economic costs of road trauma by improving the physical condition and management of black spots. This is done by implementing traffic management techniques and other road safety measures that have proven road safety value. The purpose of the evaluations was to provide information to the Australian Government about the merits of continuing to fund black spot treatment.
The most recent evaluation aimed to include all Australian Government funded black spot projects approved during the seven-year period 1996‑97 to 2002‑03 inclusive and that had been completed. The final database used for the regression analysis contained 1599 projects, which was 62 per cent of the 2578 projects in scope. Crash data from project sites covered periods up to seven years before and after project implementation. The database contained 31 522 casualty crashes and 40 302 property damage only (PDO) crashes.
The evaluation addressed three principle questions.
- How effective are black spot treatments in reducing crash rates?
- How many crashes are avoided and lives saved annually as result of the program?
- Is the program a good use of resources compared with alternatives?
The first two questions were answered using statistical analysis – Poisson regression – which compares crash counts before and after black spot projects. Crash reduction factors were estimated for 20 categories of treatment type for different crash severity groups. The National Black Spot Program was estimated to be reducing fatal and casualty crashes in total at treated sites by 30 per cent and property damage only (PDO) crashes by 26 per cent. Roundabouts were the most effective treatment, reducing casualty crashes by over 70 per cent and PDO crashes by 50 per cent. New signals during daylight hours and altering the traffic flow direction were the next most highly effective treatments for most severity levels, reducing crashes by more than 50 per cent. No treatment types were found to systematically increase crashes.
Using the statistical model to predict expected crashes with and without the treatments in a given year (2006), each project was estimated to be saving 1.7 reported crashes per year on average. For individual severity levels, average reported crashes avoided per project per annum were 0.01 fatal, 0.11 serious injury, 0.55 minor injury, 0.61 injury, 0.62 casualty and 1.1 PDO.
The third question, which concerns the economic worth of the program, was answered using cost–benefit analysis. The estimated numbers of crashes avoided over the lives of treatments were multiplied by unit crash costs to obtain the annual benefits and these were discounted over the lives the treatments. In economic terms, the National Black Spot Program was found to have performed well overall, achieving an estimated benefit–cost ratio (BCR) of 7.7 at a 3 per cent discount rate and 4.7 at a 7 per cent discount rate based on estimated casualty crashes avoided and project construction, operating and maintenance costs. Subtracting costs from benefits, the average net present value per project was $1.4 million at a 3 per cent discount rate and $0.7 million at a 7 per cent discount rate.