5. Benefits evaluation

Following the delivery of an initiative, benefits evaluation can be undertaken.

Benefits evaluation is important because:

  • It helps to confirm that the benefits established and defined in the initiative's development stage are being achieved and that the implemented initiative is running smoothly.
  • It measures the extent to which the initiative achieves the outcome(s) sought (as described in the Business Case) and to what extent further justified actions are required to achieve and/or enhance benefits.
  • It provides information that may be used to help inform future decisions.
  • It provides a level of transparency and accountability for past initiative decisions.
  • It provides learnings for future planning, assessment and decision-making.

The key activities undertaken in benefits evaluation include:

  • Collecting, analysing and comparing benefits data
  • Identifying key findings and lessons learned
  • Disseminating the findings and lessons learned
  • Updating the Benefits Management Plan and Benefits Register.

An agreed Benefits Management Plan guides the implementation of the benefit evaluation process. It confirms the scope and purpose of the review, enables an organised approach to timing and conducting the evaluation, implements the roles and responsibility of stakeholders involved in the evaluation process, and guides the benefit handover process to the benefit owners.

It should be noted that benefits evaluation is broader than a post-completion review as it encompasses planning actions necessary for the full realisation of project benefits, such as undertaking complementary projects or altering land use plans. These actions are frequently outside the direct control of the project proponent, so the Benefits Management Plan has to address how the responsible parties can be influenced to ensure their actions are aligned.

A benefit evaluation report may be prepared to document findings and lessons learned as part of this process.

5.1 Timing

Benefits evaluation should be conducted after an initiative has been delivered, at periodic intervals reflective of the timing and ramp-up of benefits (depending on the scale and associated behavioural change from an initiative). This allows for evaluations as benefits are fully materialised. This means that:

  • Initial potential disruptions and business processes do not affect the evaluation results.
  • Statistically significant collection of measurement data can be undertaken.

Benefits evaluation can also be undertaken as part of a post-completion review and findings from the evaluation process will help inform a post-completion review. Further guidance on post-completion review can be found in F7.

5.2 Collecting, analysing and comparing benefits data

5.2.1 Data collection

The approach for benefits data collection should follow the same methodology identified during baseline measurement in benefits planning. Data collection should be targeted. Since the benefits evaluation is explicitly linked to the benefits planning process, the methods outlined below should deliberately seek data and information that elucidate whether the benefits outlined in the initial plan have been achieved.

In addition to existing information about the initiative (i.e. Business Case, Benefits Management Plan and Benefits Profiles) and reports, information collection techniques may include the following:

  • Questionnaires/surveys
  • Interviews (structured or semi-structured and based upon issues developed for each stakeholder)
  • Observation (i.e. of the service or asset)
  • Collection of operational and service outcome data from the relevant agency or service provider
  • Focus groups and workshops.

All data collected before or during delivery that will be used are reviewed and a decision made about what new data are needed. This requires specifying the characteristics of the data so that what is collected is useful and comparable to other datasets that are used. This may include:

  • Specifying dates and times for data to be collected
  • Specifying levels of accuracy of data and collection equipment
  • Specifying volumes of data or sample sizes
  • Other specifications particular to a type of data.

It is important to consider the availability and reliability of data to support the evaluation of benefit performance. The time and the cost of capturing the data should be relative to the scale of the benefits and initiative.

5.2.2 Compare and analyse data

Analysis of the information gathered will involve comparing what actually happened against what was predicted (as outlined in the Benefits Management Plan and supporting Benefits Profiles). This includes analysis of:

  • Outcome performance (i.e. actual vs target)
  • Realisation date (i.e. ahead of schedule/on time/lag and including reasons)
  • Dependencies (i.e. cross-referenced to other benefits internal and/or external to the project)
  • The justified actions required to realise the benefit
  • Stakeholders impacted by the benefit
  • Assumptions (i.e. about probability and/or value of benefit)
  • Outstanding risks to future benefit realisation and corrective actions that can be taken to mitigate the risks
  • Costs associated with benefits measurement
  • Effectiveness of benefits management approach.

In examining the outcomes, consideration should be given to what caused any variances between what was expected and what was achieved. The recommendation may be to modify aspects of the initiative in question (if justified) so as to increase the likelihood of achieving the expected benefit or to take the observed variances as expected due to changes in the baseline.

5.2.3 Document findings and lessons learned

Findings and lessons learned arising from the analysis of benefits data should be consolidated and reported to inform:

  • The ongoing operation of the service or asset (which is the final output of the initiative)
  • Future initiatives.

In addition to understanding if the intended benefits of the initiatives have been fully realised, benefits evaluation should assess:

  • What was done well and contributed to achieving expected and additional benefits
  • What opportunities for improvement exist.

Lessons learned may include:

  • How practical problems were overcome with strategic assessments, Business Cases, funding models, procurement processes, project knowledge management, effectiveness of benefits management and risk management
  • Unexpected benefits that emerged
  • Experiences of stakeholders and staff that provide useful insights for future initiatives
  • Improvements to a jurisdiction’s benefits management practice.

5.3 Disseminate the findings and lessons learned

For benefits evaluation to have a positive impact, the results must be communicated to practitioners and stakeholders involved in planning and evaluating the initiative, and those involved in the management or operation of the initiative. The information gathered during a benefits evaluation can be used to inform planning and project management practices. An evaluation report may be prepared to document findings.

Those required to evaluate similar options in future will be able to use the information gathered to inform their decisions. Thus the benefits evaluation process creates a feedback mechanism through which evaluation practices can be improved.

A communications plan for the findings can be agreed with the benefit owners and key stakeholders to facilitate dissemination of the findings and report.

5.4 Update Benefit Management Plan and Benefits Register

Once benefits evaluation is completed, any changes to the Benefits Management Plan and Benefits Register should be agreed with the benefit owner and the changes made as appropriate.

Benefits evaluation may involve periodic updates to the Benefits Management Plan and Benefits Register with findings shared and documented.