3. Options assessment

Once a range of options has been identified, a structured process should be used to assess those options and move from a longer list of options to a shorter list of options and, finally, to a preferred option.

3.1 The assessment process

At the start of the options assessment process, a long list of options exists. A priority problem has been identified and an initiative consisting of the best solution option is sought to solve the problem. The proposed initiative is unspecified at this stage.

The process involves narrowing down the list of options through a ‘filtering' process. It should be structured, objective, and evidence-based. Options should not be ruled out on the basis of personal preferences, perceived political difficulties or in any way that precludes genuine consideration of certain options. Options should be judged purely on their merits and ruled out only on the basis that they do not address the problem in an efficient way. In reality, however, governments do sometimes rule out some options early in the planning process[1].

A three-tiered approach is recommended to assess and narrow down options:

  • Stage 1: Strategic Merit Test (SMT)
    • Consideration of an option's or initiative's alignment with goals, objectives and strategic plans
  • Stage 2: Rapid appraisal
    • An initial indicative assessment of the scale of an option's or initiative's benefits and costs
  • Stage 3: Detailed appraisal
    • A detailed assessment of an option's or initiative's benefits and costs, and other impacts.

The process is shown in Figure 4, and can be viewed as a series of filters where:

  • Each filter removes some options.
  • An increasing number of options are rejected as the process progresses.
  • The level of effort required for each filter increases as the number of options that require testing goes down.
  • Options that clearly fail the SMT and/or rapid appraisal stage can be rejected early.
  • In cases where multiple options cannot be eliminated easily after rapid appraisal, they should each be subjected to detailed appraisal.
  • The best or preferred option is the one that passes through all filters.

The assessment process focuses on determining:

  • Whether an option or initiative has strategic alignment with transport system objectives, strategies, plans and policies (stage 1)
  • Whether an option or initiative will deliver net benefits, i.e. benefits greater than costs (stages 2 and 3)
  • The option that delivers the largest net benefits (stage 3)
  • An understanding of other impacts such as distributional (equity) impacts or other specific impacts that may be required by a jurisdiction (stage 3).

The final outcome is the identification of the preferred option, supported by a rich set of information about its merits.

Figure 4: Moving from a long list to a preferred option

Moving from a long list to a preferred option

An important consideration in the assessment of options is the level of inter-dependence between various options being assessed. Relationships between options may either be independent, complementary or substitutable. These relationships should be identified at this stage. A full discussion on inter-relationships between options and initiatives can be found in Appendix A.

3.2 Assessment tools and information types

3.2.1 Tools

A number of tools are recommended for use in the options assessment process, as listed in Figure 5.

Figure 5: Option assessment tools

Options assessment stage Recommended tools
Strategic Merit Test Multi-criteria assessment (MCA)
e.g. Objective Impact Table (OIT))
Rapid appraisal Rapid Cost-Benefit Analysis (CBA)
Rapid Appraisal Summary Table (AST)
Detailed appraisal Detailed CBA
Detailed AST

These tools are described in detail in the following sections (and in other parts of the Guidelines). In brief, their main features are:

  • An MCA is an approach that scores an option or initiative under several different criteria (which may or may not be weighted and aggregated into a single score)
  • A CBA is an economic analysis tool for calculating the net benefits (benefits less costs) of an option or initiative expressed in money units
  • An AST is a format for summarising the results of an appraisal process, including non-monetised benefits and costs.

The core of the assessment process consists of using CBA complemented by the AST. These tools are used in the critical second half of the appraisal process to assess short-listed options and to identify the preferred option.

MCA plays a lesser role, used only in the first stage of assessment to reduce the initial long list of options to a short list (- consistent with the role for MCA proposed by IA (2017)). It should not play a role in selecting the preferred option(s) in rapid and detailed appraisal presented in the final business case.

Of course, CBA can also be used in the first stage to complement an MCA and doing so can only improve the quality of the assessment. However, there are usually insufficient resources to undertake a CBA for the full list of options, with the simpler MCA providing a more realistic tool for stage 1.

An adjusted-CBA is another tool that could be used. It is discussed in Chapter 12 of T2 as an ‘optional’ tool.

Note that cost-benefit analysis is one of a range of economic analyses discussed throughout the Guidelines. Appendix E provides a brief overview of the various types of economic analysis.

Types of appraisal information

Decision-makers have the task of making choices using the information presented to them. This is a complex task, especially given that they need to absorb several different types of information generated in the appraisal:

  • Monetised benefits and costs–these are benefits and costs that can be expressed in dollar units. Where a benefit or cost can be monetised, it is desirable to do so
  • Non-monetised benefits and costs–these are benefits and costs that cannot easily or reliably be monetised. They are as important as monetised benefits and costs and should be presented alongside those. The use of a rating to describe the nature of the non-monetised impact (e.g. +ve or –ve, small or large) can play an important role in assisting the decision-maker
  • Quantitative and qualitative impact descriptions–these are necessary inputs to calculating monetised and non-monetised benefits, costs and impacts. Presentation of these inputs can also be of assistance to the decision-maker. Non-monetised impacts that are non-quantifiable can only be described in qualitative terms.

Importantly, net benefit, the key indicator of the merit of an option or initiative, is measured as combined benefits less combined costs, both monetised and non-monetised. Monetised and non-monetised items cannot be directly added in a mathematical sense because they have different units, so a three-step process is recommended:

  • First, monetised net benefits should be calculated. This is the only step required if there are no non-monetised benefits or costs involved
  • Second, compare non-monetised benefits with non-monetised costs, and assess broadly which is larger and by what extent. This may involve comparing effects with different units, and will require some subjective judgement
  • Third, conclude whether the net effect in step two negates or reinforces the monetised net benefit calculated in step 1.

Where monetised net benefits and non-monetised net benefits oppose each other, it is helpful to ask whether the non-monetised net benefit component would be valuable enough to change the decision from justified to not justified (or the reverse). For example:

  • If an initiative has a monetised net benefit of $100 million, but non-monetised net disbenefits, ask whether the non-monetised net disbenefits would be worth more than $100 million. This is a subjective question, but the correct question to ask regarding whether the initiative is justified from a cost-benefit analysis (economic efficiency) perspective.

3.3 Stage 1: Strategic Merit Test

The purpose of the SMT in options assessment is to check how well the identified options align with the economic, environmental and social goals and transport system objectives defined in Step 1, and approved strategies and policies. This enables an initial filtering of options before further assessment and development.

The SMT is not intended to be comprehensive. It is intended to be an initial check for strategic merit that:

  • Rules an option in or out at an early stage of the assessment process
  • Identifies those options that should proceed to the next stage of appraisal, options that require further work and those that should be abandoned because they are inconsistent with the jurisdiction's objectives and strategies.

The SMT is a useful mechanism because it:

  • Requires a clear early explanation of how an option will meet higher-level objectives
  • Provides an efficient means to filter options before considerable resources are spent on further appraisal and development.

As jurisdictions are likely to differ in how they assess strategic merit, it is important for each jurisdiction to design a process that is best suited to its circumstances. The process could be as simple as a checklist for consideration by decision-makers/ministers that aligns options against government goals, objectives, policies and strategies. Alternatively, a more formal process, such as an Objective Impact Table (OIT), could be used (see Box 2).

Regardless of the approach to the SMT it is important that the analysis:

  • Adopts a logical and consistent approach for all options
  • Is objective and evidence based to avoid subjective judgments.

Box 2 Recommended tool: Strategic Merit Test

For an SMT, a number of multi objective/criteria techniques could be used, as described below.

Whichever tool is used, a decision needs to be made about the strategic merit of each option using a simple ‘yes/no’ or ‘pass/fail’ system or via a more detailed ranking scale.

a) An Objective Impact Table (OIT) is a matrix that provides high level information about an option's impacts and how it contributes to achieving the objectives defined in Step 1. The OIT aligns impact types against relevant government objectives and then describes these impacts. It is most effective when it contains quantitative information. Where that is not possible, impacts should be described in qualitative terms.

A basic OIT template is provided in Appendix C, along with a sample qualitative rating scale.

Multi Criteria Analysis (MCA)

b) A Multi-Score MCA is an extension of the OIT. Rather than the qualitative rating scale used in the OIT, a quantitative scale is used to ‘score’ options for each objective/impact type. The scores are then used to assess the relative performance of the options in each objective/impact type.

c) A Single Score MCA is an extension of (b). Weights are introduced to represent the relative importance of the objectives. Weighted scores are then calculated, with the sum providing an overall weighted single numeric score for each option. This contrasts with approach (b) which produces a score for each objective for each option.

It should be noted that MCA techniques have received strong criticisms (e.g. BTE 1999, Dobes & Bennett 2009, Ergas 2009, Australian Government 2014). Infrastructure Australia (2017, section D2.6) also highlights a range of concerns about MCA approaches:

  • Weighted scores have no units and no meaning beyond the analysis
  • While the method avoids explicit monetary values, it assigns such values implicitly–in contrast to CBA where monetary valuations are explicit
  • The method is open to influence by interest groups and likely to be biased in favour of the proposal
  • The selection of criteria is likely to be biased
  • The method is likely to have a local focus and therefore overlook system wide effects.

Approach (c) using a single weighted score is particularly problematic and can be highly misleading. The weights are necessarily arbitrary and subjective, opening MCA up to the criticism of lacking methodological rigour, being a largely subjective assessment and making the technique open to manipulation. In addition, the weights tend to be obscured by the process of combining into an overall weighted score.

The overall conclusion is that the OIT approach is preferred to MCA approaches. If, however, MCA is used:

  • Its use should be limited to the early stages of assessment (SMT) where its simplicity is often appropriate for shortlisting from a large number of options
  • Approach (b) is preferred to (c), ensuring that subjective judgments are not obscured
  • Approach (c) is not recommended and preferably avoided
  • Where practitioners opt to use approach (c), they can minimise the risk of inappropriate bias by linking the assigned weightings back to the ranked objectives established in Step 1.

Practitioners also need to be clear and transparent in selecting the most appropriate tool and any details of their application. Completing this before commencing the analysis is critical to managing the risks associated with the tools.

Finally note that the Goal Achievement Matrix (GAM ) approach is sometimes used at the SMT stage of assessment and in post-completion reviews. GAM is similar to MCA, using scores and weights, with a focus on goals rather than criteria. It is therefore considered an ‘MCA-type’ approach. TfNSW (2017) notes its attraction is its ease of use, and also notes its limitations, whilst BTE (1999) is strongly critical of it.

At the end of Stage 1, the best performing options move forward to Stage 2.

By the end of the assessment process, the strategic merit of the preferred option (as assessed through the SMT) becomes the strategic merit of the initiative.

The results of the SMT also play an important role later in the Framework during prioritisation across a number of initiatives (see F5).

3.4 Stage 2: Rapid appraisal

Stage 2[2] in the options assessment process is a rapid appraisal of the shorter list of options that have passed the Strategic Merit Test.

Rapid appraisal screens out options that have passed the SMT but are unlikely to pass more detailed assessment. It involves an initial indicative assessment of the main benefits and costs, with a lower level of accuracy than in a detailed appraisal.

Rapid appraisal is a cost-effective way of gauging whether an option is likely to pass a detailed appraisal. The resources required for a detailed appraisal can then be expended only on solutions that have a good chance of succeeding.

Options that appear to yield net benefits should proceed to a detailed appraisal. If all options perform poorly in the rapid appraisal, practitioners need to consider whether any initiative to solve the problem should be progressed at all.

It is recommended that use of MCA scoring techniques not be continued into stage 2. The recommended tools for this stage of the assessment process are (see Box 3):

  • A rapid CBA
  • A rapid AST.

Where possible, benefits and costs should be monetised and expressed in dollar units. This is the primary focus of the CBA. The remaining non-monetised benefits and costs should be listed, described in quantitative and qualitative terms and, if possible, provided a rating in terms of likely direction (e.g. +ve or -ve) and scale (e.g. small, medium or large).

The AST provides a mechanism for summarising both monetised and non-monetised results side-by-side.

At the end of Stage 2, the best performing options move forward to Stage 3.

Box 3 Recommended tools: Rapid appraisal

Rapid CBA

A CBA is a form of economic analysis that assesses the benefits and costs that can be expressed in money units. It expresses them in terms of today's money (‘present values'), providing a common metric for comparing options.

The methodology used for rapid CBA is the same as for a detailed CBA. However, the estimates for a rapid CBA are less precise; only the main monetised benefits and costs are taken into account and any benefits and costs that are small, or difficult to estimate, can be omitted altogether.

Figure 6: Outline of key steps in a rapid CBA

Outline of key steps in a rapid CBA

Further information on CBA and how to estimate monetised benefits and costs is given in T2. Appendix B provides a template for setting out the results of a rapid CBA.

Appraisal Summary Table (AST)

The AST is a form of presentation developed by the UK Department for Transport. It is a decision-support tool that brings together the various strands of assessment into a summary format to better present the whole picture.

The AST addresses the same question as CBA: is an initiative likely to produce a net benefit? Its key features are:

  • It presents a summary of all monetised and non-monetised economic, social and environmental benefits and costs on a single page, in a user-friendly format.
  • Monetised benefits and costs are presented in present value dollar units, with net present value and the benefit cost ratio from the CBA also recorded.
  • A qualitative non-monetised rating system is used that describes impacts as being either positive or negative, and whether the scale of the impact is neutral, small, moderate or large. It also allows for inclusion of a level of confidence for the non-monetised rating
  • Quantitative and qualitative descriptions of the associated impacts can also be recorded.
  • The AST does not indicate the relative importance of the objectives and their associated impacts, leaving that to the decision-maker.
  • The AST enables decision-makers to understand the economic, social and environmental components of the appraisal and to make a subjective judgement about whether the combined monetised and non-monetised impacts suggest the option will produce a net benefit.

An example of an AST and instructions for designing an AST are provided in Appendix D. Further resources to assist in preparing an AST are listed in the References section at the end of this chapter.

3.5 Stage 3: Detailed appraisal

Stage 3 in the options assessment process is a detailed appraisal of the final options using:

  • A detailed CBA, and
  • An updated AST,

reporting monetised and non-monetised benefits and costs. The CBA and AST should be updated and refined versions of those presented in stage 2.

This analysis, combined with the results of the SMT (stage 1) should lead to the selection of a preferred option.

The assessment undertaken for this stage should be of sufficient detail to understand and assess[3]:

  • Triple bottom line economic viability–the lifetime benefits of the preferred option must significantly outweigh its lifetime costs to society; that is, there must be significant net benefits
  • Implementation plan–the preferred option must have a clear and robust plan to ensure the benefits can be realised successfully. This should include considering the following questions:
    • Is the risk being managed appropriately?
    • Does the initiative's governance model provide confidence that claimed benefits will be delivered?
    • Does the delivery strategy provide confidence that the initiative benefits will be delivered?

Box 4 Recommended tool: Detailed appraisal

A detailed Cost Benefit Analysis should be the primary tool for undertaking a detailed appraisal of options that have passed through the first two stages of the assessment process.

The detailed CBA should include:

  • A robust and objective CBA supported by strong evidence
  • Consideration of as many monetised benefits and costs as possible
  • Consideration of non-monetised benefits and costs as well
  • Consideration of the overall efficiency of an initiative (the combined scale of benefits compared to costs, usually expressed as a benefit cost ratio or BCR)
  • Consideration of issues of risk and uncertainty (usually provided through sensitivity testing).

Complementary information on the initiative's distributional (equity) impacts should be presented alongside the CBA.

A detailed Appraisal Summary Table. This will be an updated and refined version of that resulted from rapid appraisal.

For more information on undertaking a CBA see T2. For information on producing an AST, see Appendix D.

More information on detailed appraisal is given in NGTSM06, Volume 2.

At the end of stage 3, the best performing option becomes the preferred option. This option moves forward as a specific and justified initiative.

Practitioners should check whether, in addition to the assessment process outlined above, their jurisdiction requires any additional assessments.

Benefits

As discussed above, benefits play a central role in the appraisal of options or initiatives. The indentification of benefits is discussed in Box 5 below, including an explanation of the relationship with the broader process of benefits management (see T6).

Box 5 Benefit identification and benefit management

Benefits result from implementing a transport option or initiative. A benefit is a measurable improvement in an outcome, perceived as positive by stakeholders, and contributes towards one or more transport system objectives.

Benefit management is the process of properly identifying, defining, measuring, evaluating and reporting benefits in order to determine whether an initiative has achieved its intended outcomes and objectives once it is delivered. It occurs over the whole life cycle of the initiative, and is an important process in the ATAP Framework (see T6 for a full discussion).

Benefit identification is common to both appraisal (as discussed in F3) and benefits management (where it is the first step of a multi-step process). It seeks to identify and define potential benefits arising from addressing an identified problem (which, in turn, is preventing transport system objectives and targets from being achieved).

In appraisal, good benefit identification and assessment, along with good cost assessment, provides a strong foundation for a robust analysis to determine the merit of an option or initiative.

In appraisal, benefit identification involves identifying, defining and describing all the benefits that arise from an option or initiative. This is followed by benefit assessment, which involves quantifying each benefit (where feasible) and expressing benefits in monetised (where feasible) or non-monetised terms. Benefit identification usually involves careful considerations of the option or initiative and the underlying problem it is addressing. This can occur through desk-top investigations and stakeholder engagement. Techniques such as Investment Logic Mapping and Benefit Dependency Mapping (see sections 2.2.1 and 2.2.2 of T6) also provide useful mechanisms for exploring benefits early in the planning process.

In the benefit management process, benefit planning follows benefit identification. It selects a sub-set of the full list of benefits to monitor and evaluate throughout the remainder of the initiative's life cycle. The monitoring and evaluation of these is then implemented through use of a Benefit Management Plan (see T6). Benefits profiles can be started in benefit planning as part of developing a Benefit Management Plan. Benefits profiles will also help in informing the business case and are further refined in Step 4.

3.6 How deep should options assessment go?

Obtaining and analysing information for a CBA, MCA or AST incurs costs. This means that choices need to be made about the level or depth to which options assessment is conducted.

Generally, the effort and cost associated with assessing options should be proportional to the scale and complexity of an initiative and the options involved. This was outlined under the ‘fit-for-purpose' principle in the ‘ATAP Overview' section of the Guidelines.

The more significant an initiative or option, and the greater the likely impacts, the more expenditure and effort can be justified.

In making a decision about the level of assessment effort, practitioners need to consider whether the option warrants detailed appraisal. Small scale options with an estimated cost below a certain threshold (say $10 million) may not warrant a detailed CBA, although the SMT and rapid appraisal stages should still be followed.

[1] For example, the position of individual jurisdictions may differ on the use of road charging or pricing options, such as the use of road tolls.

[2] A complementary discussion of the appraisal process can be found in a recent paper by the Australian Government (2014). That paper was written in close collaboration with the ATAP Guidelines revision project.

[3] This terminology aligns closely with that used by Infrastructure Australia.