6. Monitoring and Evaluation

6.1 Introduction

This section provides guidance on monitoring methodologies that will assist in the collection of useful data for ex-post evaluation of TBhC initiatives with least effort and cost. Monitoring for workplace and school travel plans is more straightforward than that for community- or household-based initiatives – there is a clearly identifiable target population and generally the focus is on specific trip types and purposes (commuting to and from work or the journey to and from school). In the case of community- and household-based initiatives, a wide variety of trip types and purposes by all members of a household are canvassed over varying times of the day and week. This creates a significant challenge to measure such potentially complex changes to travel behaviour with precision.

This section focuses on the methodology for the monitoring program, as well as identifying the monitoring focus for data for collection. No specific advice is given on questionnaire design or analysis of the data collected.

Given that there are various methodologies available to monitor TBhC initiatives (each with their own strengths, weaknesses, and complexities), only basic information can be provided here. Project proponents are advised to seek the advice of credible market research and travel measurement firms, particularly those with experience in monitoring these types of initiatives. Questionnaires and survey design must be tailored to the objectives to be met and the information required.

It is important that the monitoring task is designed as an integral part of the overall TBhC program (it often accounts for a substantial part of the overall program costs). The ‘before’ monitoring task is at least as important as the ‘after’ monitoring task in contributing to any conclusions on the effectiveness of the initiative. Thus the overall monitoring program needs to be designed at an early stage, along with the design of the initiative, and not considered only as an after-thought following implementation of the initiative itself.

6.2 Monitoring focus and outcomes

The primary outcome to be measured in monitoring TBhC initiatives is generally the overall change in VKT. Based on this indicator, it is possible to calculate benefits including decongestion, vehicle operating costs (including reductions in fuel use), environmental impacts (such as local air quality, CO₂ emissions and water quality) and accident reduction impacts.

Other key outcomes to be measured are changes in person kilometres travelled by mode. This is needed to identify the extent to which private motor vehicle trips have been replaced by public transport or cycling and walking trips as distinct from the extent to which trips have been eliminated altogether by 'trip chaining', using telecommunications or by alternative activities. This greater level of disaggregation is needed to calculate changes in mode share and benefits such as the health benefits of active modes. If monitoring is performed at this level, the results can also be aggregated to obtain overall change in VKT.

The focus of monitoring efforts should be on collecting evidence of both the mode share (for all modes) and the change in VKT as a 'single occupancy vehicle' driver by all household members living in the target population area.

Note that this focus on VKT is particularly important for household-based initiatives, as in some cases programs aim to achieve 'smarter' car use wherein the mode may not change, but the VKT does.

Various methods are available to collect VKT or person-kilometres travelled data from participants in a particular TBhC initiative. In the case of school and workplace travel plans, where the trip under observation is well defined (usually home to work or school, sometimes combined with other destinations/stops), it is more reliable to obtain geocode-able addresses than to rely on respondent estimates of the distances involved. For community and household-based initiatives, where a wide range and number of possible trips are involved, other methods may be preferable.

In addition to the mode shift and reduction of car driver vehicle kilometres, it is suggested that data be collected regarding the change in physical activity levels of the target population as this has an effect on the overall health and well-being of the community. This will be achieved if all travel by any mode is monitored at the individual person kilometre level.

It is recognised that organisations and local authorities may have other objectives within their monitoring programs, such as changes in target population attitudes (as opposed to behaviour), evaluating awareness of the TBhC initiative or the level and quality of information received. These may be added into the questionnaire(s) as desired. For example, the aims of TBhC initiatives may include reductions in congestion or improvements in safety around schools. The objectives of the proponent and funding organisations will influence the prioritisation of TBhC initiatives and most likely the factors that are to be monitored to measure performance against those objectives.

6.3 Methodology

TBhC monitoring programs establish the 'before' travel patterns for all members of the respondent household or other target population and then evaluate the change in these patterns 'after' the implementation of the various components of the TBhC initiative.

6.3.1 Monitoring approaches

The most common approach to date is to survey randomly selected households in the target population area. Historically, this focused on the use of travel diaries. These can provide full details of trips undertaken by a household and, given sufficient sample size, changes in total travel (trips, person- and vehicle-kilometres travelled) may be estimated for all modes. Usually travel diaries cover between one and seven days, although with anything over one or two days there is likely to be a significant loss of response and accuracy. Issues to do with sample size, respondent drop-off and bias are discussed further below.

Given the need to monitor changes in VKT, odometer-based surveys are an alternative (or complement) to travel diaries. Such surveys could record odometer readings for household vehicles at long intervals (such as three, six or 12 months apart) and directly derive VKT and VKT changes by household. However, using this method on its own will not permit assessment of mode shifts, changes in vehicle occupancy or changes in trip chaining behaviour. Other problems include the car may be sold, the preferred car may change in a multi-car household and/or long trips distort results.

More recently in Australia, successful monitoring of TBhC initiatives has been achieved by fitting Global Positioning System (GPS) devices to people and vehicles to record person-kilometres travelled and vehicle-kilometres travelled respectively. These devices have considerably greater accuracy than travel diaries, which have been found to have significant discrepancies between actual travel and reported travel – as high as 20 per cent. Short trips, the ones most likely affected by TBhC programs, are the most common type of trip omitted in travel diaries. GPS devices can easily produce information for one week’s travel for a household, including travel times, destinations and trip duration. The ability to collect data for a whole week substantially reduces the sample size requirements for a monitoring program and may reduce respondent fall-off rates.

In Western Australia, household programs have started to include monitoring as part of the TBhC intervention, with participating households asked a few questions about how they travel (mode and number of minutes) in the first and final coaching calls as another indicator of change. Although not statistically robust, such an approach does provide a cost effective early indicator of the effectiveness of the TBhC initiative.

6.3.2 Panel survey v independent samples

Regardless of what type of survey is adopted to monitor the project outcomes, there is the need to consider whether a 'panel survey' or 'independent samples' (also known as cross-sectional surveys) will be used. Panel surveys, wherein the same respondents are used for the before and after surveys, are considerably more statistically efficient than independent samples (where different groups are surveyed in either survey) and hence are generally preferred as lower sample sizes can be used for a given degree of confidence. However, panel surveys suffer from progressive drop-off of responses in successive surveys, which is especially important in cases of medium or longer term monitoring. This can only be partially overcome through a good survey approach and/or adopting unusually large sample sizes for the before survey (to allow for respondent loss in after surveys).

6.3.3 Survey timing

Travel patterns (especially for environmentally friendly modes) are substantially affected by seasonality: the before and after surveys should be undertaken 12 months apart (in the same month of the year), regardless of the monitoring approach. Hence, typically the initial after survey would be 12 months following the before survey, with program implementation between these two points. If this is not feasible, it is desirable that the after survey be at least nine months following project implementation to allow for new behaviour patterns to settle down and not record very short-term impacts. If this is the case, extra care will also need to be taken to account for seasonality factors that could be influencing monitoring results. It is probably beneficial to repeat after surveys in order to monitor the stability of changes. Any subsequent 'after' surveys should typically be at 12-month intervals.

It is known that there is considerable variability in travel across the days of the week and by travel mode. Given this, it is essential that surveys are spread across days of the week and, where the same households are being surveyed 'before' and 'after', the survey days of the week should be the same in both.

On any given day, travel is affected significantly by weather, special events or other factors. Some of these variation factors can be avoided (for example, avoid monitoring just before or after statutory holidays or local special events). For others, it is more difficult to do so and it may be necessary to rely on the control group being similarly affected (discussed below).

6.3.4 Monitoring program participants

The population of all households in the TBhC project area – those that participate in the program and those that do not (although it is useful to separate these in the analysis) – should be included in the monitoring program. This allows inclusion of 'diffusion' effects, where the project positively affects non-participants’ behaviour, providing a much better basis (than just participating households) for assessing aggregate effects of the project.

It is preferable to involve the whole household, rather than just an individual from the household, in the monitoring program. TBhC projects generally provide information and incentives on a household basis. As a result, the effects are likely to diffuse through the household, meaning that travel behaviour changes could be under-estimated if only one individual from that household is monitored.

6.3.5 Control group

Regardless of which monitoring approach is taken, control groups are important for robust evaluation, particularly of household/community programs. Although some adjustments for 'external factors' may be possible in the absence of a control group (see below), these are most unlikely to be sufficient on their own.

The control group area should be as comparable as possible to the program area (but unaffected by the program) in terms of similar socio-demographics, car ownership and use levels, public transport levels of service and use, topographical features and distance from the central business district. Often, control groups are selected from suburbs adjacent to those where the program is being delivered. However, care is needed to ensure that these control group areas are not subject to the indirect influence of the program (for example, through local press publicity).

It is important that other local changes do not occur in either the project area or control area that may impact on travel behaviour of residents, such as increases in public transport services in anticipation of increased demand as a result of the TBhC initiative or other transport system changes (such as new roads, changes in public transport services or other TBhC initiatives introduced). Often it will not be possible or desirable to prevent or delay these. The important thing is to record when they occur and make appropriate efforts to distinguish the effects from these changes from those of the TBhC project.

6.3.6 Sample size

The required sample size to assess a defined degree of change, with a given level of statistical confidence, will depend upon a range of factors and it is best to obtain specialist advice for establishing the appropriate sample size for a particular project.

Sample sizes are essentially (almost) independent of the size of the population concerned (for large populations).

6.3.7 Systematic survey bias issues

For a random monitoring survey of all households, the typical before and after (successful) response rates are in the order of 50 per cent. Given this, the dangers of non-response (or self-selection) bias are considerable - that is, the change in the behaviour of the responding sample may differ substantially from that of the non-responding households in a way that is unknown. Hence, there is no reliable basis to extrapolate the sample results to the whole population’s behaviour as, for example, less mobile households have a greater propensity to respond than more mobile households, but more mobile households may make bigger behavioural changes.

Some corrections can and should be made for this problem by comparing statistics (such as age, income, car ownership and so on) for the respondent sample with those for the area population as a whole, and differentially expanding from the sample to the total population. However, it should be recognised that these may not totally correct the entire problem.

6.4 Maximising survey response

The main problem relates to after surveys, as before survey data for households that do not complete the after survey cannot be used in panel surveys. It is therefore critical to minimise any drop-off from before to after responses. Suggestions for achieving this include:

• Reference may be made to standard market research texts on how to maximise survey response.
• Surveys need to be designed to be as simple as possible for respondents to complete.
• Personalised contact is likely to help (face-to-face contact is probably ideal).
• Incentives could be considered. While there is debate about the (cost) effectiveness of incentives in encouraging responses, it is reasonable to expect appropriate incentives (particularly for the after survey) would help.

6.5 External monitoring sources

A variety of ‘external’ monitoring sources might be used for before versus after evaluation to verify, supplement or potentially replace project-specific household surveys. These could use data that is collected in any event, usually for other purposes, or surveys undertaken for this particular purpose.

Examples are:

• Public transport patronage data (from electronic ticketing or other sources)
• Road traffic counts
• Cycle and pedestrian counts (not commonly undertaken).

Such data will establish changes in trips at a point or over a public transport route. They will not establish changes in trips by a given set of households.

If a program is undertaken on a metropolitan-wide scale, changes in trips by metropolitan households in total may be reasonably inferred. However, it will not generally be possible to identify to what extent the trip changes are the result of the program and to what extent they result from other factors. If a program is undertaken over a limited area (as is typically the case), then the impacts of the program on traffic counts and other sources will rapidly diminish further from the area. Even within the area, the impacts are likely to underestimate the changes in travel by area residents, because of the through traffic component. However, in particular circumstances, reasonable inference about the effects on travel by area residents might be made. In drawing any conclusions from external monitoring data, it will be necessary to compare any local changes in the area of the program with any changes in the wider area (as the ‘control group’).

For larger scale programs, such external surveys can provide useful evidence. However, this should normally be regarded as supplementary to, rather than in place of, direct household surveys.

For small scale programs, any changes in travel observed in external surveys are likely to be very small. Therefore, external monitoring sources will be of little practical use.