Appendix C Reference (Base) model validation criteria

A review of reference (base) model validation techniques will be undertaken during subsequent Stage of the ATAP update. This will include the recommendation of fit-for-purpose statistics to use for model validation. This review will include investigation of the MAD% and MAWD% statistics amongst others.

In the meantime, the following criteria should be used to validate the Reference (Base) model to ensure that modelled results are consistent with observed data (i.e. traffic counts and travel times). If the criteria are met (or if not met, and there is sufficient confidence that the transport model is still fit-for-purpose), the Reference (Base) model is considered adequate for predicting the present and is fit-for-purpose for forecasting.

Ideally, the observed data should be the most recently available traffic counts and travel times.

Link flows

Link volume plots
- For each time period, produce a map of the transport network showing modelled and observed link flows and the differences between them. The totals should be summarised for available screenlines. These plots are used to check modelled and observed flows by geographic area and level of flow.
- As a guide, a reasonable error tolerance for hourly flows on individual links is approximately ±20 per cent. A major link is considered to be one that carries at least 15• 000 vehicles per day in one direction. In the case of screenlines, an acceptable error tolerance is ±10 per cent.
Scatter plot of modelled and observed flows
- Produce an XY scatter plot of modelled versus observed flows for:
  - all individual links
  - freeway links
  - screenlines
- Superimpose the y=x line on each plot. Report on the R2 for each plot.
GEH statistic

$G E H = \sqrt{\frac{{(v_{2} - v_{1})}^{2}}{0.5 (v_{1} + v_{2})}}$

where V₁ = modelled flow (in vehicles/hour) and V₂ = observed flow (in vehicles/hour).

Percentage root mean square error (RMSE)
- The RMSE applies to the entire network and has the following formulation:

$R M S E = \frac{\sqrt{\frac{{(v_{1} - v_{2})}^{2}}{C - 1}}}{\frac{\sum v_{2}}{C} \times 100}$

where:

V₁ = modelled flow (in vehicles/hour)

V₂ = observed flow (in vehicles/hour)

C = number of count locations in set.

Travel times

Provide a comparison of modelled and observed travel times as an XY scatter plot for each time period modelled. The scatter plot should also include the 95 per cent confidence limits for the modelled data. More specifically, modelled versus observed distance against time can be plotted for individual travel time routes.

Assignment convergence

Provide evidence of assignment convergence by detailing the:

Type of assignment (equilibrium, volume averaging, incremental)
Convergence achieved at the final iteration and the number of iterations required in achieving convergence
Percentage change in total generalised user cost in the final iteration
Proportion of links with flows changing <5%
Normalised gap δ: this is the flow-weighted difference between current total costs and the costs incurred if all traffic could use the minimum cost routes – should be less than 1%.