5. Model development

5.1 Estimation techniques

Frequently, analysts may wish to identify relationships between the factors contained in their data. These relationships are investigated through the application of statistical models that provide a useful, but simplified, view of the process being studied. This simplicity is limited by the need to provide a reasonable representation of the process. Statistical models use sample data to develop mathematical relationships between factors.

In the process of deriving relationships in the trip generation step, a multivariate linear model is often used. It has the form:

$Y_i=b_0+b_1{X}_{i1}+b_2{X}_{i2}+\dots +b_m{X}_{im}+{\epsilon }_i$

where the Y is the dependent variable (trip productions and attractions) and Xjj represents the m independent variables (typically demographic variables) for 1 ≤ j ≤ m. The subscript I represents the ith observation, the b0,b1,…bm terms represent the model parameters, and the ɛi is an error term.

Multivariate linear regression is a general statistical tool through which the analyst can investigate the relationship between dependent and independent variables. While multivariate linear regression is a straightforward, powerful and widely-used model estimation technique, care is needed in the development of any multivariate regression model intended for practical application. The advice of a statistician is recommended.

The computational ease with which such models can be estimated can mask the fact that such models are based upon assumptions that the inexperienced user typically does not validate, or even understand. These assumptions are based upon the absence of:

• Multicollinearity
• Heteroskedaticity
• Autocorrelation.

Multicollinearity occurs when (as is often the case) some of the ‘independent’ variables are highly correlated with each other – the statistical theory on which multiple regression is based assumes them to be independent of each other. Care is needed in selecting a subset of the independent variables for inclusion in the multiple regression relationship. The usual approach is to enter or delete independent variables one-by-one in some pre-established manner. Some of the approaches used are forward inclusion, backward elimination and stepwise solution. With forward inclusion, independent variables are entered one-by-one only if they meet certain statistical criteria. The order of inclusion is determined by the respect contribution of each variable to the explained variance. In backward elimination, variables eliminated one-by-one form a regression equation that initially contains all variables. With a stepwise solution, forward inclusion is combined with the deletion of variables that no longer meet the pre-established criteria at each successive step.

Heteroskedaticity is also a common phenomenon. It relates to the variation of the absolute errors in measurement of an independent variable with the actual magnitude of that variable. Statistical theory for regression analysis assumes that the variance of the error term is the same for all values of the independent variable. This is called homoskedasticity. There are many instances where this may not be the case. For instance, in considering the masses of consignment loads to be transported, an error in measurement of 1 kg would be insignificant for an item of machinery of mass (say) 400 kg, but would be very significant for a parcel of mass 5 kg. For standard regression analysis, the consequences of heteroskedaticity in an independent variable are to underestimate the standard error of the estimated (dependent) variable (Y) and to falsely raise the apparent significance of independent variables (X_j). It does not affect the estimated values of the model coefficients, but does reduce the efficiency of the model as an estimating tool. One solution is to use weighted least squares analysis, in which each observation of an independent variable is adjusted for the expected size of its error term. Another possibility is to transform an independent variable (for example, by using its logarithm, log (X)). This approach is useful for variables where there is a great range of sizes in the observations (such as the distribution of household incomes in a region).

Autocorrelation occurs when successive observations of a given variable are highly correlated. This phenomenon frequently arises in data observations taken over time. Here again, the problem for multiple regression analysis is the underlying assumption that observed values of the independent variables are independent of each other. In fact, autocorrelation is most important in its own right and there is a special field of statistical analysis devoted to ‘time series’ analysis. Autocorrelation reflects cyclical behaviour in a variable over time, such as seasonal variations in agricultural production, travel demand or road crashes in a region. Knowledge of these cycles is important in understanding the patterns of demand for transport services, among other things.

There are a number of statistical tests and procedures to identify and cope with these potential problems in regression modelling.

Chapter 17 of Taylor, Bonsall and Young (2000, pp. 392–410) provides a more detailed discussion of regression modelling techniques. See also Fahrmeir and Tutz (1994).

Multivariate linear regression is not the only technique available to estimate model parameters. It is recommended that statistical advice is sought to identify the most appropriate statistical, or other estimation technique to apply in estimating model parameters.

Fit for purpose techniques exist to determine the coefficients in logit models of discrete choice processes. (e.g. see Louviere, Hensher & Swait 2000).

A further method for model parameter estimation is ‘entropy maximisation’ or the use of the mathematical theory of information (Shannon 1948; Van Zuylen & Willumsen 1980). This method is used, for instance, in the estimation of origin-destination matrices from observed link counts – see Taylor, Bonsall and Young (2000, pp. 116–20).

5.2 Network checking

Model networks are a critical component of the ‘supply’ side of transport models. It is therefore essential that the model network is appropriate for its purpose. Inaccuracies in the alignment, connectivity or element components of the model network can lead to erroneous assignment of traffic to routes or inaccurate estimations of congestion and travel times.

Despite increases in geographic data and computer power, there is a still a necessity for all three levels of transport models to simplify real world networks into usable representations. For example, strategic transport model road networks will often exclude local roads and road classification may be limited to link class, lane number and posted speed. Similarly, microsimulation networks may simplify the gradient or road surface characteristics of a link.

In many cases the attributes of a network will have a degree of subjectivity. For example, strategic transport model networks will group roads into a fixed number of link classes or link types. Classifying a new or existing road into one of these groups will often involve the transport modeller making a judgement based on available information and a comparison to similar links in the network. In cases where subjective classifications directly affect a project being modelled, the details of the classification should be documented.

The degree of a network audit will often be constrained by time or budgetary constraints. At a minimum, the network should be checked in detail for any project links or elements as well as links or elements that are directly influenced by the project. The audit should include checks on internal consistencies as well as checks against reality.

It is desirable that checks be carried out across the entire modelled network. However, the detail of these checks will often be limited to ‘broad brush’ audits or automated investigations.

Strategic transport model network auditing

The auditing of a strategic model network has two roles

1. Ensure that the network has appropriate connectivity and
2. Ensure that the link and node attributes are appropriate for the model.

Some general techniques that can be used to check transport network models are:

• Visual checking of the network in the study area against aerial and ‘streetview’, photography (GoogleMaps or equivalent). The modeller should endeavour to align the date of the photography with the base model year.
• Compare the model network to reputable maps (street directories, public transport maps etc)

It is increasingly common for network data to be processed directly from electronic sources (road networks maintained for satellite navigation tools, and public transport timetables). In these cases particular emphasis needs to be placed on the interpretation of these data, in particular the connectivity represented at junctions and stops. Some techniques that can be used to check the connectivity transport network models are:

• Use the model to build paths to or from selected zones (on the basis of minimum distance, time or generalised cost) – this is one of the best ways of checking the network coding; the path building process will ensure the paths built by the model are logical and any errors in the coding are identified and corrected, eliminating any illogical paths during the assignment step.
• Checking connection points of centroid connectors by visual inspection or assigning the demand to the network and confirming that all demand has been assigned (excepting intrazonal trips)

Some techniques that can be used to check the link and node attributes are:

• Use the model or other GIS to display or map link and node attributes. This will assist the modeller or others who are familiar with the network to identify errors.
• Create link attribute frequency data for each link class. For example, a tabulation of posted speed for a freeway link class can be useful for identify links with erroneously low posted speeds
• Use the model to assign a peak period demand. Examine the results for links with zero volume or unexpected results such as high volume over capacity ratios or low speeds to identify outliers. Results such as these can indicate errors in coding.
• Plot observed traffic volumes against link capacity. Examine links where observed volumes exceed capacity.
• Comparing actual travel times along key links with those from the network model
• Comparing link distances from the model with actual distances

Some techniques that can be used to check Public transport networks are:

• Checking the frequency and stopping patterns of public transport routes. GIS can be used to display headways, speeds, number of services, capacity, etc.
• Use the model to assign Public Transport trips. Examine the results for links with zero volume or unexpected results.
• Compare model travel times to Public Transport timetables or Automatic Vehicle Location (AVL) data. In some instances modellers have analysed time stamped smart card ticketing data to assess vehicle running time intervals between stops.

Mesoscopic transport model network auditing

The techniques described above for strategic models are also applicable for mesoscopic transport models.

In addition, the following checks should be conducted for mesoscopic models

• Intersection coding (e.g. lane and queueing configurations)
• Signal phasing and timing data

Microscopic transport model network auditing

Microsimulation models can be more sensitive to network errors than larger scale models. It is essential that the modeller ensures that the model network is an accurate representation of the true network.

Some network checks and recommendations include:

• It is recommended that networks are built from site layout drawings and supplemented by Aerial photographs where required
• The modeller should ensure all drawings and aerial photographs are up to date. Accuracy should be checked via site visits where possible.
• Model link and connector overlap should be avoided or minimised
• Node structure should be logical and consistent throughout the model.
• Reduced speed areas should be present on all links where horizontal road geometry is expected to cause driver deceleration
• All conflict points should be managed with appropriate priority treatments or signals
• Signal operations should be checked against operations sheets, or during site visits, to ensure accuracy
• Any model error files should be checked and justifications provided

For more details on microsimulation networks see Traffic Modelling Guidelines (Transport Roads and Maritime Services, NSW, 2013).

5.3 Calibration and validation

Estimation involves the use of statistical methods to estimate the parameters of the equations of the various components of the transport modelling system (trip generation, distribution, mode choice). Calibration of transport models involves manual and automated procedures to adjust model parameters so the modelled travel patterns, traffic volumes and patronage estimates replicate observed survey data. Estimation of transport models is predicated partly on the availability of large-scale origin–destination (O–D) survey data, which - due to the high costs involved - is becoming scarce. Household travel surveys (see Section 4.1.7) and census journey-to-work data can provide detailed information for model estimation.

The validation process attempts to quantify how accurately a transport model reproduces a set of Reference (Base) Year conditions (such as traffic volumes or patronage estimates) and is used, together with dynamic indicators of the responsiveness of the model to changes in inputs, to define the transport model’s degree of ‘fit-for-purpose’.

The validation process should use data separate to that used in the model calibration. Calibration and estimation data are critical for establishing the parameters and equations used in the transport modelling system. Validation data are critical to testing the overall validity of the model against a set of criteria. Appendix C sets out criteria to be used in assessing the ability of a transport model to reproduce a set of Reference (Base) Year conditions. There are constraints on the quantity of data available for model development that cause tensions in holding data aside for independent validation. Statistical techniques, such as ‘bootstrapping’ can assist to maximise the value of sample data.

In general, transport model validation reporting should cover the following:

• Description of the data used in estimating, calibrating and validating the model
• Reporting on the ‘fit’ achieved to the estimation and calibration data
• Reporting on the validation outcomes for a Reference (Base) Year.

5.4 Vehicle operating costs and the value of time

Part PV2 of the Guidelines provide estimates of various road use unit costs (such as value of time, crash costs and vehicle operating costs) for Australia. These are suitable for using in appraisal and may be interpreted as input parameter values of behavioural time and vehicle operating costs in the transport modelling process. Part PV1 of the Guidelines provides a detailed coverage of parameter values for public transport appraisal.

5.5 Generalised cost weightings

When calculating the generalised cost of public transport travel, it is usual practice to weight the different components of public transport travel to reflect the passengers’ utility (or perception of utility) for each component. Part 1 of NGTSM06 Volume 4 provides a detailed discussion of these weights.

5.6 Matrix estimation

Matrix Estimation is a process in which numerical methods are used to find an ‘optimal’ solution to an Origin-Destination (O-D) trip matrix that when assigned to a network will replicate screenline traffic counts. The mathematics behind matrix estimation has undergone a number of developments since it came into use in the 1970’s. Works by Willumsen (1978), Van Zuylen (1980), Cascetta (1984) and Speiss (1987) set the foundations on which most modern matrix estimation software are based. A good summary of this work and an introduction to the mathematics of matrix estimation can be found in Ortúzar, J. d. D. and Willumsen, L. G. (2011).

Generally, the matrix estimation is considered acceptable as a legitimate technique that could be used to improve the representation of an O-D trip matrix for a strategic, mesoscopic or microsimulation model. However the user needs to pay special care when applying the technique to ensure the estimated matrix representing the true pattern and not be distorted by the process.

Input data into the matrix estimation process can include trip ends, travel costs, a prior matrix together with traffic counts Confidence weights are used to constrain the solution best to reflect these inputs.
Matrix Estimation can be thought of as a search for a matrix that fits the data while staying consistent with other available inputs or assumptions. It can be a powerful low cost method to estimate an O-D matrix when direct observation is not feasible.

However care needs to be taken to avoid finding a spurious result that fits traffic count data at the expense of reason or is contrary to other reliable sources of data. In all but the most simple of matrices, there will not be a unique O-D matrix that when assigned to a network will reproduce the traffic counts.

The misuse of Matrix Estimation can result in apparently well validated traffic models that have origin-destination patterns that greatly differ to actual trip patterns and are consequently unsuitable for forecasting. This potential for misuse has often led agencies model end users to specifically require that Matrix Estimation is not used when requesting transport modelling work.

The following sections provide information on common pitfalls as well as guidance which should assist the modeller in the appropriate use of Matrix Estimation.

The modeller should always acknowledge when Matrix Estimation has been used in the calibration of a transport model. The acknowledgement will assist reviewers and users of the model to make a more informed assessment of the model validation and to the method of forecasting. It is important that the provenance of the matrix should be fully documented and repeated sequential use of matrix estimation as models are refined should in most cases be avoided.

5.6.1 Input Data

Traffic counts

The traffic counts should be grouped into short screenlines by direction covering corridors of traffic movement. The use of individual counts (single count screenlines) should be avoided where possible unless they represent a distinguishable transport corridor.

The modeller should also ensure that the set of traffic counts being used in the matrix estimation are internally consistent, that is, they ‘add up’ along a route. Inconsistency can result in localised over or underestimating trip ends in an attempt to match the incompatible traffic counts.

It is recommended that a selection of traffic count data is withheld from the matrix estimation to allow for ‘out of sample’ testing. These counts may be spot counts or screenline counts. In doing so, the modeller will provide some evidence that the linkage to the base model has not been broken.

Trip Ends

Trip Ends may be sourced from models or observed data.

Prior Matrix

A prior matrix can be input into the matrix estimation. Typically the prior matrix will be the output of a four step model or obtained through surveys.

For many models, observed trip matrices are very sparse with many empty cells. The Matrix Estimation process can only adjust non-zero cells.

If the modeller has reason to believe a cell is empty due to the limitations of the survey and are not justified (for example conflicts with the trip end inputs), the modeller can ‘seed’ these cells with small non-zero values. The seeding will increase the flexibility of the process without significantly changing the total size of the prior matrix.

If the modeller has reason to believe a substantial proportion of cells are empty due to the limitations of the survey matrix estimation should not be applied; consideration may be given to constraining synthetic matrices at a sector level, or similar techniques applied to develop a suitably structured prior matrix.

If the modeller finds that the matrix estimation is making large changes to either the trip ends or trip length distribution it may be necessary to growth factor the prior matrix before its use in matrix estimation. Matrix estimation should not be applied in itself to make significant adjustments to matrices.

The UK WebTag guidelines Unit M3.1 Highway Assignment modelling – Jan 2014 provides prescriptive steps for the use of matrix estimation. The reader will find that WebTag proscribes a strict minimalist implementation of matrix estimation to avoid all but the smallest changes to the prior matrix. Such strictness is arguably less appropriate in the Australian context where confidence in the prior matrix is typically lower than in UK, therefore if required matrix estimation could be used to improve the OD matrix with quality traffic counts and trip ends.

5.6.2 Confidence levels

In most cases, the confidence in the input data is proportional to the accuracy of the data. Data errors arise from sources including measurement error, sampling error and synthesis of modelling error.

A common result is:

Traffic count confidence > Trip End confidence > Trip Cost confidence >Trip pattern confidence

Typically, a relatively high confidence will be assigned to screenline traffic counts. This is a reasonable approach if the counts are recent and are of good quality. However, the modeller may need to reduce the confidence to some of the traffic counts in consideration of the traffic survey methodology. For example, a manual traffic count taken on a single day would be expected to be less accurate than an automatic tube count taken over two weeks, and counts of heavy vehicles may be less accurate than counts of all vehicles.

The trip end input data to matrix estimation is typically obtained through observation (surveys) and/or through reliable demographic data and a trip generation model. The modeller may wish to apply a higher confidence to the observed trip ends compared to those derived from a trip end model.

The confidence on Trip Cost can be relatively high if the cost is equal to distance. However, in most cases the trip cost will include elements of time and/or toll. The inclusion of time, tolls or other penalties into the trip cost will necessarily reduce the confidence in the accuracy of the data, although it will almost certainly improve the correlation between the cost estimates and travel patterns.

5.6.3 Recommended Matrix Estimation Checks

The following is a list of four checks that a modeller should always conduct when using Matrix Estimation. They will help both the modeller and reviewer understand the size and nature of changes to the prior matrix.

1. Assign both the prior and post matrices to the network and view a plot of network differences. This plot will help show where large changes have occurred.
2. Prepare a scatter plot of prior and post trip ends showing line of best fit and R2 (which should be considered with and without large external zones).
3. Prepare a scatter plot of prior and post matrix cells showing line of best fit and R2. (It will usually also be helpful to aggregate the matrices using a sectoring that reflects the broad patterns of movements that may be affected by interventions the model is intended to assess)
4. Compare trip length distributions of the prior and post matrices with and without intrazonals.

The results of checks should be investigated to explain the causes and reported.

In addition to the four tests listed above, the modeller may find value in conducting a select link analysis of one or more of the traffic screenlines to assess the changes that have occurred due to the matrix estimation. By conducting the select link first using the prior matrix followed by the post Matrix Estimation matrix, the modeller is able to use bandwidth plots to examine the location and nature of the changes that have occurred.

Interpreting the results

The optimum result of Matrix Estimation occurs when the process results in a good fit to traffic counts whilst making only minimal changes to the prior matrix. However, if the modeller finds that the process is in fact only making minimal changes to the prior matrix the modeller should reassess the need for using matrix estimation. The ‘cost’ to the model in terms of future year matrix adjustments, reporting requirements and perhaps reputation may exceed the benefits coming from the of improved validation to traffic counts.

In practice, the modeller needs to make a trade-off between the level of fit to traffic counts and the size and nature of the changes to the prior matrix.

The changes to the prior matrix should always be interpreted in comparison to the quality of the prior input data.

For example, a modeller should be concerned if the Matrix Estimation results in large changes to prior trip ends that are the result of a well calibrated trip generation model. In some cases, Matrix estimation may help ‘uncover’ previously overlooked special generators, but this will be the exception.

Conversely, larger changes to the prior trip ends may be expected if the prior trip ends are based on sparse observations or an unsophisticated trip end model.

If changes due to matrix estimation are excessive, the modeller should examine the nature of the changes and review the input data and the prior matrix before rerunning the matrix estimation.

Particular care needs to be taken in examining changes in average trip length and the shape of the trip length distribution as a result of Matrix Estimation.

Large increases or decreases in average trip length are usually a sign that significant changes have been made to the prior matrix.

It is important to compare the prior and post trip length distributions both with and without internals as it will help show if the matrix estimation is increasing the number of short trips to match underestimated traffic counts.

5.6.4 Application to future year matrices

If matrix estimation is used in the calibration of the base year, the changes due to the matrix estimation must be applied to the future year matrices.

There are several approaches to applying the changes. One common method is to find the difference matrix between the post and prior matrices and apply this difference to each of the future year matrices. A shortfall of this approach is that it may underestimate the size of adjustments required in growth areas. Care also needs to be taken to avoid negative values in future year matrices once the adjustment has been made.

A second common method is to apply a factor matrix of the ratio between the post and prior matrices to each of the future years. A shortfall of this approach occurs when the ratio deviates excessively from one. Large ratios can result in unrealistically high cell values in future year matrices once the adjustments have been made.

An alternative approach is to use a combination of difference and factor matrices where the choice to use either approach is dependent on the size of the factor and the future year matrix cell before the adjustment has been made. A good description of the problem is given by Daly, A., Fox, J., Patruni, B., & Milthorpe, F. (2011). In which they describe an eight case method to pivoting.

5.7 Transport network validation

In general terms, transport network validation will usually include:

• Comparing modelled travel times with observed travel times
• Comparing modelled screenline volumes with observed screenline volumes
• Comparing modelled service patronage volumes with observed service patronage volumes
• Checking that paths through the network are realistic
• Comparing distances between specified O–D pairs derived from the model with actual distances
• Comparing modelled public transport travel times with timetables
• Checking public transport routes in terms of stopping patterns, timetables and frequency.

5.8 Assignment validation

The assignment of vehicles or passengers onto a network is typically the final step in the modelling process. To get to this point usually involves the culmination of several sub models and processes. For this reason and perhaps the importance of volumes to project outcomes, the level of assignment validation is of great interest to modellers and model users. Care must be taken tot to give undue weight to this aspect of the model validation, which would risk prejudicing the integrity or performance of other models and degrading the forecasting capability of the whole model system

Assignment validation is traditionally the comparison of measurable model outputs with sets of observations. However, given that transport models are used to forecast how a change in inputs may lead to a change in modelled conditions, some practitioners include assignment sensitivity tests or ‘dynamic validation’ as a component of the assignment validation. While these tests rarely have observed data for comparison, they are relatively easy to perform and can provide useful insights into the nature and reasonableness of the model. Where interventions have occurred within the model and recent changes have been observed it is beneficial to undertake ‘back casting’ to demonstrate how well the model reproduces these observed changes.

5.8.1 Purpose of Model Validation and validation criteria

The purpose of model validation is to give confidence in the ability of the model to replicate a set of observations given a set of base data and consequently have confidence in the fitness of the model to forecast traffic using a set of forecast data.

Model Validation criteria give a benchmark to facilitate both discussion and critical analysis of a model’s performance. In particular, the criteria can assist a transport modeller to identify the strengths and weaknesses of a model and they can contribute to evidence that the model is accurate enough for the desired purpose of the forecasts.

The ability or inability of a model to meet a set of validation criteria against a set of observations is not sufficient to conclude the model’s suitability for forecasting. Within Australia and New Zealand there has been a growing appreciation in the industry, as discussed by Clark (2015), about the value of intelligent use of model guidelines rather than the strict adherence to target values. A similar shift is apparent in the language used in the “Model checking and reasonableness guidelines” by FHWA (2010) compared to earlier versions, and in the reference to ‘acceptability guidelines’ in UK guidance.

The intelligent use of model guidelines could include:

• avoiding over calibrating the model to meet target values
• using targets that are appropriate for the model type and intended purpose (i.e. context specific)
• making a spatial distinction between areas or links of importance to a project
• understanding the limitations and variability of the observed data
• Understanding the stability of the observed data and the influence of new developments on demographics and traffic counts.

The modeller should also be aware of and preferably document in a model validation report the quality and characteristics of

• The model input data (demographics, networks, economic parameters etc.)
• The model functions (demand model, the use of matrix estimation etc.)
• The model validation data

To be able to have confidence in model forecasts, it is of primary importance that the modeller can demonstrate that the model is built on sound foundations. A model validation that relies on unreasonable parameters or unexplained factors is unlikely to be fit for forecasting future conditions.

5.8.2 Agreement of validation criteria

A number of validation statistics are used by modellers to assess and critically compare the differences between modelled values and observed data. Model validation criteria for the different levels of transport modelling have been published by a number of agencies (VicRoads, RMS, NZTA, DRMB) to provide a standardised system of validation reporting for their respective jurisdictions. See for example VicRoads (2010), RMS (2013), NZTA(2010) and DRMB(2010). Each of these sets of criteria differs in both their requirements and strictness.

It is recommended that an agreement is made prior to the commencement of the model validation to a set of appropriate validation criteria that is both suitable for the project and the scale of the model. This may include predetermining the importance of specific criteria for specific locations. For example, it could be agreed that a GEH of less than 5 is achieved for 90% traffic counts within project study area while it is necessary for only 75% of traffic counts outside of project study area.

5.8.3 Validation criteria for traffic volumes

Modelled traffic volumes are used at either the link level or screenline level. Link data are generally concerned with the ability of the model to represent routeing and screenline data with the pattern of regional movements.

Link volume plots

For each time period, a map should be produced of the transport network showing modelled and observed link flows and the differences between them. The totals should be summarised for available screenlines. These plots are used to check modelled and observed flows by geographic area and level of flow.

Scatter plot of modelled and observed flows

• For each time period, XY scatter plots of modelled versus observed flows should be produced for:

• all individual links
• freeway links
• screenlines

Each plot should include the y=x line. Report on the R2 and slope for each plot.

XY Scatter plots give the modeller a quick and easy visual of the match between the model and traffic counts. These plots are included in most validation guidelines including VicRoads, RMS and NZTA.

The modeller should aim for a slope close to one, a high R² and an intercept close to zero. Outliers and points with zero model volume should be investigated.

Screenline percent differences versus screenline volumes

For each time period, the modeller should produce a scatter plot showing the total for each screenline by direction versus the percentage difference compared to counts.

The VicRoads modelling guidelines require comparing each point in these plots to criteria curves that decrease in magnitude as the screenline volume increase. These curves were originally derived by FHWA and are based on both an expectation of the error inherent in traffic counts as well as a judgement on the importance of accuracy for a given traffic volume.

GEH statistic

The GEH statistic, a form of Chi-squared statistic, is designed to be tolerant of larger errors in low flows. It is computed for hourly link flows and also for hourly screenline flows. It is generally not applicable to aggregate period or daily flows. The GEH statistic has the following formulation:

$GEH=\sqrt{\frac{{(v_2-v_1)}^2}{0.5(v_1+v_2)}}$

where V₁ = modelled flow (in vehicles/hour) and V₂ = observed flow (in vehicles/hour).

GEH targets are present in modelling guidelines in NZ, UK and US, although they are not as widely used in Australia. These targets are usually in the form of the desired percentage of links with a GEH

Percentage root mean square error (RMSE)

The RMSE applies to the entire network (as opposed to GEH which is calculated for each link or screenline) and has the following formulation:

$RMSE=\frac{\sqrt{\frac{{(v_1-v_2)}^2}{C-1}}}{\frac{\sum v_2}{C}\times 100}$

where:

V₁ = modelled flow (in vehicles/hour)

V₂ = observed flow (in vehicles/hour)

C = number of count locations in set.

The RMSE statistic is present in the modelling guidelines of Australia, NZ, UK and US.

Mean Absolute Deviation (MAD)

The MAD expressed as percentage is calculated as the sum of the absolute differences between each observed count and modelled volume divided by the number of observed counts. The MAD statistic is not commonly found in transport modelling guidelines. Due to using the absolute difference, the MAD statistic cannot provide information on whether the model is under or overestimating.

It is perhaps most useful for providing a quick single value comparison between models or validation attempts.

Mean Absolute Weighted Deviation (MAWD)

The MAWD expressed as percentage is calculated as the weighted sum of the absolute differences between each observed count and modelled volume divided by the number of observed counts. In the modelling context, the applied weights are determined by the modeller and are usually linked to the importance or confidence of the count. The MAWD statistic is not commonly found in transport modelling guidelines. Like the MAD statistic it is most useful as use as a comparator.

The following table provides examples of target model validation criteria.

Table 6: Examples of Target Model Validation Criteria

Model Type	R2	Slope	Screenline	GEH	RMSE
Strategic four step – Daily	>0.85	0.9-1.1	+-10%	N/A	<30%
Strategic – peak period	>0.85	0.9-1.1	+-10%	60% individual links GEH<5 95% individual links GEH<10 100% individual links GEH<15 GEH<5 for each directional screenline	<30%
Mesoscopic	>0.88	0.9-1.1	+-10%	85% individual links GEH<5 GEH<4 for each directional screenline	<25%
Microscopic	>0.90 (network wide) >0.95 (core area)	0.9-1.1(network wide) 0.95-1.05 (core area)	+-5%	85% individual links GEH<5 GEH<3 for each directional screenline	<20%

5.8.4 Criteria for Travel Times

Travel time validation of a model assignment is an essential component of assignment validation, especially where the purpose of the model is to assess the impacts of a project on route choice, or to provide economic appraisal. Validation of travel times usually consists of the two following checks.

Comparison of Total observed and modelled journey times

The percentage difference is calculated between the average total observed and total modelled journey times for each route by direction for each time period. Modelling guidelines typically set a desired minimum difference or use confidence intervals of the observations to assess the differences.

The comparison of total journey time differences presents some challenges. Firstly, the difference in totals does not provide information on whether the error is systematic or localised. And secondly, the measure does not standardise for the journey length, therefore, all else being equal, longer routes will tend to validate total travel times better than short routes.

The following table provides an example of travel time validation target criteria.

Model Type	% of Travel time routes within the greater of 15% or 1 minute
Strategic	85%
Mesoscopic	90%
Microscopic	90%

Comparison plot of observed and modelled journey times by distance

A comparison plot using cumulative time-distance graphs are a good method of assessing the nature of the differences between modelled and observed journey times. Plots should be shown for each route by direction and by time period.

Some modelling guidelines include the recommendation that 95% confidence intervals for the observed travels are included in the plots.

5.8.5 Sensitivity testing and dynamic validation

In addition to checks of the model against observed data the assignment validation should include some testing of the sensitivity of the model to changes in inputs and assumptions.

These tests may include:

• Modifying volume delay functions for different highway classes to identify whether the changes in assigned traffic are logical
• Modifying public transport fares (for example, increasing fares by 10% and noting the change in forecast patronage and corresponding change in road traffic volumes)
• Modifying the value of time used in the generalised cost and path building formulations, and noting the changes in mode share
• Modifying public transport frequencies and noting the change in forecast public transport patronage and traffic volumes
• Modifying the zonal trip generation (employment and population levels) and noting the change in vehicle-kilometres of travel on the highway network.
• Modifying toll levels and noting the change in toll users.
• Modifying demand matrices and noting the changes in congestion and operation.

5.9 Model convergence

It is necessary to assess the stability of the trip assignment process referred to in Section 3 before the results of the assignment process are used to influence decisions or for input to economic appraisal, or both.

The iterative^[1] nature of the assignment process leads to the issue of defining an appropriate level of assignment convergence. In practical terms, an assignment process may be deemed to have reached convergence when the iteration-to-iteration flow and cost differences on the modelled network are within predetermined criteria.

The recommended indicator for assessing the convergence of urban transport models is the delta (δ) indicator. This indicator is the difference between the costs along the chosen routes and those along the minimum cost routes, summed across the whole network and expressed as a percentage of the minimum costs. An urban transport model is deemed to have reached convergence when δ is less than one per cent (see Table 7).

Summary of convergence for 2031 Base Case two-hr am peak.

Table 7: Example of model convergence output

ITERATION	DELTA		AAD	RAAD	% FLOW
41	1.03119	%	4.53441	0.0029626	0.18298	%
42	0.68327	%	9.86224	0.0080850	2.70688	%
43	0.93032	%	6.95290	0.0047448	0.63728	%

Source: Melbourne Integrated Transport Model

Generally, the number of iterations required to reach convergence increases:

• The more trips are in the demand tables
• The more zones that are in the model
• The more links in the network.

Consideration of the time required for the model to converge should be made during specification of the model.

5.10 Transport model documentation

Transport model documentation is a step towards improving the understanding and usefulness of travel demand models. If the model documentation is too brief, or it is not updated with changes to the model, then it will not be useful to transport modellers.

Model documentation may contain a variety of information. The following is a list of suggested topics:

• Description of the modelled area and transport network coverage
• Land use and demographic data for all years modelled, by transport zone or the level of geography adopted for the modelling and analysis
• Description and summaries of all variables in the networks
• Source and coverage of traffic counts used in the model development process
• Description of the trip generation model and any core assumptions
• Identification of special generator and external trips input to trip generation
• Summary of trip generation results
• Description of the trip distribution model and any core assumptions
• Description of the impedance measures used in trip distribution, including intra-zonal and terminal times
• Summary of trip distribution results
• Description of the mode choice model by trip purpose
• Description of the variables used in the mode choice model
• Summary of the mode choice results
• Identification of the source and value of inter-regional trips
• Description, if applicable, of the time period models and any core assumptions
• Description of the trip assignment models and any core assumptions
• Description of the impedance measures used in the trip assignment models
• Identification of the volume–delay and path-building algorithms applied in trip assignment
• Summary of the trip assignment results, for traffic (vehicle-kilometres of travel, vehicle-hours of travel, passenger kilometres travelled, delay and average speed), and for public transport (passenger-km by mode, passenger boardings, interchange, passenger travel time)
• Summary of model parameters and statistics obtained during the estimation of model components
• Identification of model validation tests and results for each model step.

5.11 Auditing

In common with many other activities, transport modelling quality is defined by process. This means that a forecast traffic volume or patronage level cannot be determined as being ‘good’ simply by looking at the forecasts. Instead, confidence in the processes used to derive the forecasts should be sought via the structure of the transport model and its calibration and validation. It should be stressed that forecasts are only as good as the input assumptions.
Generally, using transport modelling in the appraisal of initiatives and impact assessment involves four broad processes:

• Data collection
• Model specification
• Model estimation and calibration
• Model application to the scheme appraisal.

Each of these processes should either follow accepted guidelines or accord with good practice. The following is a suggested list of information that should provide a suitable basis for the evaluation of transport models in the context of the stated objectives of users:

• A statement of the modelling objectives and the elements of the model specification that serve to meet them
• a specification of the base data:
  • description of travel surveys
  • sample sizes
  • bias assessments and validation, where available
• description of transport networks
  • structure
  • sources of network data (such as inventory surveys, timetables)
• description of demographic and employment data (such as sources, summary statistics)
• A document reporting on model specification and model estimation:
  • model structures, variables and coefficients
  • outputs of statistical estimation procedures
  • model fit to data
• Evidence of validation:
  • fit to independent data
  • comparison with other models
  • sensitivity tests and elasticities
• Description of the forecast year inputs (such as networks, demographic data, economic assumptions):
  • sources of data
  • statistics describing the main features of the data
• Documented validation of the forecasts, paying attention to the types of model runs and types of output most vulnerable to error (e.g. tests of small changes, economic benefit estimates):
  • comparison with other forecasts, where available
  • comparison with historic trends, if relevant
  • reasoned explanations of the forecasts (such as the sources of the diverted traffic, the reasons for diversion – size of time saving)
• A record of model applications.

Appendix D provides a Model Audit Checklist.

[1] Feedback of generalised travel costs derived from the trip assignment process to the trip generation and mode split sub-models within an urban transport model, until a pre-defined level of convergence is achieved.