Should you trust forecasts?

Author: Hannah Mason

With so much information at our fingertips, working out which data to trust can be an overwhelming task. To help you with this, we have taken a look back in order to explore the reliability of our research and provide a retrospective understanding of how accurate our pay forecast analyses have been.

In autumn 2023, we conducted our biannual survey of forecasts for pay awards in the next 12 months, where organisations were asked to give us their estimate of their next pay rise. These forecasts can help employers understand what a typical increase might look like for the next year and assess whether or not their own offering to employees is in line with the general trend.

We estimate that around two-thirds of pay award budgets are set between September and January, and our two forecasts surveys are conducted in this timeframe. Based on this, we might assume that individuals responding to our surveys make their pay forecast on plausible information communicated to them. We have put this theory to the test by investigating how these forecasts compare with the pay rises ultimately made by each organisation.

More than 50 years of data

At XpertHR, we have been collecting pay settlement information for more than 50 years. As of November 2023, we had electronic records going back to 1984 comprising more than 51,000 pay settlements from around 8,000 organisations. This data is collected in a variety of ways, for instance researchers contacting organisations, an ongoing online survey, recording details of published pay awards, etc. For more information on how your organisation can take part and access the results, click here. We publish a monthly analysis of pay settlements that have come into effect in the last quarter to help keep track of how the pay award landscape is evolving.

In this analysis, forecasts were matched to settlements made in the 12 months following the survey to help understand the accuracy of the forecasts. Not all organisations that completed our survey also submitted their awarded settlement, so our analysis is based only on those that provided us with both a forecast and the settlement information for the year following the survey.

2022/2023 findings

The findings from the most recent year are promising, highlighting small differences between the pay forecasts and the final award value. Forecasts for this period were based on responses to our August/September 2022 and January/February 2023 surveys. We were able to match a total of 161 employee groups, which is around one-third of all forecasts we received.

The key findings are as follows:

• our overall forecast was similar to the actual pay award, with the median pay forecast (the midpoint of all estimates) being 5%, while the median actual pay award was 6%;
• around half of estimates were within one percentage point of the true award value; and
• a quarter of all forecasts were exact predictions (the forecast was equal to the actual settlement) for the next award.

Factors impacting the accuracy of forecasts

One-third of forecasts were found to be underestimations of the given award, where the award ultimately made was more than one percentage point higher than the forecast.

Examining this finding in the context of the 2022/2023 period is important. Measures of inflation had begun increasing rapidly from the start of 2022 and were still rising in August/September, when we conducted our survey. By the time of our survey in January/February 2023, inflation remained high and in double figures.

With the high levels of inflation and pressure from employees and trade unions, predicting the upcoming pay award proved challenging. We saw examples of organisations topping up pay awards; for example, giving two awards within a year, or bringing forward the date the award was made. Due to this uncertainty, it is probable that organisations were not expecting to grant awards as high as they ended up giving; and with settlement values agreed or adjusted closer to the effective date than typical, all of these factors may have impacted predictions.

Based on our analysis, we can see that pay rises that came into effect closer in time to the survey were more accurate. Around three-fifths (58.3%) of settlements that came into effect within the six months following the forecast being made were accurate (within one percentage point). In contrast, just two-fifths (41.5%) of awards that came into effect more than six months after the forecast was made were within one percentage point. This indicates that organisations were less able to accurately predict the value of their next award when the prediction was made further in advance.

To try to combat this issue, we also assessed the difference between forecasts and awards in the most recent period that was not exposed to the issues of COVID-19 and the cost-of-living crisis - 2018/2019. For this timeframe, all settlements would have had to come into effect by 31 December 2019 to be included in our analysis, so were not impacted by the onset of the pandemic, which likely provided a great deal of uncertainty for organisations and influenced their pay awards.

Comparing our 2022/2023 analysis with this earlier period also helps to build on our findings to provide an insight into the accuracy of our forecasts over several years. When you are doing research, you want to see consistency in your findings over time in order to have confidence in your results.

2018/2019 findings

The findings for the 2018/2019 period are based on a similar number of matched settlements to the 2022/2023 period, containing information representing 164 employee groups. Forecasts were recorded from the XpertHR August/September 2018 and January/February 2019 pay forecasts surveys.

The key findings are as follows:

• forecasts correctly predicted the value of awards, with the median for both being 2.5%;
• the majority (82.3%) of estimates were within one percentage point of the actual award value; and
• one-third of all forecasts were exact predictions for the next award (where the forecast was equal to the actual settlement).

Despite our forecasts matching the settlement median exactly, we were still keen to understand the impact that the time between the forecast and the deal coming into effect had on the accuracy of the estimates. With this information, it would help us to assess how best to communicate our forecasts in the future. If we found that forecasts were more accurate if there was limited time between providing the estimate and the award coming into effect, it would allow us to consider whether we should still include predictions that were further in the future.

The results for the 2018/2019 period contrasted with the most recent 2022/2023 findings, with there being very little difference in the accuracy based on the time between forecast and the actual settlement being made. For 2018/2019, 83.7% of deals that came into effect up to six months after the forecast were within one percentage point of the prediction, showing a good degree of accuracy. A similar proportion (80.3%) of settlements with effective dates more than six months on from the forecasts were within one percentage point. This shows that for the earlier data, forecasts displayed a high level of accuracy, even when the predictions were made well in advance of the pay deal coming into effect.

What we can conclude from our analysis

Overall, our forecasts provided a good indication of the future median pay award. While the most recent 2022/2023 data showed a small level of variability, the findings were promising and using the median forecast would have provided a slightly conservative estimate compared with the actual award value. Given that 2023 saw the highest median pay award in over 30 years, it's understandable that organisations may not have foreseen precisely where pay awards were heading. For 2018/2019, our median forecast was spot-on.

Although we were able to match only around one-third of forecasts to the actual settlement, we are happy that this sample looks to be representative of all deals across the whole economy. In the 2022/2023 period, based on our matched sample the median actual settlement was 6%, while the whole-economy median stood at 5.7%. For 2018/2019, both our matched sample pay award and whole-economy medians were 2.5%.

Take part and improve our predictions

It is important to take our pay award forecasts as they are intended - a best guess as to what will happen. One way to ensure the reliability of our estimates is for more people to share what is happening at their organisation with us, therefore if you would like to be kept up to date with our latest surveys, please sign up to receive details of our upcoming surveys or provide us with details of your organisation's latest pay settlement.

Additional resources

• Pay trends January 2024: A first look at new year deals
• Six strategies to attract talent and remain competitive in 2024
• XpertHR surveys