Predicting business outcomes often happens in the form of forecasts. These carry the implicit assumption “what happened in the past reflects the future”. This is very often the case. However, external factors are present in the market and obviously have huge impacts on business outcomes. Taking these factors into account is critical for making reasonable projections of business outcomes, and in the process, can identify the market sensitivity to these external factors. Often it is just as important to know “how sensitive is the market to factor X?” as it is “when do we expect the market to recover?” Taking external factors into account requires multi-layer forecasting, since the external factors themselves need to be projected into the future in order to support forecasts of a metric of interest. What’s the process for making predictions with multi-layer forecasting? Let’s discuss a few guiding principles to get us started.
Principle 1: Stack models to achieve layered forecasts
Before we consider which models to use, or how to stick them together, let’s talk about the first principle of multi-layer forecasting. The goal of layering forecasts is to integrate more information to better inform a prediction. Not only do we wish to forecast the metric of interest (in today’s example, the effect of COVID case load on business outcomes), but we also want to predict the underlying drivers that influence this metric.
Below is a diagram showing three unique values and a time threshold, which could be today or the end of a known or observed period.
There are many types of values that can work together in a stacked forecast, but here is one set of labels that would make sense for a COVID to sales scenario.
Principle 2: All models should make sense
The “smell test” or “common sense test” should apply to all models. The primary metric should be predicted within reasonable limits: in our scenario, the unemployment rate shouldn’t be 30%, but it also shouldn’t be 0.01%. It’s important to note: be prepared for outcomes that are dire, or in some cases, rosier than expected. Of course, simple plots often do enough to show whether something makes sense, but be careful of plots that don’t show enough scale to be useful. Sometimes, a log scale is needed to fit the data and trends onto one graph.
Data types are also an important consideration, since they drive the model. We are in the era of machine learning, which has few assumptions, is often incredibly accurate with only moderate amounts of work, and has become easier than ever to use. Machine learning models have just a few drawbacks:
- ML models are not readily interpretable. Few give us the ability to see the effects of an individual term on an outcome.
- ML models don’t readily offer us confidence intervals, which are an important way to check the feasibility of our models.
- ML models often exchange low error for high variance. In our case, we can accept some error, but want the forecasts from our model to have a low volatility, also known as variance.
- Few ML models offer constraints on the outcome. The most important of these is the Poisson constraint, i.e., the restriction that the output can never be negative. It doesn’t make sense, for instance, to have negative sales, a negative unemployment rate, or negative COVID cases. These are all strictly positive, and any model which can deviate below zero, likely will at some future point.
Principle 3: Use common sense to anticipate driving factors
Creating a meaningful forecast can be overwhelming, so start simple. Often just a few of the most critical factors are enough to get reasonable projections. Why not use hundreds? In this rare case, it may be wiser to reduce the number of driving factors to a much smaller range, say anywhere from 1 to 5 “support” models. Why? Uncertainty stacks, and since we also have to forecast each of the driving factors, the model can quickly become overly sensitive to the model fits of the driving factor forecasts. We are seeking simplicity and even more so, stability. We don’t want drastic sensitivities to tiny deviations in a future forecast.
Step-by-step instructions for multi-layer forecasting
- Choose metric of interest and collect proposed driving factors. Optionally, you can assess correlations between metric of interest and driving factors, reduce to those which are highly (but not above say 80%) correlated to the metric of interest, and try to select driving factors which are not highly correlated to each other. Use common sense to select those that you might want to later explain the relationship to the metric of interest. For instance, you could state “as unemployment decreases by 1%, and with COVID cases held constant, sales increase by $4 million dollars a quarter in the Southwest region.”
- Create forecasts for each of the driving factors. These should extend to the range desired for forecasting the metric of interest.
- Create a forecast of the metric of interest which uses the forecasts of driving factors as “exogenous” or “endogenous” factors as appropriate. Specification of this model depends on knowledge of the mathematics or statistics of the chosen forecasting method. Note that most forecasting methods do not accept external variables.
Example Walkthrough
1. Choosing metric of interest and collect proposed driving factors
Let’s say the metric we want to predict is sales, divided up by region. We suspect that the number of active COVID cases in the US might be driving a change in the amount of sales. Also, we believe that unemployment rate is a factor affecting sales as well.
More advanced users can gather dozens or even hundreds of factors, anything from sales in other sectors to average daily temperature to the Gross Domestic Product of Japan. Econometric factors have complicated and sometimes spurious relationships to a given metric, and so discerning whether the relationship is cause or effect is not usually possible. Common sense prevails. Look for high correlation and autocorrelation between your proposed factor and the factor of interest.
2. Creating forecasts for each of the driving factors
Now it is time to build forecasts of any underlying factors. All these forecasts should be extended to match the duration of the original metric of interest. First, we take published unemployment rates, and then extend them out to the next year, training the model (in this case, a time series model based on the negative binomial distribution), and give the date of February 2020 as an intervention date. An intervention is a way to tell the model that there was a disruptive or anomalous event. The model terms and intervention term are all statistically significant.
Similarly, we take the Institute for Health Metrics and Evaluation (IHME) forecast for COVID-19 cases across the United States, and project it out a year past the last date of their projection. This model used an intervention point at the partial reopening of most states, in roughly May of 2020. This model uses a Poisson distribution, which is appropriate for counting events.
3. Creating a forecast of the metric of interest
Lastly, the two underlying factors, unemployment rate and COVID-19 positive case count, are added to a time series forecasting model, along with the two intervention times. The underlying factors are treated as external variables, which makes this model synonymous with commonly used GARCH model for econometrics. This model is again using the Poisson assumption, since sale in US dollars are strictly positive, count-like values. No confidence intervals or accuracies are shown, since the goal of this walk-through is to highlight the layered nature of forecasting, as well as demonstrate how forecasting during an anomalous period of time can work to improve your ability to predict the trends within your business.
Summary
It takes a few steps to achieve an advanced forecast of a metric of interest. However, when external factors are at play, which they very often are, these predictions are well worth the work. Two scenarios in particular make good candidates for this type of layered forecast: pandemics and recessions. As we have seen in 2020, they very likely are related, and very likely drive each other.
Being able to predict sales given these economic volatilities can be a huge advantage over presuming that sales will likely follow past trends. The usual assumptions of forecasting models include the assumption that future values will simply follow the pattern of past patterns. When we have good reason to suspect large market shifting forces at play, we serve our business well by taking these factors into account. The results can help guide the business to new strategies and towards a realistic and informed response to shifting economic factors.
Like what you see?
Author
-
Your additional text goes here
