Metric of the Quarter

Metric of the Quarter:
Measuring Call Center Forecast Accuracy

By Maggie Klenke

Forecasting is the foundation of the entire workforce management process. If the forecast is accurate, the schedules that match it will plan for the staff to be in the right place at the right time, and the intraday adjustments should be minimal. But when the forecast is off target, even the most carefully created schedules have little chance of success and there is likely to be a more chaotic intraday process. Therefore, taking the time to get the forecast right produces multiple benefits.

Measuring the accuracy of the forecast is a good place to start so that you know how close or far off the target the current process is. The most common way to analyze the accuracy of a forecast is to measure the percent of variation between the forecast and the actual results. It is important to measure the call volume as well as the average handle time accuracy separately so that you can identify any anomalies that occur in each of them. Since one is multiplied by the other to compute the total workload, errors in either have significant effects. However, if you only analyze one or combine them into the workload before your analysis for accuracy, you are likely to find it difficult to find the root cause of any patterns of discrepancy.

The following example shows the forecast call volume for the entire day for each day of the week this call center is open. The actual call volume in also shown along with the percent that the actual varies from the forecast, calculated using the difference between forecast and actual divided by the forecast. (Some analysts prefer to divide by the Actual rather than the Forecast. Either is acceptable as long as you understand what you are measuring and are consistent throughout the analysis.)

swpp-fall-2015-metric-graph-01

In this case, we have about a 4% variance for the week and only the low call volume Saturday has a higher than 5% variance.  Now let’s compare this to another week in which the variance for the week is less than 1%.

swpp-fall-2015-metric-graph-02

As you can see, the total weekly variance is much lower for the second example because the plus 29% and the minus 22% essentially offset each other.  Which one of these call centers do you want to manage?  The first one has at least some hope of making the intraday adjustments needed by adding people to make up the 4% variance, but the second one is going to be crazy!  There is no realistic hope to add enough people to meet the requirements for 29% more calls than planned and the overstaffing on Friday will the opposite challenge.  This analysis points out the fallacy of looking at forecasting accuracy over long intervals.  What might look great at the weekly level could be widely variable by day.

Measuring how accurate your forecast is serves as a good metric to use in measuring the performance of the forecasting team.  But it has a more important role as well.  As you consider measuring forecasting accuracy as a way to identify how to improve, you can see the need to analyze accuracy all the way down to the half- or quarter-hour level.  When you see that the 2:00pm to 2:30pm period each day seems to experience consistently higher than forecast call volume or the 8:30pm to 9:00pm period seems to have a consistently lower than forecast AHT, you can begin to formulate a plan to dig into the details of those periods and identify why these inaccuracies are showing up.  When you only know that the day was off by 4%, you don’t really have enough information to look for improvements.

The question comes up frequently as to what is the “industry standard” for forecasting accuracy.  Frankly, there just is no standard that makes sense to all types of companies or even to different call types within the same center.  A utility or technical support team is subject to a huge influx of calls when there is some kind of failure in their network while a customer support team might have a very stable workload from one hour or day to the next.  A sales organization may be responding to TV ads that create rushes of calls over 5 to 10 minute periods and then long periods of idleness.  We recommend that each organization measure the accuracy of the call volume and AHT forecasts for each type of contact that they handle and develop a plan for small but continuous improvements over time.  This is a much more realistic approach than to look to others to set a standard for acceptable performance.  After all, if your accuracy is as good or better than some other company, does it make sense to quit trying to make it better?

Standard Deviation

Another analysis that is useful when measuring forecasting accuracy is to use standard deviation.  When you are looking at a small sample of data such as the previous tables, you can easily see how varied the results are.  But when you look at all the half-hours for a week or month, it is much more difficult.  A table with a year of half-hourly data for several separate skills can run to thousands of lines in a spreadsheet.  Standard deviation provides an easy way to see how much variance there is in the percentages.  And the good news is that you can just use a simple tool like Excel to perform the analysis.  (Using tools such as Excel Pivot Tables can also make it possible to discern patterns across large amounts of data.  This can reveal day of week or time of day patterns as well as seasonal variations.  Every element of the data can be analyzed with a wide variety of calculations quickly and easily.)

metric-bellcurveWhen you calculate the standard deviation you will get a single number and it is important to understand what that number means and how to use it.  In the drawing you see a typical Bell curve distribution with the average or mean noted at the center of the Bell curve.  When you know what the mean is of your data, you can see that by adding one standard deviation to that mean or subtracting one standard deviation from it, we will have covered 68% of all of the data in our analysis.  If we want to expand that to cover 95% of all the data, we add and subtract one more standard deviation from the mean.

Let’s apply this to an example in the forecasting accuracy analysis.  Let’s say that the AHT goal is 320 seconds and the average or mean AHT is in fact 320 seconds. Calculation of the standard deviation results in a metric of 32 seconds.  That means that if we add 32 seconds to the mean of 320 seconds and subtract 32 seconds from that mean, we find that 68% of the agents have AHTs between 288 and 352 seconds.  If we want to pick up 95% of the agents, the range of AHTs is 256 to 384 which is quite a wide spread.  We want to locate those 5% of agents who are lower than 256 and higher than 384 and focus some attention on them.  As we improve the ones who are considered “outliers” we can bring down the standard deviation.  Simply put, the smaller the standard deviation the better, since it means that our agents are more tightly clustered around the average.

You can also use standard deviation to analyze the percent variance in the forecast versus actual, calibrate scores in quality monitoring, look at the variance in absenteeism, and a host of other possibilities.  Doing it in Excel is pretty simple as you simply select the function STANDEV from the math functions (in the drop down box next to the Auto Sum character) and highlight the column or row of items you want to analyze and the result is displayed.

Correlation Coefficient

There is another tool you might want to consider to analyze your forecasting accuracy to help find where a problem might exist.  It is called correlation coefficients and it is a mechanism that allows comparison of one pattern to another.  It doesn’t matter if the numbers in the list being compared are wildly different from another; it only looks at the pattern.  Let’s use the example below to explore how this might be useful.

swpp-fall-2015-metric-graph-03

While the data in Week 2 is dramatically lower than in Week 1, the correlation coefficient is 1.0, which indicates that the distribution pattern is a perfect match.  In fact, Week 2 is exactly one-third of Week 1 in each day so from a pure pattern perspective they are the same.  Now let’s look at a different example.

swpp-fall-2015-metric-graph-04

In this case, the total call volume for the week is exactly the same, but the way the calls distribute among the days of the week is entirely different.  In this case, the correlation coefficient is less than the perfect match of 1.0. It is 0.68, which represents a very poor match.  (You want a number as high as possible here and as close to 1.0 as possible to show a good match.)

This can also be done easily in tools like Excel.  Once again setting up the data much as is shown in the chart above, you would choose the math function CORREL and highlight the two sets of data you want to compare.  Next, highlight each of the rows or columns of data (making sure they each have the same number of cells) and the system will compute the coefficient.

This is a great tool to look for any significant differences in the patterns of call arrivals in the two sets of data such as forecast and actual.  It is also very useful in cycle planning when searching for a pattern that matches up to the causal event that you are trying to forecast such as a weather pattern, marketing campaign, mailing, etc.

When you have analyzed your half-hourly data for percent variance, standard deviation and correlation coefficient, you will have the information you need to begin the search for ways to improve your forecasting accuracy.  If certain periods seem to be the typical trouble spots, dig further to identify the reasons.  When the reasons are better known, you can decide whether to try to fix the cause of the variances or simply build them into your forecast as inevitable.  Either way, forecasting accuracy should be improved.

Maggie Klenke was a Co-Founder of The Call Center School.  She has written numerous call center management books, including Business School Essentials for Call Center Leaders.  She may be contacted at Maggie.klenke@mindspring.com or 615-651-3324.