Implementing Time Series Cross Validation to Evaluate the Forecasting Model Performance
Abstract
Theoretically, forecast error increases as the forecast horizon increases. This study aims to assess whether the statement is generally accepted or not. This study applies time series cross-validation to evaluate forecasting results up to seven steps ahead. As an illustration, we use Malaysia’s hourly electricity load data. Each hour is considered a series of each, so there are 24 daily series. Time series cross-validation with a 334 window was applied to 24 data series, and then each daily series was modeled with the Autoregressive Integrated Moving Average (ARIMA), Neural Network Autoregressive (NNAR), ExponenTial Smoothing (ETS), Singular Spectrum Analysis (SSA), and General Regression Neural Network (GRNN) models. In terms of mean absolute percentage error (MAPE) from one to seven steps ahead, we then evaluate the performance of all models. The experimental results show that the MAPEs obtained from the GRNN model tend to increase along with the theory. However, MAPEs obtained from ETS increase by up to three steps ahead and decrease after that. Among the five models, ARIMA, NNAR, and SSA produce a reasonably stable MAPE value for one to seven steps ahead. However, SSA has the most stable error value compared to ARIMA and NNAR.
Keywords: time series, cross-validation, evaluate, forecasting model performance
References
[1] Tashman LJ. Out-of-sample tests of forecasting accuracy: an analysis and review. Int J Forecast. 2000;16(4):437–50.
[2] Haris NA, Aziz AA, Nor NA, Sharif N. Improving Air Pollution Index (API) predictive accuracy using time series cross-validation technique. Journal of Fundamental and Applied Sciences. 2018;10( June):1257–67.
[3] R.J. Hyndman and G. Athanasopoulos, Forecasting: principles and practice. OTexts, 2018.
[4] Liu X, Yang MC. Simultaneous curve registration and clustering for functional data. Comput Stat Data Anal. 2009;53(4):1361–13776.
[5] R.J. Hyndman and G. Athanasopoulos, Forecasting: principles and practice. OTexts, 2018.
[6] De Livera AM, Hyndman RJ, Snyder RD. Forecasting time series with complex seasonal patterns using exponential smoothing. J Am Stat Assoc. 2011;106(496):1513–27.
[7] Lee JY, Kim S. Forecasting daily peak load by time series model with temperature and special days effect. The Korean Journal of Applied Statistics. 2019;32(1):161–71.
[8] Soares LJ, Medeiros MC. Modeling and forecasting short-term electricity load: A comparison of methods with an application to Brazilian data. Int J Forecast. 2008;24(4):630–44.
[9] Sulandari W, Subanar S, Suhartono S, Utami H, Lee MH, Rodrigues PC. SSA-based hybrid forecasting models and applications. Bulletin of Electrical Engineering and Informatics. 2020;9(5):2178–88.
[10] Arora S, Taylor JW. Short-term forecasting of anomalous load using rule-based triple seasonal methods. IEEE Trans Power Syst. 2013;28(3):3235–42.
[11] Sulandari W, Utami H. “Forecasting time series with trend and seasonal patterns based on SSA.,” In: 2017 3rd International Conference on Science in Information Technology (ICSITech). pp. 648–653. IEEE (2017). https://doi.org/10.1109/ICSITech.2017.8257193.
[12] Sulandari W, Subanar S, Lee MH, Rodrigues PC. “Time series forecasting using singular spectrum analysis, fuzzy systems and neural networks.,” MethodsX. vol. 7, p. 2020. https://doi.org/10.1016/j.mex.2020.101015.
[13] Box GE, Jenkins GM, Reinsel GC, Ljung GM. Time series analysis: forecasting and control. John Wiley & Sons; 2015.
[14] Hyndman R, Koehler AB, Ord JK, Snyder RD. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media; 2008. https://doi.org/10.1007/978-3-540-71918-2.
[15] Golyandina N, Korobeynikov A. Basic singular spectrum analysis and forecasting with R. Comput Stat Data Anal. 2014;71:934–54.
[16] Golyandina N, Korobeynikov A, Zhigljavsky A. Singular spectrum analysis with R. Springer Berlin Heidelberg; 2018. https://doi.org/10.1007/978-3-662-57380-8.
[17] Specht DF. Brief Papers A General Regression Neural Network.
[18] Martínez F, Charte F, Rivera AJ, Frías MP. Automatic time series forecasting with GRNN: A comparison with other models. In International Work-Conference on Artificial Neural Networks. Cham: Springer International Publishing; 2019. pp. 198– 209.
[19] Sadei HJ, Silva PC, Guimaraes FG, Lee MH. Short-term load forecasting by using a combined method of convolutional neural networks and fuzzy time series. Energy. 2019;175:365–77.
[20] Soares LJ, Medeiros MC. Modeling and forecasting short-term electricity load: A comparison of methods with an application to Brazilian data. Int J Forecast. 2008;24(4):630–44.
[21] Sulandari W, Subanar S, Suhartono S, Utami H, Lee MH, Rodrigues PC. SSA-based hybrid forecasting models and applications. Bulletin of Electrical Engineering and Informatics. 2020;9(5):2178–88.
[22] Hyndman RJ, Khandakar Y. Automatic time series forecasting: The forecast Package for R. J Stat Softw. 2008;27(3):22. Bottom of Form