Evaluating the Effectiveness of Early Warning Indicators: An Application of Receiver Operating Characteristic Curve Approach to Panel Data
DOI:
https://doi.org/10.47743/saeb-2022-0025Keywords:
EWIs, ROC, area under the curve, shrinkage regressions.Abstract
Early warning indicators (EWIs) of banking crises should ideally be judged on how well they function in relation to the choice issue faced by macroprudential policymakers. However, the effectiveness of EWIs depends upon the strength of the predicting power, stability, and timeliness of the signal. Using a balanced panel of 6 countries’ experience with banking and currency crises in recent times, this paper evaluates the effectiveness of EWIs using Receiver Operating Characteristics. Following the drivers of the banking crisis and currency crisis, the paper evaluates the effectiveness of aggregate credit growth, sectoral deployment of credit along and other macroeconomic indicators generally used as EWI. The paper observes that credit disbursements to non-financial sectors and the central government provides stable signals about systemic risks. Further debt service ratio, interbank rates and total reserves are also found to be useful in predicting these crises. Lastly, the effective EWIs are combined using shrinkage regression methods to evaluate the improvement of signal strength of the combination of EWIs. The predictive power of the combination of EWIs provides better signal strength in predicting the macroprudential crisis.
References
Alessi, L., Antunes, A., Babecky, J., Baltussen, S., Behn, M., Bonfim, D., . . . Zigraiova, D. (2015). Comparing Different Early Warning Systems: Results from a Horse Race Competition Among Members of the Macro-Prudential Research Network. MRPA Working Paper, 62914. Retrieved from https://mpra.ub.uni-muenchen.de/62194/1/MPRA_paper_62194.pdf
Alessi, L., & Detken, C. (2011). Quasi-real time early warning indicators for costly asset price boom/bust cycles: A role for global liquidity. European Journal of Political Economy, 27(3), 520-533. http://dx.doi.org/10.1016/j.ejpoleco.2011.01.003
Allaj, E., & Sanfelici, S. (2020). Early Warning Systems for Identifying Financial Instability SSRN Working Paper, 3738936. http://dx.doi.org/10.2139/ssrn.3738936
Arzamasov, V., & Penikas, H. (2014). A Financial Stability Index for Israel. Procedia Computer Science, 31, 985-994. http://dx.doi.org/10.1016/j.procs.2014.05.351
Babecký, J., Havranek, T., Mateju, J., Rusnak, M., Smidkova, K., & Vasicek, B. (2011). Early Warning Indicators of Economic Crises: Evidence from a Panel of 40 Developed Countries. CNB Working Paper, 8. Retrieved from https://www.researchgate.net/publication/241754213_Early_Warning_Indicators_of_Economic_Crises_Evidence_from_a_Panel_of_40_Developed_Countries
Babecký, J., Havranek, T. J., Mateju, J., Rusnak, M., Smidkova, K., & Vasicek, B. (2014). Banking, Debt, And Currency Crises in Developed Countries: Stylized Facts and Early Warning Indicators. Journal of Financial Stability, 15(December), 1-17. http://dx.doi.org/10.1016/j.jfs.2014.07.001
Balakrishnan, R., Danninger, S., Elekdag, S., & Tytell, I. (2011). The Transmission of Financial Stress from Advanced to Emerging Economies. Emerging Markets Finance & Trade, 47(sup2), 40-68. http://dx.doi.org/10.2753/REE1540-496X4703S203
Baldwin, R., & Giavazzi, F. (2015). The Eurozone Crisis. A Consensus View of the Causes and a Few Possible Solutions (1st ed.). London: CEPR Press.
Berg, A., Borensztein, E., & Pattillo, C. (2004). Assessing early warning systems: how have they worked in practice? IMF Working Paper, 52. Retrieved from https://www.imf.org/external/pubs/ft/wp/2004/wp0452.pdf http://dx.doi.org/10.5089/9781451847284.001
Berg, A., & Pattillo, C. (1999). Predicting currency crises: The indicators approach and an alternative. Journal of International Money and Finance, 18(4), 561-586. http://dx.doi.org/10.1016/S0261-5606(99)00024-8
Berge, T. J., & Jordà, O. (2011). Evaluating The Classification of Economic Activity into Recessions and Expansions. American Economic Journal. Macroeconomics, 3, 246-277. http://dx.doi.org/10.1257/mac.3.2.246
Borio, C., Drehmann, M., Gambacorta, L., Jimenez, G., & Trucharte, C. (2010). Countercyclical Capital Buffers: Exploring Options. BIS Working Paper, 317. Retrieved from https://www.bis.org/publ/work317.pdf
Camlica, F., & Gunes, D. (2016). Turkiye'de Finansal Stresin olculmesi : Yontemsel Bir Karsilastirma. CBT Research Notes in Economics, 1606.
Cardarelli, R., Elekdag, S., & Lall, S. (2009). Financial Stress, Downturns and Recoveries. IMF Working Paper Series, 100. Retrieved from https://www.imf.org/external/pubs/ft/wp/2009/wp09100.pdf
Caruana, J. (2010). Macroprudential Policy: Working towards a New Consensus. BIS’s Financial Stability Institute and the IMF Institute, Washington, DC, United States. High-Level Meeting on "The Emerging Framework for Financial Regulation and Monetary Policy". Retrieved from http://www.bis.org/speeches/sp100426.pdf
Casabianca, E. J., Catalano, M., Forni, L., Giarda, E., & Passeri, S. (2019). An Early Warning System for Banking Crises: From Regression-based Analysis to Machine Learning Techniques MF Department of Economics Working Paper, 235. Retrieved from https://economia.unipd.it/sites/decon.unipd.it/files/20190235.pdf
Chadwick, M. G., & Ozturk, H. (2018). Measuring Financial Systemic Stress for Turkey: A Search for the Best Composite Indicator. Economic Systems, 43(1), 151-172. http://dx.doi.org/10.1016/j.ecosys.2018.09.004
Chen, S., & Svirydzenka, K. (2021). Financial Cycles - Early Warning Indicators of Banking Crises? IMF Working Papers, 116. Retrieved from http://www.imf.org/external/pubs/cat/longres.aspx?sk=50257
Csortos, O., & Szalai, Z. (2014). Early warning indicators: Financial and macroeconomic imbalances in Central and Eastern European countries. MNB Working Paper, 2. Retrieved from http://www.mnb.hu/letoltes/wp-2014-2.pdf
Deloitte. (2016). Rand Depreciation. Retrieved from https://www2.deloitte.com/content/dam/Deloitte/za/Documents/process-and-operations/ZA_Rand_Depreciation_brochure.pdf
Dembiermont, C., Drehmann, M., & Muksakunratana, S. (2013). How Much Does the Private Sector Really Borrow? A New Database for Total Credit to the Private Non-Financial Sector. BIS Quarterly Review, March. Retrieved from https://www.bis.org/publ/qtrpdf/r_qt1303h.htm
Demirgüç-Kunt, A., & Detragiache, E. (2000). Monitoring banking sector fragility: A multivariate logit approach. The World Bank Economic Review, 14(2), 287-307. http://dx.doi.org/10.1093/wber/14.2.287
Detken, C., Weeken, O., Alessi, L., Bonfim, D., Bouchina, M., Castro, C., . . . Welz, P. (2014). Operationalising the countercyclical capital buffer: indicator selection, threshold identification and calibration options ESRB Occasional Paper Series, 5.
Drehmann, M., Borio, C., & Tsatsaronis, K. (2011). Anchoring Countercyclical Capital Buffers: The Role of Credit Aggregates. International Journal of Central Banking, 7(4), 189-240.
Drehmann, M., & Juselius, M. (2012). Do Debt Service Costs Affect Macroeconomic and Financial Stability? BIS Quarterly Review, September, 21-35.
Drehmann, M., & Juselius, M. (2014). Evaluating Early Warning Indicators of Banking Crises-Satisfying Policy Requirements. International Journal of Forecasting, 30(3), 759-780. http://dx.doi.org/10.1016/j.ijforecast.2013.10.002
Duprey, T., & Klaus, B. (2017). How to Predict Financial Stress? An Assessment of Markov Switching Models. ECB Working Paper, 2057. Retrieved from https://www.ecb.europa.eu/pub/pdf/scpwps/ecb.wp2057.en.pdf
Edison, H. J. (2003). Do indicators of financial crises work? An evaluation of an early warning system. International Journal of Finance & Economics, 8(1), 11-53. http://dx.doi.org/10.1002/ijfe.197
Ekinci, A. (2013). Financial Stress Index for Turkey. Dogus University Journal, 14(2), 213-229.
Engeline, N., & Matondang, P. S. (2016). Early Warning System and Currency Volatility Management in Emerging Market. Buletin of Monetary Economics and Banking, 19(2), 129-152. http://dx.doi.org/10.21098/bemp.v19i2.627
Fernández de Lis, S., & Garcia-Herrero, A. (2012). Dynamic provisioning: a buffer rather than a countercyclical tool? BBVA Economic Research Working Paper, 1222. Retrieved from https://www.bbvaresearch.com/wp-content/uploads/mult/WP_1222_tcm348-360026.pdf
Frankel, J., & Rose, A. (1996). Currency crashes in emerging markets: An empirical treatment. Journal of International Economics, 41(3-4), 351-366. http://dx.doi.org/10.1016/S0022-1996(96)01441-9
Frankel, J., & Saravelos, G. (2012). Can leading indicators assess country vulnerability? Evidence from the 2008–09 global financial crisis. Journal of International Economics, 87(2), 216-231. http://dx.doi.org/10.1016/j.jinteco.2011.12.009
Gersl, A., & Jasova, M. (2018). Credit-Based Early Warning Indicators of Banking Crises in Emerging Markets. Economic Systems, 42(1), 18-31. http://dx.doi.org/10.1016/j.ecosys.2017.05.004
Giese, J., Andersen, H., Bush, O., Castro, C., Farag, M., & Kapadia, S. (2014). The credit-to-gdp gap and complementary indicators for macroprudential policy: Evidence from the UK. International Journal of Finance & Economics, 19(1), 25-47. http://dx.doi.org/10.1002/ijfe.1489
Gourinchas, P. O., & Obstfeld, M. (2012). Stories of the Twentieth Century for the Twenty-First. American Economic Journal. Macroeconomics, 4(1), 226-265. http://dx.doi.org/10.1257/mac.4.1.226
Grimaldi, M. B. (2010). Detecting and Interpreting Financial Stress in the Euro Area. ECB Working Paper Series, 1214. Retrieved from http://www.ecb.europa.eu/pub/pdf/scpwps/ecbwp1214.pdf http://dx.doi.org/10.2139/ssrn.1622165
Hahm, J., Shin, H. S., & Shin, K. (2012). Non-Core Bank Liabilities and Financial Vulnerability. Journal of Money, Credit and Banking, 45(1), 3-36. http://dx.doi.org/10.1111/jmcb.12035
Hakkio, C. S., & Keeton, W. R. (2009). Financial Stress: What is it, How Can it Be Measured, and Why Does it Matter? Federal Reserve Bank of Kansas City Economic Review, 94(2), 5-50.
Harvard Business School. (2022). Global Crises Data by Country Retrieved from https://www.hbs.edu/behavioral-finance-and-financial-stability/data/Pages/global.aspx
IMF. (2010). Annual report 2010. Supporting a balanced global recovery. Retrieved from https://www.imf.org/-/media/Websites/IMF/imported-flagship-issues/external/pubs/ft/ar/2010/eng/pdf/_ar10engpdf.ashx
Ishikawa, A., Kamada, K., Kan, K., Kojima, R., Kurachi, Y., Nasu, K., & Teranishi, Y. (2012). The Financial Activity Index. BoJ Working Paper, 12. Retrieved from https://www.boj.or.jp/en/research/wps_rev/wps_2012/data/wp12e04.pdf
Jahn, N., & Kick, T. (2012). Early warning indicators for the German banking system: A macroprudential analysis. DB Discussion Papers, 27. Retrieved from https://www.econstor.eu/obitstream/10419/65859/1/729175642.pdf
Janes, H., Longton, G., & Pepe, M. (2009). Accommodating covariates in ROC analysis. The Stata Journal, 9(1), 17-39. http://dx.doi.org/10.1177/1536867X0900900102
Jordà, O., Schularick, M., & Taylor, A. M. (2015). Leveraged bubbles. Journal of Monetary Economics, 76(Supplement), S1-S20. http://dx.doi.org/10.1016/j.jmoneco.2015.08.005
Kaminsky, G. L., Lizondo, S., & Reinhart, C. M. (1998). Leading Indicators of Currency Crises. IMF Staff Papers, 45(1).
Kaminsky, G. L., & Reinhart, C. M. (1999). The twin crises: The causes of banking and balance-of-payments problems. The American Economic Review, 89(3), 473-500. http://dx.doi.org/10.1257/aer.89.3.473
Lo Duca, M., & Peltonen, T. (2013). Assessing systemic risks and predicting systemic events. Journal of Banking & Finance, 37, 2185-2188.
Martinez, J. F., & Oda, D. (2021). Characterization of the Chilean financial cycle, early warning indicators and implications for macro-prudential policies. Latin American Journal of Central Banking, 2(1), 100024. http://dx.doi.org/10.1016/j.latcb.2021.100024
Morales, M. A., & Estrada, D. (2010). A Financial Stability Index for Colombia. Annals of Finance, 6(4), 555-581. http://dx.doi.org/10.1007/s10436-010-0161-7
Nelson, R. M. (2018). Turkey’s Currency Crisis. Congressional Research Service. Retrieved from https://sgp.fas.org/crs/mideast/IF10957.pdf
Oet, M., Eiben, R., Bianco, T., Gramlich, D., & Ong, S. (2011). The Financial Stress Index: Identification of Systemic Risk Conditions. FRBC Working Paper, 11.
Ortiz Vidal-Abarca, A., & Ugarte-Ruiz, A. (2015). Introducing a New Early Warning System Indicator (EWSI) of banking crises. BBVA Working Papers, 1502. Retrieved from https://www.bbvaresearch.com/wp-content/uploads/2015/01/WP_EWS-SystemVersion-Sep2014_i.pdf
Park, C. Y., & Mercado, R. V. (2014). Determinants of Financial Stress in Emerging Market Economies. Journal of Banking & Finance, 45, 199-224. http://dx.doi.org/10.1016/j.jbankfin.2013.09.018
Pepe, M., Longton, G., & Janes, H. (2009). Estimation and comparison of receiver operating characteristic curves. The Stata Journal, 9(1), 1-16. http://dx.doi.org/10.1177/1536867X0900900101
Pietrzak, M. (2021). Can Financial Soundness Indicators Help Predict Financial Sector Distress? IMF Working Paper, 197. Retrieved from https://www.imf.org/en/Publications/WP/Issues/2021/07/23/Can-Financial-Soundness-Indicators-Help-Predict-Financial-Sector-Distress-462145 http://dx.doi.org/10.5089/9781513593005.001
Ponomarenko, A., & Tatarintsev, S. (2020). Incorporating Financial Development Indicators into Early Warning Systems. Bank of Russia Working Paper, 58. Retrieved from http://www.cbr.ru/content/document/file/111729/wp-58_e.pdf
Reinhart, C. M., & Rogoff, K. S. (2009). This Time is Different: Eight Centuries of Financial Folly (1st ed. ed.): Princeton University Press.
Romano, S. (2021). The 2011 Crisis in Italy: A Story of Deep-Rooted (and Still Unresolved) Economic and Political Weaknesses. In B. De Souza Guilherme, C. Ghymers, S. Griffith-Jones, & A. Ribeiro Hoffmann (Eds.), Financial Crisis Management and Democracy (pp. 173-184): Springer. http://dx.doi.org/10.1007/978-3-030-54895-7_10
Schularick, M., & Taylor, A. (2012). Credit Booms Gone Bust: Monetary Policy, Leverage Cycles, and Financial Crises, 1870-2008. The American Economic Review, 102(2), 1029-1061. http://dx.doi.org/10.1257/aer.102.2.1029
Străchinaru, A. I. (2022). Early Warning Systems for Banking Crisis and Sovereign Risk. Journal of Financial Studies & Research, 2022, 1-9. http://dx.doi.org/10.5171/2022.441237
Su, J. Q., & Liu, S. (1993). Linear Combinations of Multiple Diagnostic Markers. Journal of the American Statistical Association, 88(424), 1350-1355. http://dx.doi.org/10.1080/01621459.1993.10476417
Tarkocin, C., & Donduran, M. (2021). Constructing Early Warning Indicators for the Banks Using Machine Learning Models. SSRN Working Paper, 4157876. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4157876
Tölö, E., Laakkonen, H., & Kalatie, S. (2018). Evaluating indicators for use in setting the countercyclical capital buffer. International Journal of Central Banking, 14(2), 51-111.
Vartanian, P. R., & Garbe, H. (2019). The Brazilian Economic Crisis During the Period 2014-2016: Is There Precedence of Internal or External Factors? Journal of International and Global Economic Studies, 12(1), 66-86.
Viktorov, I., & Abramov, A. (2020). The 2014-15 Financial Crisis in Russia and the Foundations of Weak Monetary Power Autonomy in the International Political Economy. New Political Economy, 25(4), 487-510. http://dx.doi.org/10.1080/13563467.2019.1613349
Witte, D. M. (2012). What Effect Did the Credit Crisis of 2008 Have on European Exchange Rates? Eastern European Economics, 50(3), 79-93. http://dx.doi.org/10.2753/eee0012-8775500304
Yildirim, F. (2021). Banking Soundness Index for Turkey: The Principal Component Analysis Approach. Journal of Research in Economics. Journal of Research in Economics. Politics and Finance, 6(3), 845-861. http://dx.doi.org/10.30784/epfad.991688
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Yusuf Yıldırım, Anirban Sanyal
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All accepted papers are published on an Open Access basis.
The Open Access License is based on the Creative Commons license.
The non-commercial use of the article will be governed by the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License as currently displayed on https://creativecommons.org/licenses/by-nc-nd/4.0
Under the Creative Commons Attribution-NonCommercial-NoDerivatives license, the author(s) and users are free to share (copy, distribute and transmit the contribution) under the following conditions:
1. they must attribute the contribution in the manner specified by the author or licensor,
2. they may not use this contribution for commercial purposes,
3. they may not alter, transform, or build upon this work.
This work is licenced under a 
