Why scoring functions cannot assess tail properties

Brehmer, Jonas R. and Strokorb, Kirstin ORCID: https://orcid.org/0000-0001-8748-3014 2019. Why scoring functions cannot assess tail properties. Electronic Journal of Statistics 13 (2) , pp. 4015-4034. 10.1214/19-EJS1622

Preview

PDF - Accepted Post-Print Version Download (264kB) | Preview

Abstract

Motivated by the growing interest in sound forecast evaluation techniques with an emphasis on distribution tails rather than average behaviour, we investigate a fundamental question arising in this context: Can statistical features of distribution tails be elicitable, i.e. be the unique minimizer of an expected score? We demonstrate that expected scores are not suitable to distinguish genuine tail properties in a very strong sense. Specifically, we introduce the class of max-functionals, which contains key characteristics from extreme value theory, for instance the extreme value index. We show that its members fail to be elicitable and that their elicitation complexity is in fact infinite under mild regularity assumptions. Further we prove that, even if the information of a max-functional is reported via the entire distribution function, a proper scoring rule cannot separate max-functional values. These findings highlight the caution needed in forecast evaluation and statistical inference if relevant information is encoded by such functionals.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Mathematics
Publisher:	Institute of Mathematical Statistics (IMS): OAJ / Institute of Mathematical Statistics
ISSN:	1935-7524
Funders:	German Research Foundation via RTG 1953 (PhD funding for Jonas R. Brehmer)
Date of First Compliant Deposit:	5 October 2019
Date of Acceptance:	30 September 2019
Last Modified:	10 Nov 2023 20:29
URI:	https://orca.cardiff.ac.uk/id/eprint/125886

Citation Data

Cited 9 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item