A utility-based performance metric for ROC analysis of N-class classification tasks

Darrin C. Edwards; Charles E. Metz

doi:10.1117/12.710083

20 March 2007 A utility-based performance metric for ROC analysis of N-class classification tasks

Darrin C. Edwards, Charles E. Metz

Proceedings Volume 6515, Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment; 651503 (2007) https://doi.org/10.1117/12.710083
Event: Medical Imaging, 2007, San Diego, CA, United States

Abstract

We have shown previously that an obvious generalization of the area under an ROC curve (AUC) cannot serve as a useful performance metric in classification tasks with more than two classes. We define a new performance metric, grounded in the concept of expected utility familiar from ideal observer decision theory, but which should not suffer from the issues of dimensionality and degeneracy inherent in the hypervolume under the ROC hypersurface in tasks with more than two classes. In the present work, we compare this performance metric with the traditional AUC metric in a variety of two-class tasks. Our numerical studies suggest that the behavior of the proposed performance metric is consistent with that of the AUC performance metric in a wide range of two-class classification tasks, while analytical investigation of three-class "near-guessing" observers supports our claim that the proposed performance metric is well-defined and positive in the limit as the observer's performance approaches that of the guessing observer.

Citation Download Citation

Darrin C. Edwards and Charles E. Metz "A utility-based performance metric for ROC analysis of N-class classification tasks", Proc. SPIE 6515, Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment, 651503 (20 March 2007); https://doi.org/10.1117/12.710083

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available