Skip to content

Statistical Comparisons of Classifiers over Multiple Data Sets

Janez Demšar

2006 · 12,051 citations

No retraction on record in OpenAlex.Check Crossref now

Cite this paper

Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. 7(1), 1–30. https://doi.org/10.5555/1248547.1248548

In text: (Demšar, 2006)

Ask a question about this research

One question, free, no account. Anchorcite finds the papers, checks each against Crossref and links every sentence of the answer to the passage it comes from.

Abstract

While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.

  1. Scikit-learn: Machine Learning in Python2012
  2. LIBSVM2011
  3. SMOTE: Synthetic Minority Over-sampling Technique2002
  4. The WEKA data mining software2009
  5. The Elements of Statistical Learning: Data Mining, Inference, and Prediction2010

Metadata from OpenAlex (CC0). Citations are generated from the published record.