learning-ai.cafe learning-ai.cafe / Metrics / 17

17 · Metrics

References

The receiver operating characteristic curve accurately assesses imbalanced datasets — Richardson, Trevizani, Greenbaum, Carter, Nielsen & Peters. Patterns 5, 2024.

@article{richardson2024receiver,
  title   = {The receiver operating characteristic curve accurately assesses imbalanced datasets},
  author  = {Richardson, Eve and Trevizani, Raphael and Greenbaum, Jason A. and Carter, Hannah and Nielsen, Morten and Peters, Bjoern},
  journal = {Patterns},
  volume  = {5},
  pages   = {100994},
  year    = {2024},
  doi     = {10.1016/j.patter.2024.100994}
}

A Closer Look at AUROC and AUPRC under Class Imbalance — McDermott, Zhang, Hansen, Angelotti & Gallifant. NeurIPS 2024.

@inproceedings{mcdermott2024closer,
  title     = {A Closer Look at AUROC and AUPRC under Class Imbalance},
  author    = {McDermott, Matthew B. A. and Zhang, Haoran and Hansen, Lasse Hyldig and Angelotti, Giovanni and Gallifant, Jack},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  volume    = {37},
  year      = {2024},
  eprint    = {2401.06091},
  archivePrefix = {arXiv}
}

Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs — Flores, Smith, Fukuyama & Wilson. NeurIPS 2025.

@inproceedings{flores2025aligning,
  title     = {Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs},
  author    = {Flores, Gerardo A. and Smith, Alyssa H. and Fukuyama, Julia A. and Wilson, Ashia C.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  volume    = {38},
  year      = {2025},
  eprint    = {2506.14540},
  archivePrefix = {arXiv}
}

The Relationship Between Precision-Recall and ROC Curves — Davis & Goadrich. ICML 2006.

@inproceedings{davis2006relationship,
  title     = {The Relationship Between Precision-Recall and ROC Curves},
  author    = {Davis, Jesse and Goadrich, Mark},
  booktitle = {Proceedings of the 23rd International Conference on Machine Learning (ICML)},
  pages     = {233--240},
  year      = {2006},
  doi       = {10.1145/1143844.1143874}
}

An introduction to ROC analysis — Tom Fawcett. Pattern Recognition Letters 27(8), 2006.

@article{fawcett2006introduction,
  title   = {An introduction to ROC analysis},
  author  = {Fawcett, Tom},
  journal = {Pattern Recognition Letters},
  volume  = {27},
  number  = {8},
  pages   = {861--874},
  year    = {2006},
  doi     = {10.1016/j.patrec.2005.10.010}
}

Analyzing a Portion of the ROC Curve — Donna K. McClish. Medical Decision Making 9(3), 1989.

@article{mcclish1989analyzing,
  title   = {Analyzing a Portion of the ROC Curve},
  author  = {McClish, Donna K.},
  journal = {Medical Decision Making},
  volume  = {9},
  number  = {3},
  pages   = {190--195},
  year    = {1989},
  doi     = {10.1177/0272989X8900900307}
}

learning-ai.cafe

Metrics

Leaving out parts of the truth: an analysis of metric estimation, theoretical assumptions, and points of structural failure in ML/DL.

  1. 01 What a Metric Actually Is
  2. 02 TPR and FPR
  3. 03 Accuracy
  4. 04 Balanced Accuracy
  5. 05 Precision
  6. 06 F1
  7. 07 ROC and AUROC
  8. 08 Partial AUROC
  9. 09 PR Curves and AUPRC
  10. 10 Variance at Low Prevalence
  11. 11 Calibration and ECE
  12. 12 Brier Score
  13. 13 Log Score
  14. 14 Net Benefit
  15. 15 Bounded DCA Log Score
  16. 16 What to Report
  17. 17 References

Who built this page

Every merged contribution is typed and scored at review. Chips next to a heading show who touched that section. Scores feed the contributor share of project support.

History

    Propose a change

    Settings

    Language
    Appearance
    Accent
    Font
    Text size
    Column width
    Book