TY - STD TI - Cabitza F, Zeitoun J-D. The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence. Ann Transl Med. 2019; 7(8). http://atm.amegroups.com/article/view/25300. UR - http://atm.amegroups.com/article/view/25300 ID - ref1 ER - TY - JOUR AU - Siau, K. AU - Wang, W. PY - 2018 DA - 2018// TI - Building trust in artificial intelligence, machine learning, and robotics JO - Cutter Bus Technol J VL - 31 ID - Siau2018 ER - TY - JOUR AU - Valiant, L. G. PY - 1984 DA - 1984// TI - A theory of the learnable JO - Commun. ACM VL - 27 ID - Valiant1984 ER - TY - BOOK AU - Mitchell, T. M. PY - 1997 DA - 1997// TI - Machine learning PB - McGraw-Hill Education CY - The address ID - Mitchell1997 ER - TY - JOUR AU - Cabitza, F. AU - Campagner, A. AU - Albano, D. AU - Aliprandi, A. AU - Bruno, A. AU - Chianca, V. AU - Corazza, A. AU - Di Pietto, F. AU - Gambino, A. AU - Gitto, S. PY - 2020 DA - 2020// TI - The elephant in the machine: Proposing a new metric of data reliability and its application to a medical case to assess classification reliability JO - Appl Sci VL - 10 ID - Cabitza2020 ER - TY - JOUR AU - Schoorman, D. AU - Mayer, R. AU - Davis, J. PY - 2007 DA - 2007// TI - An integrative model of organizational trust: Past, present, and future JO - Acad Manag Rev VL - 32 ID - Schoorman2007 ER - TY - STD TI - Campagner A, Cabitza F, Ciucci D. Exploring medical data classification with three-way decision tree. In: Proceedings of the 12th BIOSTEC International Joint Conference, vol. 5: 2019. p. 147–58. ID - ref7 ER - TY - STD TI - Holzinger A. From machine learning to explainable ai. In: 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA). IEEE: 2018. p. 55–66. ID - ref8 ER - TY - JOUR AU - De Bièvre, P. PY - 2012 DA - 2012// TI - The 2012 international vocabulary of metrology:vim JO - Accred Qual Assur VL - 17 ID - De Bièvre2012 ER - TY - JOUR AU - Saal, F. E. AU - Downey, R. G. AU - Lahey, M. A. PY - 1980 DA - 1980// TI - Rating the ratings: Assessing the psychometric quality of rating data JO - Psychol Bull VL - 88 ID - Saal1980 ER - TY - JOUR AU - Cabitza, F. AU - Locoro, A. AU - Alderighi, C. AU - Rasoini, R. AU - Compagnone, D. AU - Berjano, P. PY - 2019 DA - 2019// TI - The elephant in the record: on the multiplicity of data recording work JO - Health Inform J VL - 25 ID - Cabitza2019 ER - TY - JOUR AU - Hayes, A. F. AU - Krippendorff, K. PY - 2007 DA - 2007// TI - Answering the call for a standard reliability measure for coding data JO - Commun Methods Measures VL - 1 ID - Hayes2007 ER - TY - JOUR AU - Quarfoot, D. AU - Levine, R. A. PY - 2016 DA - 2016// TI - How robust are multirater interrater reliability indices to changes in frequency distribution? JO - Am Stat VL - 70 ID - Quarfoot2016 ER - TY - JOUR AU - Dempster, A. PY - 1967 DA - 1967// TI - Upper and lower probabilities induced by a multivalued mapping JO - Ann Math Stat VL - 38 ID - Dempster1967 ER - TY - BOOK AU - Shafer, G. PY - 1976 DA - 1976// TI - A Mathematical Theory of Evidence vol. 42 PB - Princeton university press CY - Princeton, New Jersey ID - Shafer1976 ER - TY - JOUR AU - Haenni, R. AU - Hartmann, S. PY - 2006 DA - 2006// TI - Modeling partially reliable information sources: a general approach based on dempster–shafer theory JO - Inf Fusion VL - 7 ID - Haenni2006 ER - TY - JOUR AU - Schubert, J. PY - 2011 DA - 2011// TI - Conflict management in dempster–shafer theory using the degree of falsity JO - Int J Approx Reason VL - 52 ID - Schubert2011 ER - TY - JOUR AU - Scotney, B. AU - McClean, S. PY - 2003 DA - 2003// TI - Database aggregation of imprecise and uncertain evidence JO - Inf Sci VL - 155 ID - Scotney2003 ER - TY - JOUR AU - Xiao, F. AU - Qin, B. PY - 2018 DA - 2018// TI - A weighted combination method for conflicting evidence in multi-sensor data fusion JO - Sensors VL - 18 ID - Xiao2018 ER - TY - BOOK AU - Gwet, K. L. PY - 2014 DA - 2014// TI - Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters PB - Advanced Analytics, LLC CY - Gaithersburg, MD ID - Gwet2014 ER - TY - BOOK AU - Sentz, K. AU - Ferson, S. PY - 2002 DA - 2002// TI - Combination of evidence in dempster-shafer theory. Technical report PB - Sandia National Laboratories CY - Albuquerque ID - Sentz2002 ER - TY - BOOK AU - Rasch, G. PY - 1980 DA - 1980// TI - Probabilistic models for some intelligence and attainment tests, 1960 PB - Danish Institute for Educational Research CY - Copenhagen ID - Rasch1980 ER - TY - CHAP AU - Heinecke, S. AU - Reyzin, L. PY - 2019 DA - 2019// TI - Crowdsourced pac learning under classification BT - Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol 7 PB - AAAI Press CY - Austin ID - Heinecke2019 ER - TY - JOUR AU - Bennell, K. AU - Talbot, R. AU - Wajswelner, H. AU - Techovanich, W. AU - Kelly, D. AU - Hall, A. J. PY - 1998 DA - 1998// TI - Intra-rater and inter-rater reliability of a weight-bearing lunge measure of ankle dorsiflexion JO - Aust J Physiother VL - 44 ID - Bennell1998 ER - TY - JOUR AU - Gianinazzi, M. E. PY - 2015 DA - 2015// TI - Intra-rater and inter-rater reliability of a medical record abstraction study on transition of care after childhood cancer JO - PloS ONE VL - 10 ID - Gianinazzi2015 ER - TY - CHAP AU - Cabitza, F. AU - Campagner, A. AU - Ciucci, D. PY - 2019 DA - 2019// TI - New frontiers in explainable ai: Understanding the gi to interpret the go BT - International Cross-Domain Conference for Machine Learning and Knowledge Extraction PB - Springer CY - Cham ID - Cabitza2019 ER - TY - CHAP AU - Jin, R. AU - Ghahramani, Z. PY - 2003 DA - 2003// TI - Learning with multiple labels BT - Advances in Neural Information Processing Systems PB - MIT Press CY - Cambridge ID - Jin2003 ER - TY - STD TI - Sriperumbudur BK, Fukumizu K, Gretton A, Schölkopf B, Lanckriet GR. On integral probability metrics, ϕ-divergences and binary classification. arXiv preprint arXiv:0901.2698. 2009. ID - ref28 ER - TY - BOOK AU - Corder, G. W. AU - Foreman, D. I. PY - 2014 DA - 2014// TI - Nonparametric statistics: a step-by-step approach PB - Sons CY - Hoboken, New Jersey ID - Corder2014 ER - TY - STD TI - Gretton A, Borgwardt K, Rasch M, Schölkopf B, Smola AJ. A kernel method for the two-sample-problem. In: Advances in Neural Information Processing Systems. Curran Associates, Inc.: 2007. p. 513–20. ID - ref30 ER - TY - JOUR AU - Van Erven, T. AU - Harremos, P. PY - 2014 DA - 2014// TI - Rényi divergence and kullback-leibler divergence JO - IEEE Trans Inf Theory VL - 60 ID - Van Erven2014 ER - TY - JOUR AU - Endres, D. M. AU - Schindelin, J. E. PY - 2003 DA - 2003// TI - A new metric for probability distributions JO - IEEE Trans Inf Theory VL - 49 ID - Endres2003 ER - TY - STD TI - Pérez-Cruz F. Estimation of information theoretic measures for continuous random variables. In: Advances in Neural Information Processing Systems. Curran Associates, Inc.: 2009. p. 1257–64. ID - ref33 ER - TY - JOUR AU - Nguyen, X. AU - Wainwright, M. J. AU - Jordan, M. I. PY - 2010 DA - 2010// TI - Estimating divergence functionals and the likelihood ratio by convex risk minimization JO - IEEE Trans Inf Theory VL - 56 ID - Nguyen2010 ER - TY - BOOK AU - Pardo, L. PY - 2005 DA - 2005// TI - Statistical inference based on divergence measures PB - CRC press CY - Boca Raton, FL ID - Pardo2005 ER - TY - BOOK AU - McDonald, J. H. PY - 2009 DA - 2009// TI - Handbook of Biological Statistics vol. 2 PB - sparky house publishing Baltimore, MD CY - Baltimore ID - McDonald2009 ER - TY - BOOK AU - Grabisch, M. AU - Marichal, J. -. L. AU - Mesiar, R. AU - Pap, E. PY - 2009 DA - 2009// TI - Aggregation Functions vol. 127 PB - Cambridge University Press CY - Cambridge, United Kingdom ID - Grabisch2009 ER - TY - JOUR AU - Justel, A. AU - Peña, D. AU - Zamar, R. PY - 1997 DA - 1997// TI - A multivariate kolmogorov-smirnov test of goodness of fit JO - Stat Probab Lett VL - 35 ID - Justel1997 ER - TY - JOUR AU - Rosenbaum, P. R. PY - 2005 DA - 2005// TI - An exact distribution-free test comparing two multivariate distributions based on adjacency JO - J R Stat Soc Ser B Stat Methodol VL - 67 ID - Rosenbaum2005 ER - TY - CHAP AU - Ramdas, A. AU - Reddi, S. J. AU - Póczos, B. AU - Singh, A. AU - Wasserman, L. PY - 2015 DA - 2015// TI - On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions BT - Twenty-Ninth AAAI Conference on Artificial Intelligence PB - AAAI Press CY - Austin ID - Ramdas2015 ER - TY - STD TI - Ahuja K. Estimating kullback-leibler divergence using kernel machines. In: 2019 53rd Asilomar Conference on Signals, Systems, and Computers. IEEE: 2019. p. 690–6. ID - ref41 ER - TY - STD TI - Boltz S, Debreuve E, Barlaud M. knn-based high-dimensional kullback-leibler distance for tracking. In: Eighth International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’07). IEEE: 2007. p. 16–16. ID - ref42 ER - TY - JOUR AU - Bien, N. AU - Rajpurkar, P. AU - Ball, R. L. AU - Irvin, J. AU - Park, A. AU - Jones, E. AU - Bereket, M. AU - Patel, B. N. AU - Yeom, K. W. AU - Shpanskaya, K. AU - Halabi, S. AU - Zucker, E. AU - Fanton, G. AU - Amanatullah, D. F. AU - Beaulieu, C. F. AU - Riley, G. M. AU - Stewart, R. J. AU - Blankenberg, F. G. AU - Larson, D. B. AU - Jones, R. H. AU - Langlotz, C. P. AU - Ng, A. Y. AU - Lungren, M. P. PY - 2018 DA - 2018// TI - Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of mrnet JO - PLOS Medicine VL - 15 ID - Bien2018 ER - TY - STD TI - Paxton C, Niculescu-Mizil A, Saria S. Developing predictive models using electronic medical records: challenges and pitfalls. In: AMIA Annual Symposium Proceedings, vol. 2013. AMIA: 2013. p. 1109. ID - ref44 ER - TY - JOUR AU - Kiani, A. AU - Uyumazturk, B. AU - Rajpurkar, P. AU - Wang, A. AU - Gao, R. AU - Jones, E. AU - Yu, Y. AU - Langlotz, C. P. AU - Ball, R. L. AU - Montine, T. J. PY - 2020 DA - 2020// TI - Impact of a deep learning assistant on the histopathologic classification of liver cancer JO - NPJ Digit Med VL - 3 ID - Kiani2020 ER - TY - STD TI - Cabitza F, Campagner A, Balsano C. Bridging the last mile gap between ai implementation and operation: data awareness that matters. Ann Transl Med. 2020; 8(7). http://atm.amegroups.com/article/view/39228. UR - http://atm.amegroups.com/article/view/39228 ID - ref46 ER - TY - JOUR AU - Landis, J. R. AU - Koch, G. G. PY - 1977 DA - 1977// TI - The measurement of observer agreement for categorical data JO - Biometrics VL - 33 ID - Landis1977 ER - TY - BOOK AU - Krippendorff, K. PY - 2018 DA - 2018// TI - Content analysis: an introduction to its methodology PB - Sage publications CY - Sage UK: London, England ID - Krippendorff2018 ER - TY - CHAP AU - Cabitza, F. AU - Ciucci, D. AU - Rasoini, R. PY - 2019 DA - 2019// TI - A giant with feet of clay: on the validity of the data that feed machine learning in medicine BT - Organizing for the Digital World PB - Springer CY - Cham ID - Cabitza2019 ER - TY - STD TI - Jiang H, Nachum O. Identifying and correcting label bias in machine learning. arXiv preprint arXiv:1901.04966. 2019. ID - ref50 ER - TY - JOUR AU - Stand, J. PY - 2000 DA - 2000// TI - The hawthorne effect what did the original hawthorne studies actually show JO - Scand J Work Environ Health VL - 26 ID - Stand2000 ER - TY - JOUR AU - Gur, D. AU - Bandos, A. I. AU - Cohen, C. S. AU - Hakim, C. M. AU - Hardesty, L. A. AU - Ganott, M. A. AU - Perrin, R. L. AU - Poller, W. R. AU - Shah, R. AU - Sumkin, J. H. PY - 2008 DA - 2008// TI - The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations JO - Radiology VL - 249 ID - Gur2008 ER - TY - JOUR AU - Graber, M. L. PY - 2013 DA - 2013// TI - The incidence of diagnostic error in medicine JO - BMJ Qual Saf VL - 22 ID - Graber2013 ER - TY - STD TI - Oakden-Rayner L, Dunnmon J, Carneiro G, Ré C. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. arXiv preprint arXiv:1909.12475. 2019. ID - ref54 ER - TY - STD TI - Campagner A, Ciucci D, Svensson C-M, Figge MT, Cabitza F. Ground truthing from multi-rater labelling with three-way decisions and possibility theory. In: Cambiare rivista in Information Sciences. Elsevier: 2019. ID - ref55 ER - TY - STD TI - Svensson C-M, Figge MT, Hübler R. Automated classification of circulating tumor cells and the impact of interobsever variability on classifier training and performance. J Immunol Res. 2015; 2015. https://pubmed-ncbi-nlm-nih-gov.proxy.unimib.it/26504857/. UR - https://pubmed-ncbi-nlm-nih-gov.proxy.unimib.it/26504857/ ID - ref56 ER - TY - STD TI - Chatterjee S. Learning and memorization. In: International Conference on Machine Learning. PMLR: 2018. p. 755–63. ID - ref57 ER - TY - CHAP AU - Feldman, V. PY - 2020 DA - 2020// TI - Does learning require memorization? a short tale about a long tail BT - Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing PB - Association for Computing Machinery CY - New York ID - Feldman2020 ER - TY - JOUR AU - Heinrichs, B. AU - Eickhoff, S. B. PY - 2019 DA - 2019// TI - Your evidence? machine learning algorithms for medical diagnosis and prediction JO - Hum Brain Mapp VL - 41 ID - Heinrichs2019 ER - TY - JOUR AU - Kelp, C. PY - 2015 DA - 2015// TI - Understanding phenomena JO - Synthese VL - 192 ID - Kelp2015 ER - TY - STD TI - Lipton ZC, Steinhardt J. Troubling trends in machine learning scholarship. arXiv preprint arXiv:1807.03341. 2018. ID - ref61 ER - TY - STD TI - Sculley D, Snoek J, Wiltschko A, Rahimi A. Winner’s curse? on pace, progress, and empirical rigor. OpenReview. 2018. ID - ref62 ER - TY - CHAP AU - Cabitza, F. ED - Pelillo, M. ED - Scantamburlo, T. PY - 2020 DA - 2020// TI - Cobra ai: exploring some unintended consequences of artificial intelligence BT - Machines We Trust - Getting Along with Artificial Intelligence PB - MIT Press CY - Boston, MA, USA ID - Cabitza2020 ER - TY - CHAP AU - Dubois, D. AU - Prade, H. PY - 1992 DA - 1992// TI - On the combination of evidence in various mathematical frameworks BT - Reliability Data Collection and Analysis PB - Springer CY - Dordrecht ID - Dubois1992 ER - TY - BOOK AU - Ferson, S. AU - Kreinovich, V. PY - 2002 DA - 2002// TI - Representation, propagation, and aggregation of uncertainty. Technical report PB - Sandia National Laboratories CY - Albuquerque ID - Ferson2002 ER - TY - JOUR AU - Valiant, L. G. PY - 1984 DA - 1984// TI - A theory of the learnable JO - Commun. ACM VL - 27 ID - Valiant1984 ER - TY - JOUR AU - Angluin, D. AU - Laird, P. PY - 1988 DA - 1988// TI - Learning from noisy examples JO - Mach Learn VL - 2 ID - Angluin1988 ER - TY - JOUR AU - N. Vapnik, V. AU - Ya. Chervonenkis, A. PY - 1971 DA - 1971// TI - On the uniform convergence of relative frequencies of events to their probabilities JO - Theor Probab Applicactions VL - 17 ID - N. Vapnik1971 ER - TY - BOOK AU - Koller, D. AU - Friedman, N. PY - 2009 DA - 2009// TI - Probabilistic graphical models: principles and techniques PB - MIT press CY - Cambridge ID - Koller2009 ER - TY - STD TI - Ramshaw L, Tarjan RE. On minimum-cost assignments in unbalanced bipartite graphs. Labs, HP, Palo Alto, CA, USA, Tech. Rep. HPL-2012-40R1. 2012. ID - ref70 ER - TY - STD TI - Mastin A, Jaillet P. Greedy online bipartite matching on random graphs. arXiv preprint arXiv:1307.2536. 2013. ID - ref71 ER -