Package: RJafroc 2.1.3

RJafroc: Artificial Intelligence Systems and Observer Performance

Analyzing the performance of artificial intelligence (AI) systems/algorithms characterized by a 'search-and-report' strategy. Historically observer performance has dealt with measuring radiologists' performances in search tasks, e.g., searching for lesions in medical images and reporting them, but the implicit location information has been ignored. The implemented methods apply to analyzing the absolute and relative performances of AI systems, comparing AI performance to a group of human readers or optimizing the reporting threshold of an AI system. In addition to performing historical receiver operating receiver operating characteristic (ROC) analysis (localization information ignored), the software also performs free-response receiver operating characteristic (FROC) analysis, where lesion localization information is used. A book using the software has been published: Chakraborty DP: Observer Performance Methods for Diagnostic Imaging - Foundations, Modeling, and Applications with R-Based Examples, Taylor-Francis LLC; 2017: <https://www.routledge.com/Observer-Performance-Methods-for-Diagnostic-Imaging-Foundations-Modeling/Chakraborty/p/book/9781482214840>. Online updates to this book, which use the software, are at <https://dpc10ster.github.io/RJafrocQuickStart/>, <https://dpc10ster.github.io/RJafrocRocBook/> and at <https://dpc10ster.github.io/RJafrocFrocBook/>. Supported data collection paradigms are the ROC, FROC and the location ROC (LROC). ROC data consists of single ratings per images, where a rating is the perceived confidence level that the image is that of a diseased patient. An ROC curve is a plot of true positive fraction vs. false positive fraction. FROC data consists of a variable number (zero or more) of mark-rating pairs per image, where a mark is the location of a reported suspicious region and the rating is the confidence level that it is a real lesion. LROC data consists of a rating and a location of the most suspicious region, for every image. Four models of observer performance, and curve-fitting software, are implemented: the binormal model (BM), the contaminated binormal model (CBM), the correlated contaminated binormal model (CORCBM), and the radiological search model (RSM). Unlike the binormal model, CBM, CORCBM and RSM predict 'proper' ROC curves that do not inappropriately cross the chance diagonal. Additionally, RSM parameters are related to search performance (not measured in conventional ROC analysis) and classification performance. Search performance refers to finding lesions, i.e., true positives, while simultaneously not finding false positive locations. Classification performance measures the ability to distinguish between true and false positive locations. Knowing these separate performances allows principled optimization of reader or AI system performance. This package supersedes Windows JAFROC (jackknife alternative FROC) software V4.2.1, <https://github.com/dpc10ster/WindowsJafroc>. Package functions are organized as follows. Data file related function names are preceded by 'Df', curve fitting functions by 'Fit', included data sets by 'dataset', plotting functions by 'Plot', significance testing functions by 'St', sample size related functions by 'Ss', data simulation functions by 'Simulate' and utility functions by 'Util'. Implemented are figures of merit (FOMs) for quantifying performance and functions for visualizing empirical or fitted operating characteristics: e.g., ROC, FROC, alternative FROC (AFROC) and weighted AFROC (wAFROC) curves. For fully crossed study designs significance testing of reader-averaged FOM differences between modalities is implemented via either Dorfman-Berbaum-Metz or the Obuchowski-Rockette methods. Also implemented is single modality analysis, which allows comparison of performance of a group of radiologists to a specified value, or comparison of AI to a group of radiologists interpreting the same cases. Crossed-modality analysis is implemented wherein there are two crossed modality factors and the aim is to determined performance in each modality factor averaged over all levels of the second factor. Sample size estimation tools are provided for ROC and FROC studies; these use estimates of the relevant variances from a pilot study to predict required numbers of readers and cases in a pivotal study to achieve the desired power. Utility and data file manipulation functions allow data to be read in any of the currently used input formats, including Excel, and the results of the analysis can be viewed in text or Excel output files. The methods are illustrated with several included datasets from the author's collaborations. This update includes improvements to the code, some as a result of user-reported bugs and new feature requests, and others discovered during ongoing testing and code simplification.

Authors:Dev Chakraborty [cre, aut, cph], Peter Phillips [ctb], Xuetong Zhai [aut], Lucy D'Agostino McGowan [ctb], Alejandro RodriguezRuiz [ctb]

RJafroc_2.1.3.tar.gz
RJafroc_2.1.3.zip(r-4.5)RJafroc_2.1.3.zip(r-4.4)RJafroc_2.1.3.zip(r-4.3)
RJafroc_2.1.3.tgz(r-4.4-x86_64)RJafroc_2.1.3.tgz(r-4.4-arm64)RJafroc_2.1.3.tgz(r-4.3-x86_64)RJafroc_2.1.3.tgz(r-4.3-arm64)
RJafroc_2.1.3.tar.gz(r-4.5-noble)RJafroc_2.1.3.tar.gz(r-4.4-noble)
RJafroc_2.1.3.tgz(r-4.4-emscripten)RJafroc_2.1.3.tgz(r-4.3-emscripten)
RJafroc.pdf |RJafroc.html
RJafroc/json (API)
NEWS

# Install 'RJafroc' in R:
install.packages('RJafroc', repos = c('https://dpc10ster.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/dpc10ster/rjafroc/issues

Uses libs:
  • c++– GNU Standard C++ Library v3
Datasets:

On CRAN:

ai-optimizationartificial-intelligence-algorithmscomputer-aided-diagnosisfroc-analysisroc-analysistarget-classificationtarget-localization

5.86 score 19 stars 64 scripts 305 downloads 65 exports 50 dependencies

Last updated 20 days agofrom:edb840e5f0. Checks:OK: 4 NOTE: 5. Indexed: yes.

TargetResultDate
Doc / VignettesOKOct 29 2024
R-4.5-win-x86_64NOTEOct 29 2024
R-4.5-linux-x86_64NOTEOct 29 2024
R-4.4-win-x86_64NOTEOct 29 2024
R-4.4-mac-x86_64NOTEOct 29 2024
R-4.4-mac-aarch64NOTEOct 29 2024
R-4.3-win-x86_64OKOct 29 2024
R-4.3-mac-x86_64OKOct 29 2024
R-4.3-mac-aarch64OKOct 29 2024

Exports:ChisqrGoodnessOfFitDf2RJafrocDatasetDfBinDatasetDfCreateCorCbmDatasetDfExtractCorCbmDatasetDfExtractDatasetDfFroc2LrocDfFroc2RocDfLroc2FrocDfLroc2RocDfReadDataFileDfReadSPDfReadXModalitiesDfSaveDataFileDfWriteExcelDataFileFitBinormalRocFitCbmRocFitCorCbmFitRsmRocisBinnedDatasetisValidDatasetPlotBinormalFitPlotCbmFitPlotEmpiricalOperatingCharacteristicsPlotRsmOperatingCharacteristicsRSM_FPFRSM_LLFRSM_NLFRSM_pdfDRSM_pdfNRSM_TPFRSM_wLLFSimulateCorCbmDatasetSimulateFrocDatasetSimulateFrocFromLrocDatasetSimulateLrocDatasetSimulateRocDatasetSsFrocNhRsmModelSsFrocSampleSizeSsPowerGivenJKSsPowerGivenJKDbmVarComSsPowerGivenJKOrVarComSsPowerTableSsSampleSizeKGivenJStStCadVsRadStOldCodeStSPtestCppUtil2IntrinsicUtil2PhysicalUtilAnalyticalAucsRSMUtilAucBINUtilAucCBMUtilAucPROPROCUtilDBM2ORVarComUtilDBMVarCompUtilFigureOfMeritUtilLesDistrUtilLesWghtsDSUtilLesWghtsLDUtilMeanSquaresUtilOR2DBMVarComUtilORVarCompUtilPseudoValues

Dependencies:bbmlebdsmatrixbinomcellrangerclicolorspacecpp11crayondplyrfansifarvergenericsggplot2gluegtablegtoolshmsisobandlabelinglatticelifecyclemagrittrMASSMatrixmgcvmunsellmvtnormnlmenumDerivopenxlsxpillarpkgconfigprettyunitsprogressR6RColorBrewerRcppreadxlrematchrlangscalesstringistringrtibbletidyselectutf8vctrsviridisLitewithrzip

Readme and manuals

Help Manual

Help pageTopics
Artificial Intelligence Systems and Observer PerformanceRJafroc-package
Compute the chisquare goodness of fit statistic for ROC fitting modelChisqrGoodnessOfFit
TONY FROC datasetdataset01
Van Dyke ROC datasetdataset02
Franken ROC datasetdataset03
Federica Zanca FROC datasetdataset04
John Thompson FROC datasetdataset05
Magnus FROC datasetdataset06
Lucy Warren FROC datasetdataset07
Monica Penedo ROC datasetdataset08
Nico Karssemeijer ROC dataset (CAD vs. radiologists)dataset09
Mark Ruschin ROC datasetdataset10
Dobbins 1 FROC datasetdataset11
Dobbins 2 ROC datasetdataset12
Dobbins 3 FROC datasetdataset13
Federica Zanca real (as opposed to inferred) ROC datasetdataset14
Binned dataset suitable for checking 'FitCorCbm'; seed = 123datasetBinned123
Binned dataset suitable for checking 'FitCorCbm'; seed = 124datasetBinned124
Binned dataset suitable for checking 'FitCorCbm'; seed = 125datasetBinned125
Nico Karssemeijer LROC dataset (CAD vs. radiologists)datasetCadLroc
Simulated FROC CAD vs. RAD datasetdatasetCadSimuFroc
Simulated degenerate ROC dataset (for testing purposes)datasetDegenerate
#' Simulated FROC SPLIT-PLOT-C datasetdatasetFROCSpC
Simulated ROI datasetdatasetROI
John Thompson crossed modality FROC datasetdatasetX
Convert ratings arrays to an RJafroc datasetDf2RJafrocDataset
Returns a binned datasetDfBinDataset
Create paired dataset for testing 'FitCorCbm'DfCreateCorCbmDataset
Extract two arms of a pairing from an MRMC ROC datasetDfExtractCorCbmDataset
Extract a subset of treatments and readers from a datasetDfExtractDataset
Simulates an "AUC-equivalent" LROC dataset from an FROC datasetDfFroc2Lroc
Convert an FROC dataset to an ROC datasetDfFroc2Roc
Simulates an "AUC-equivalent" FROC dataset from an LROC datasetDfLroc2Froc
Convert an LROC dataset to a ROC datasetDfLroc2Roc
Read a factorial data file (not SPLIT-PLOT)DfReadDataFile
Read a SPLIT PLOT data file (not factorial)DfReadSP
Read a crossed-modality data fileDfReadXModalities
Save ROC dataset in different formatsDfSaveDataFile
Save dataset object as a JAFROC format Excel fileDfWriteExcelDataFile
Fit the binormal model to selected modality and reader in an ROC datasetFitBinormalRoc
Fit the contaminated binormal model (CBM) to selected modality and reader in an ROC datasetFitCbmRoc
Fit CORCBM to a paired ROC datasetFitCorCbm
Fit the radiological search model (RSM) to an ROC datasetFitRsmRoc
Determine if a dataset is binnedisBinnedDataset
Check the validity of a dataset for FOM and other input parametersisValidDataset
Plot binormal fitPlotBinormalFit
Plot CBM fitted curvePlotCbmFit
Plot empirical operating characteristics, ROC, FROC or LROCPlotEmpiricalOperatingCharacteristics
RSM predicted operating characteristics, ROC pdfs and AUCsPlotRsmOperatingCharacteristics
RSM predicted ROC-abscissa as function of zRSM_FPF
RSM predicted FROC ordinateRSM_LLF
RSM predicted FROC abscissaRSM_NLF
RSM predicted ROC-rating pdf for diseased casesRSM_pdfD
RSM predicted ROC-rating pdf for non-diseased casesRSM_pdfN
RSM predicted ROC-ordinate as function of zRSM_TPF
RSM predicted wAFROC ordinate, cpp codeRSM_wLLF
Simulate paired binned data for testing FitCorCbmSimulateCorCbmDataset
Simulates an MRMC uncorrelated FROC dataset using the RSMSimulateFrocDataset
Simulates an "AUC-equivalent" FROC dataset from an LROC datasetSimulateFrocFromLrocDataset
Simulates an uncorrelated FLROC FrocDataset using the RSMSimulateLrocDataset
Simulates a binormal model ROC datasetSimulateRocDataset
Construct RSM NH model for FROC sample size estimationSsFrocNhRsmModel
RSM fitted model for FROC sample sizeSsFrocSampleSize
Statistical power for specified numbers of readers and casesSsPowerGivenJK
Power given J, K and Dorfman-Berbaum-Metz variance componentsSsPowerGivenJKDbmVarCom
Power given J, K and Obuchowski-Rockette variance componentsSsPowerGivenJKOrVarCom
Generate a power table using the OR methodSsPowerTable
Number of cases, for specified number of readers, to achieve desired powerSsSampleSizeKGivenJ
DBM or OR significance testing for a one treatment factorial or two-treatment crossed modality factorial dataset (not SPLIT_PLOT)St
Significance testing: standalone CAD vs. radiologistsStCadVsRad
Performs OR significance testing for SPLIT-PLOT A or C datasetsStSP
Convert from physical to intrinsic RSM parametersUtil2Intrinsic
Convert from intrinsic to physical RSM parametersUtil2Physical
RSM ROC/AFROC/wAFROC AUC calculatorUtilAnalyticalAucsRSM
Binormal model AUC functionUtilAucBIN
CBM AUC functionUtilAucCBM
PROPROC AUC functionUtilAucPROPROC
Convert from DBM to OR variance componentsUtilDBM2ORVarCom
Utility for Dorfman-Berbaum-Metz variance componentsUtilDBMVarComp
Calculate empirical figures of merit (FOMs) for factorial dataset, standard one-treatment or two-treatment cross-modalityUtilFigureOfMerit
The 'lesionID' distribution of a dataset *or* a supplied 1D-arrayUtilLesDistr
Lesion weights distribution matrixUtilLesWghts UtilLesWghtsDS UtilLesWghtsLD
Calculate mean squares for factorial datasetUtilMeanSquares
Convert from OR to DBM variance componentsUtilOR2DBMVarCom
Obuchowski-Rockette variance components for datasetUtilORVarComp
Pseudovalues for given factorial or crossed modality dataset and FOMUtilPseudoValues