Contains Nonbinding Recommendations
10
(see Section V. Study Population) (powering each cohort for statistical significance is not
necessary unless you are making specific subset performance claims); and
· confidence intervals (CIs) that account for reader variability, case variability, and truth
variability or other sources of variability when appropriate.
We recommend that you identify and validate your analysis software.
You should provide a
reference to the analysis approach used, clarify the software implementation, and specify a
version number if appropriate. Certain validated MRMC analysis approaches, examples of which
can be found in the literature or obtained online, may be appropriate for your device evaluation
depending on its intended use and conditions of use.
,
The definitions of a true positive, true negative, false positive, and false negative CADe mark
should be consistent with the intended use of the device and the characterization of the reference
standard (see Section VI. Reference Standard).
B. Control Arm
We recommend that you assess the clinical performance of your CADe device relative to a
control modality. A study control arm that uses conventional clinical interpretation (i.e., reader
interpretation without the CADe device) should generally be the most relevant comparator in
CADe performance assessment. For CADe devices intended as second readers, another possible
control is double reading by two clinicians. These controls or a direct comparison with the
predicate CADe device should generally be appropriate for establishing substantial equivalence.
Other control arms may be valid. We recommend that you contact the Agency to discuss your
choice of a control arm prior to conducting your clinical study.
The study control arm should utilize the same reading methodology as the device arm and be
consistent with clinical practice. The same population of cases, if not the same cases themselves,
should be in all study arms to minimize potential bias. For designs that include distinct cases in
each study arm, we recommend that you provide a description and flow chart demonstrating how
you randomized patients and readers into the different arms.
For more information on MRMC analysis software, see, for example, Obuchowski, N.A., Beiden, S.V., Berbaum,
K.S., Hillis, S.L., Ishwaran, H., Song, H.H., and Wagner, R.F., Multi-reader, multi-case ROC analysis: An empirical
comparison of five methods. Acad. Radiol., 2004. 11(9); 980–995. https://doi.org/10.1016/j.acra.2004.04.014.
For MRMC literature references, see, for example: Metz, C.E., Fundamental ROC analysis. Handbook of Medical
Imaging, Vol. 1. Physics and Psychophysics. SPIE Press, 2000. Chapter 15, 751–769.
https://doi.org/10.1117/3.832716.ch15; Wagner, R.F., Metz, C.E., and Campbell, G., Assessment of medical
imaging systems and computer aids: A tutorial review. Acad. Radiol., 2007. 14(6):723–48.
https://doi.org/10.1016/j.acra.2007.03.001; and Obuchowski, N.A., Beiden, S.V., Berbaum, K.S., Hillis, S.L.,
Ishwaran, H., Song, H.H., and Wagner, R.F., Multi-reader, multi-case ROC analysis: An empirical comparison of
five methods. Acad. Radiol., 2004. 11(9); 980–995. https://doi.org/10.1016/j.acra.2004.04.014.
For online access to software that analyzes MRMC data based on validated techniques, see, for example:
LABMRMC software and general ROC software, The University of Chicago: http://metz-roc.uchicago.edu/ (for
either quasi-continuous or categorical data); University of Iowa MRMC software:
https://perception.lab.uiowa.edu/OR-DBM-MRMC-program-manual (for categorical data); or OBUMRM software:
https://www.lerner.ccf.org/qhs/software/.