We assume a three-class setting, where one or more classifier yields
measurements \(X = x\) on a continuous
scale for the groups of healthy (\(D^-\)), intermediate (\(D^0\)) and diseased (\(D^+\)) individuals. By convention, larger
values of \(x\) represent a more severe
status of the disease. The package trinROC provides
statistical tests to assess the discriminatory power of such
classifiers. These tests are:
In this document, we refer to the tests by the corresponding R
function name. All tests can assess a single classifier, as well as
compare two paired or unpaired classifiers.
As their names suggest, the underlying testing approach is different
and either based on VUS or on ROC. Hence, their null hypotheses differ
as well, as illustrated now.
VUS based statistical tests
Given two classifiers, boot.test and
trinVUS.test are based on the null hypothesis \(VUS_1 = VUS_2\) with the \(Z\)-statistic
\[\begin{align*}
Z = \frac{\widehat{VUS}_1 -
\widehat{VUS}_2}{\sqrt{\widehat{Var}(\widehat{VUS}_1) +
\widehat{Var}(\widehat{VUS}_2) - 2
\widehat{Cov}(\widehat{VUS}_1,\widehat{VUS}_2)}}.
\end{align*}\]
If the data of the two classifiers is unpaired, the term \(Cov(VUS_1,VUS_2)\) is zero. If a single
classifier is investigated, the null hypothesis is \(VUS_1 = 1/6\) with the \(Z\)-statistic
\[\begin{align*}
Z = \frac{\widehat{VUS}_1 -
1/6}{\sqrt{\widehat{Var}(\widehat{VUS}_1)}},
\end{align*}\]
which is equivalent to compared the VUS of the classifier to the
volume of an uninformative classifier. Details about the estimators are
given in the aforementioned papers.
The trinormal based ROC test
In the trinormal model, the ROC surface is given by
\[\begin{align*}
ROC(t_-,t_+) = \Phi \left(\frac{\Phi^{-1} (1-t_+) + d}{ c} \right) -
\Phi \left(\frac{\Phi^{-1} (t_-)+b}{a} \right),
\end{align*}\]
where \(\Phi\) is the cdf of the
standard normal distribution and \(a\),
\(b\), \(c\) and \(d\) are functions of the means and standard
deviations of the three groups:
\[\begin{align*}
a = \frac{{\sigma}_0}{{\sigma}_-}, \qquad b = \frac{ {\mu}_- -
{\mu}_0}{{\sigma}_-}, \qquad
c = \frac{{\sigma}_0}{{\sigma}_+}, \qquad d = \frac{ {\mu}_+ -
{\mu}_0}{{\sigma}_+}.
\end{align*}\]
Given two classifiers, trinROC.test investigates the
shape of the two ROC surfaces. Hence, the resulting null hypothesis is
\(a_1=a_2\), \(b_1=b_2\), \(c_1=c_2\) and \(d_1=d_2\), i.e., the the surfaces have the
same shape. Under the null hypothesis, the test statistic
\[\begin{align*}
\chi^2 =
\begin{pmatrix} \widehat{a}_1 - \widehat{a}_2 &\widehat{b}_1
-\widehat{b}_2 & \widehat{c}_1-\widehat{c}_2 &
\widehat{d}_1-\widehat{d}_2 \end{pmatrix}
{ \widehat{\boldsymbol{W}}}^{-1}
\begin{pmatrix} \widehat{a}_1 - \widehat{a}_2 \\
\widehat{b}_1 - \widehat{b}_2 \\ \widehat{c}_1-\widehat{c}_2 \\
\widehat{d}_1- \widehat{d}_2 \end{pmatrix},
\end{align*}\]
is distributed approximately as a chi-squared random variables with
four degrees of freedom, where \(\widehat{\boldsymbol{W}}\) contains the
corresponding estimated variances and covariances. The test rejects if
\(\chi^2 > \chi_{\alpha}^2\), i.e.,
if the test statistics exceeds the chi-squared quantile with four
degrees of freedom of a pre-defined confidence level \(\alpha\).
When the data is unpaired, \(\widehat{a}_1\), \(\widehat{b}_1\), \(\widehat{c}_1\), \(\widehat{d}_1\) are independent from \(\widehat{a}_2\), \(\widehat{b}_2\), \(\widehat{c}_2\), \(\widehat{d}_2\), respectively, and hence
every combination of covariances between them is zero.
It is also possible to investigate a single classifier with the above
method. Instead of an existing second classifier, we compare the
estimates \(\widehat{a}_1\) , \(\widehat{b}_1\), \(\widehat{c}_1\) and \(\widehat{d}_1\) of a single marker with
those from an artificial marker. If we want to detect whether a single
marker is significantly better in allocating individuals to the three
classes than a random allocation function we would set the parameters of
our null hypothesis \(a_{Ho} = 1=
c_{Ho}\), as we assume equal spread in the classes and \(b_{Ho} = 0 = d_{Ho}\), as we impose equal
means. This yields the null hypothesis \(a_1=1\), \(b_1=0\), \(c_1=1\), \(d_1=0\), which leads to the chi-squared
test
\[\begin{align*}
{\chi}^2 = &
\begin{pmatrix} \widehat{a}_1 -1 & \widehat{b}_1 & \widehat{c}_1
-1 & \widehat{d}_1 \end{pmatrix}
\widehat{\boldsymbol{W}}^{-1}
\begin{pmatrix} \widehat{a}_1 -1 \\ \widehat{b}_1 \\ \widehat{c}_1 -
1 \\ \widehat{d}_1 \end{pmatrix}.
\end{align*}\]
trinROC_vignette.R
