An R Package which implements model comparison tests using cross-validated log-likelihood (CVLL) values.

`modeLLtest`

is an R package which implements model
comparison tests. This package includes functions for the
cross-validated difference in means (CVDM) test and the cross-validated
median fit (CVMF) test. The CVDM and CVMF tests assist researchers and
students in selecting among models describing the same process.
Selection among estimation methods describing the same process is a
crucial methodological step within the social sciences. Other tools in
`modeLLtest`

include a function to output a vector of
cross-validated log-likelihood (CVLL) values and a function to perform
the CVDM test on an input of two CVLL vectors. The relevant papers
including details can be found below:

Harden, J. J., & Desmarais, B. A. (2011). Linear Models with Outliers: Choosing between Conditional-Mean and Conditional-Median Methods. State Politics & Policy Quarterly, 11(4), 371-389. https://doi.org/10.1177/1532440011408929

Desmarais, B. A., & Harden, J. J. (2012). Comparing partial likelihood and robust estimation methods for the Cox regression model. Political Analysis, 20(1), 113-135. https://doi.org/10.1093/pan/mpr042

Desmarais, B. A., & Harden, J. J. (2014). An unbiased model comparison test using cross-validation. Quality & Quantity, 48(4), 2155-2173. https://doi.org/10.1007/s11135-013-9884-7

The easiest way to install `modeLLtest`

is to use
`install.packages()`

and download it from CRAN.

`install.packages("modeLLtest")`

Installing from CRAN should avoid any compilation issues that can
arise and is the quickest option. However, the newest version of the
package can also be downloaded with the `devtools`

package in
R from GitHub. To do this, install devtools by calling:

`install.packages("devtools")`

Now we can install from GitHub with the following line:

`devtools::install_github("ShanaScogin/modeLLtest")`

Once you have installed the package from either CRAN or GitHub, you can access it by calling:

`library(modeLLtest)`

After the package is loaded, check out the `?modeLLtest`

to see a help file. You can also see the documentation for the functions
with `?cvdm`

, `?cvll`

, `?cvlldiff`

, or
`?cvmf`

. If you have issues or questions, please email me at
sscogin@nd.edu.

Some users might experience gfortran errors due to Rcpp, RcppArmadillo, and MacOS. To fix this problem, consider installing gfortran 6.1 from https://cran.r-project.org/bin/macosx/tools/. (Also check out Yiqing Xu and Licheng Liu’s note on this type of error in the gsynth package for some helpful links.)

This package has four main functions: `cvdm()`

,
`cvll()`

, `cvmf()`

, and `cvlldiff()`

.
The function `cvdm()`

deploys the CVDM test, which uses a
bias-corrected Johnson’s t-test to choose between the leave-one-out
cross-validated log-likelihood outputs of two non-nested models. The
function `cvll()`

outputs a vector of leave-one-out
cross-validated log-likelihoods for a given method. Currently, these
functions accommodate linear regression, median regression (from the
package `quantreg`

), and two methods of robust regression
(from the package `MASS`

). The`cvlldiff()`

function performs the bias-corrected Johnson’s t-test on two vectors of
cross-validated log-likelihoods. Finally, the `cvmf()`

function test between the partial likelihood maximization (PLM) and the
iteratively reweighted robust (IRR) methods of estimation for a given
application of the Cox model.

After loading the package, you can find the documentation for the
functions with `?cvdm`

, `?cvll`

,
`?cvlldiff`

, or `?cvmf`

. For the output, type
`?cvdm_object`

, `?cvll_object`

,
`?cvlldiff_object`

, or `?cvmf_object`

to view the
documentation. If you encounter a bug or have comments or questions,
please email me at sscogin@nd.edu.

Here are some examples of the functions in this package. First, we’ll
look at `cvdm()`

, which applies cross-validated
log-likelihood difference in means (CVDM) test to compare two methods of
estimating a formula. The example compares ordinary least squares (OLS)
estimation to median regression (MR).

```
library(modeLLtest)
set.seed(123456)
b0 <- .2 # True value for the intercept
b1 <- .5 # True value for the slope
n <- 500 # Sample size
X <- runif(n, -1, 1)
Y <- b0 + b1 * X + rnorm(n, 0, 1) # N(0, 1 error)
obj_cvdm <- cvdm(Y ~ X, data.frame(cbind(Y, X)), method1 = "OLS", method2 = "MR")
obj_cvdm
```

Next, let’s do the same as we did above, but with `cvll()`

and `cvlldiff()`

. These are general functions that create
vectors of the cross-validated log-likelihood functions and then compute
the bias-corrected Johnson’s t-test, respectively.

```
library(modeLLtest)
set.seed(123456)
b0 <- .2 # True value for the intercept
b1 <- .5 # True value for the slope
n <- 500 # Sample size
X <- runif(n, -1, 1)
Y <- b0 + b1 * X + rnorm(n, 0, 1) # N(0, 1 error)
obj_cvll_ols <- cvll(Y ~ X, data.frame(cbind(Y, X)), method = "OLS")
obj_cvll_mr <- cvll(Y ~ X, data.frame(cbind(Y, X)), method = "MR")
obj_cvlldiff <- cvlldiff(obj_cvll_ols$cvll, obj_cvll_mr$cvll,
obj_cvll_ols$df)
obj_cvlldiff
```

Finally, let’s look at the `cvmf()`

function. This
function compares the partial likelihood maximization (PLM) and the
iteratively reweighted robust (IRR) methods of estimation for a given
application of the Cox model. Note: This function currently runs slowly
(approximately 3 seconds for one run). Future developments look to
optimize this function.

```
library(modeLLtest)
library(survival)
set.seed(12345)
x1 <- rnorm(100)
x2 <- rnorm(100)
x2e <- x2 + rnorm(100, 0, 0.5)
y <- rexp(100, exp(x1 + x2))
y <- Surv(y) # Changing y into a survival obj with survival package
dat <- data.frame(y, x1, x2e)
form <- y ~ x1 + x2e
obj_cvmf <- cvmf(formula = form, data = dat)
obj_cvmf
```

This package includes two datasets from real-world analyses to facilitate examples. These datasets are publicly available and have been included in this package with the endorsement of the authors. More on the data and original analysis can be found in the following papers:

Joshi, M., & Mason, T. D. (2008). Between democracy and revolution: peasant support for insurgency versus democracy in Nepal. Journal of Peace Research, 45(6), 765-782. https://doi.org/10.1177/0022343308096155

Golder, S. N. (2010). Bargaining delays in the government formation process. Comparative Political Studies, 43(1), 3-32. https://doi.org/10.1177/0010414009341714

For an example of the CVDM test utilizing real-world analysis, we can look at a study by Joshi and Mason (2008, Journal of Peace Research 45(6): 765-782). This study employs robust regression to analyze district-level election turnout among peasants in Nepal. Specifically, Joshi and Mason hypothesize that peasant dependence on landed elite for survival will result in higher voter turnout. Using their model of the 1999 parliamentary elections, we can see how the use of a robust regression is supported by the CVDM test. These data are available on the Journal of Peace Research Replication Datasets website and have been included in the package for ease of replication. For full replication and discussion of the CVDM test, see Desmarais and Harden (2014, Quality and Quantity 48(4): 2155-2173).

```
library(MASS)
library(modeLLtest)
data(nepaldem)
set.seed(978)
obj_cvdm_jm <- cvdm(percent_regvote1999 ~ landless_gap +
below1pa_gap + sharecrop_gap + service_gap + fixmoney_gap +
fixprod_gap + per_without_instcredit + totoalkilled_1000 +
hdi_gap1 + ln_pop2001 + totalcontestants1999 + cast_eth_fract,
data = nepaldem, method1 = "OLS", method2 = "RLM")
obj_cvdm_jm
model_1999 <- rlm(percent_regvote1999 ~ landless_gap +
below1pa_gap + sharecrop_gap + service_gap + fixmoney_gap +
fixprod_gap + per_without_instcredit + totoalkilled_1000 +
hdi_gap1 + ln_pop2001 + totalcontestants1999 + cast_eth_fract,
data = nepaldem)
model_1999
```

Next, we can look at a study by Golder (2010, Comparative Political Studies 43(1): 3-32) to see an example of the CVMF test. Golder employs the PLM method of estimating a Cox model to investigate Western European government formation duration. She hypothesizes that bargaining complexity leads to increasing delays in government formation as uncertainty increases. The CVMF test indicates that her choice of PLM is the better performing estimator (p < .05) compared to IRR. These data are available on the Harvard Dataverse page (https://doi.org/10.7910/DVN/BUWZBA) and have been included in the package for ease of replication. For full replication and discussion of the CVMF test, see Desmarais and Harden (2012, Political Analysis 20(1): 113-135).

```
library(survival)
library(coxrobust)
library(modeLLtest)
data(govtform)
golder_surv <- Surv(govtform$bargainingdays)
golder_x <- cbind(govtform$postelection, govtform$legislative_parties,
govtform$polarization, govtform$positive_parl,
govtform$post_legislative_parties,
govtform$post_polariz, govtform$post_positive, govtform$continuation,
govtform$singleparty_majority)
colnames(golder_x) <- c("govtform$postelection",
"govtform$legislative_parties",
"govtform$polarization", "govtform$positive_parl",
"govtform$post_legislative_parties", "govtform$post_polariz",
"govtform$post_positive", "govtform$continuation",
"govtform$singleparty_majority")
obj_cvmf_golder <- cvmf(golder_surv ~ golder_x, method = "efron")
obj_cvmf_golder
govtform_plm <- coxph(golder_surv ~ golder_x, method = "efron")
govtform_plm
```

Next steps for this package include adding more methods to the
`cvdm()`

and `cvll()`

functions and optimizing
functions - especially `cvmf()`

- to improve speed.

Please submit an issue if you encounter any bugs or problems with the package.