To cite randomForest in publications use:

Winham, Stacey; Jenkins, Gregory; Biernacka, Joanna (2015). Addressing X-inactivation in Random Forest Classifiers. blah journal (?)?, ???--???.

Corresponding BibTeX entry:

  @Article{,
    title = {Classification and Regression by randomForest},
    author = {Stacey Winham and Gregory Jenkins and Joanna Biernacka},
    journal = {???},
    year = {????},
    volume = {?},
    number = {?},
    pages = {???-???},
    url = {???},
  }