tokenizers: A Consistent Interface to Tokenize Natural Language Text

Convert natural language text into tokens. The tokenizers have a consistent interface and are compatible with Unicode, thanks to being built on the 'stringi' package. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, lines, and regular expressions.

Version: 0.1.4
Depends: R (≥ 3.1.3)
Imports: stringi (≥ 1.0.1), Rcpp (≥ 0.12.3), SnowballC (≥ 0.5.1)
LinkingTo: Rcpp
Suggests: testthat, covr, knitr, rmarkdown
Published: 2016-08-29
Author: Lincoln Mullen [aut, cre], Dmitriy Selivanov [ctb]
Maintainer: Lincoln Mullen <lincoln at>
License: MIT + file LICENSE
NeedsCompilation: yes
Materials: README NEWS
In views: NaturalLanguageProcessing
CRAN checks: tokenizers results


Reference manual: tokenizers.pdf
Vignettes: Introduction to the tokenizers Package
Package source: tokenizers_0.1.4.tar.gz
Windows binaries: r-devel:, r-release:, r-oldrel:
OS X El Capitan binaries: r-release: tokenizers_0.1.4.tgz
OS X Mavericks binaries: r-oldrel: tokenizers_0.1.4.tgz
Old sources: tokenizers archive

Reverse dependencies:

Reverse imports: covfefe, proustr, ptstem, tidytext
Reverse suggests: text2vec


Please use the canonical form to link to this page.