tokenizers: Fast, Consistent Tokenization of Natural Language Text

Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, tweets, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.

Version: 0.2.0
Depends: R (≥ 3.1.3)
Imports: stringi (≥ 1.0.1), Rcpp (≥ 0.12.3), SnowballC (≥ 0.5.1)
LinkingTo: Rcpp
Suggests: covr, knitr, rmarkdown, stopwords (≥ 0.9.0), testthat
Published: 2018-03-21
Author: Lincoln Mullen ORCID iD [aut, cre], Os Keyes ORCID iD [ctb], Dmitriy Selivanov [ctb], Jeffrey Arnold [ctb], Kenneth Benoit ORCID iD [ctb]
Maintainer: Lincoln Mullen <lincoln at>
License: MIT + file LICENSE
NeedsCompilation: yes
Materials: README NEWS
In views: NaturalLanguageProcessing
CRAN checks: tokenizers results


Reference manual: tokenizers.pdf
Vignettes: Introduction to the tokenizers Package
The Text Interchange Formats and the tokenizers Package
Package source: tokenizers_0.2.0.tar.gz
Windows binaries: r-devel:, r-release:, r-oldrel:
OS X El Capitan binaries: r-release: tokenizers_0.2.0.tgz
OS X Mavericks binaries: r-oldrel: tokenizers_0.1.4.tgz
Old sources: tokenizers archive

Reverse dependencies:

Reverse imports: covfefe, proustr, ptstem, tidytext
Reverse suggests: text2vec


Please use the canonical form to link to this page.