tokenizers: A Consistent Interface to Tokenize Natural Language Text

Convert natural language text into tokens. The tokenizers have a consistent interface and are compatible with Unicode, thanks to being built on the 'stringi' package. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, lines, and regular expressions.

Version: 0.1.4
Depends: R (≥ 3.1.3)
Imports: stringi (≥ 1.0.1), Rcpp (≥ 0.12.3), SnowballC (≥ 0.5.1)
LinkingTo: Rcpp
Suggests: testthat, covr, knitr, rmarkdown
Published: 2016-08-29
Author: Lincoln Mullen [aut, cre], Dmitriy Selivanov [ctb]
Maintainer: Lincoln Mullen <lincoln at lincolnmullen.com>
BugReports: https://github.com/ropensci/tokenizers/issues
License: MIT + file LICENSE
URL: https://github.com/ropensci/tokenizers
NeedsCompilation: yes
Materials: README NEWS
In views: NaturalLanguageProcessing
CRAN checks: tokenizers results

Downloads:

Reference manual: tokenizers.pdf
Vignettes: Introduction to the tokenizers Package
Package source: tokenizers_0.1.4.tar.gz
Windows binaries: r-devel: tokenizers_0.1.4.zip, r-release: tokenizers_0.1.4.zip, r-oldrel: tokenizers_0.1.4.zip
OS X Mavericks binaries: r-release: tokenizers_0.1.4.tgz, r-oldrel: tokenizers_0.1.4.tgz
Old sources: tokenizers archive

Reverse dependencies:

Reverse imports: ptstem, tidytext

Linking:

Please use the canonical form https://CRAN.R-project.org/package=tokenizers to link to this page.