morphemepiece: Morpheme Tokenization

Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.

Version: 1.0.1
Imports: dlr (≥ 1.0.0), magrittr,, piecemaker (≥ 1.0.0), purrr (≥ 0.3.4), rlang, stringr (≥ 1.4.0)
Suggests: dplyr, fs, ggplot2, here, knitr, remotes, rmarkdown, testthat (≥ 3.0.0), utils
Published: 2021-09-17
Author: Jonathan Bratt ORCID iD [aut, cre], Jon Harmon ORCID iD [aut], Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [cph]
Maintainer: Jonathan Bratt <jonathan.bratt at>
License: Apache License (≥ 2)
NeedsCompilation: no
Materials: README NEWS
CRAN checks: morphemepiece results


Reference manual: morphemepiece.pdf
Vignettes: Testing the fall-through algorithm
Generating a Vocabulary and Lookup
Package source: morphemepiece_1.0.1.tar.gz
Windows binaries: r-devel:, r-release:, r-oldrel:
macOS binaries: r-release (arm64): morphemepiece_1.0.0.tgz, r-release (x86_64): morphemepiece_1.0.1.tgz, r-oldrel: morphemepiece_1.0.1.tgz
Old sources: morphemepiece archive


Please use the canonical form to link to this page.