# Introduction

An R package for managing and analyzing text.

quanteda makes it easy to manage texts in the form of a corpus, defined as a collection of texts that includes document-level variables specific to each text, as well as meta-data for documents and for the collection as a whole. quanteda includes tools to make it easy and fast to manuipulate the texts in a corpus, by performing the most common natural language processing tasks simply and quickly, such as tokenizing, stemming, or forming ngrams. quanteda’s functions for tokenizing texts and forming multiple tokenized documents into a document-feature matrix are both extremely fast and extremely simple to use. quanteda can segment texts easily by words, paragraphs, sentences, or even user-supplied delimiters and tags.

Built on the text processing functions in the stringi package, which is in turn built on C++ implementation of the ICU libraries for Unicode text handling, quanteda pays special attention to fast and correct implementation of Unicode and the handling of text in any character set, following conversion internally to UTF-8.

quanteda is built for efficiency and speed, through its design around three infrastructures: the string package for text processing, the data.table package for indexing large documents efficiently, and the Matrix package for sparse matrix objects. If you can fit it into memory, quanteda will handle it quickly. (And eventually, we will make it possible to process objects even larger than available memory.)

quanteda is principally designed to allow users a fast and convenient method to go from a corpus of texts to a selected matrix of documents by features, after defining what the documents and features. The package makes it easy to redefine documents, for instance by splitting them into sentences or paragraphs, or by tags, as well as to group them into larger documents by document variables, or to subset them based on logical conditions or combinations of document variables. The package also implements common NLP feature selection functions, such as removing stopwords and stemming in numerous languages, selecting words found in dictionaries, treating words as equivalent based on a user-defined “thesaurus”, and trimming and weighting features based on document frequency, feature frequency, and related measures such as tf-idf.

# quanteda Features

## Corpus management tools

The tools for getting texts into a corpus object include:

• loading texts manually’’ by inserting them into a corpus using helper functions
• managing text encodings and conversions from source files into corpus texts
• attaching variables to each text that can be used for grouping, reorganizing a corpus, or simply recording additional information to supplement quantitative analyses with non-textual data
• recording meta-data about the sources and creation details for the corpus.

The tools for working with a corpus include:

• summarizing the corpus in terms of its language units
• reshaping the corpus into smaller units or more aggregated units
• adding to or extracting subsets of a corpus
• resampling texts of the corpus, for example for use in non-parametric bootstrapping of the texts
• Easy extraction and saving, as a new data frame or corpus, key words in context (KWIC)

## Natural-Language Processing tools

For extracting features from a corpus, quanteda provides the following tools:

• extraction of word types
• extraction of word n-grams
• extraction of dictionary entries from user-defined dictionaries
• feature selection through
• stemming
• random selection
• document frequency
• word frequency
• and a variety of options for cleaning word types, such as capitalization and rules for handling punctuation.

## Document-Feature Matrix analysis tools

For analyzing the resulting document-feature matrix created when features are abstracted from a corpus, quanteda provides:

• scaling methods, such as correspondence analysis, Wordfish, and Wordscores
• topic models, such as LDA
• classifiers, such as Naive Bayes or k-nearest neighbour
• sentiment analysis, using dictionaries

• the ability to explore texts using key-words-in-context;

• fast computation of a variety of readability indexes;

• fast computation of a variety of lexical diversity measures;

• quick computation of word or document association measures, for clustering or to compute similarity scores for other purposes; and

• a comprehensive suite of descriptive statistics on text such as the number of sentences, words, characters, or syllables per document.

Planned features coming soon to quanteda are:

• bootstrapping methods for texts that makes it easy to resample texts from pre-defined units, to facilitate computation of confidence intervals on textual statistics using techniques of non-parametric bootstrapping, but applied to the original texts as data.

• expansion of the document-feature matrix structure through a standard interface called textmodel(). (As of version 0.8.0, textmodel works in a basic fashion only for the “Wordscores” and “wordfish” scaling models.)

## Working with other text analysis packages

quanteda is hardly unique in providing facilities for working with text – the excellent tm package already provides many of the features we have described. quanteda is designed to complement those packages, as well to simplify the implementation of the text-to-analysis workflow. quanteda corpus structures are simpler objects than in tms, as are the document-feature matrix objects from quanteda, compared to the sparse matrix implementation found in tm. However, there is no need to choose only one package, since we provide translator functions from one matrix or corpus object to the other in quanteda.

Once constructed, a quanteda “dfm”" can be easily passed to other text-analysis packages for additional analysis of topic models or scaling, such as:

• topic models (including converters for direct use with the topicmodels, LDA, and stm packages)

• document scaling (using quanteda’s own functions for the “wordfish” and “Wordscores” models, direct use with the ca package for correspondence analysis, or scaling with the austin package)

• document classification methods, using (for example) Naive Bayes, k-nearest neighbour, or Support Vector Machines

• more sophisticated machine learning through a variety of other packages that take matrix or matrix-like inputs.

• graphical analysis, including word clouds and strip plots for selected themes or words.

# How to Install

Through a normal installation of the package from CRAN, or for the GitHub version, see the installation instructions at https://github.com/kbenoit/quanteda.

# Creating and Working with a Corpus

require(quanteda)

## Currently available corpus sources

quanteda has a simple and powerful companion package for loading texts: readtext. The main function in this package, readtext(), takes a file or fileset from disk or a URL, and returns a type of data.frame that can be used directly with the corpus() constructor function, to create a quanteda corpus object.

readtext() works on:

• text (.txt) files;
• comma-separated-value (.csv) files;
• XML formatted data;
• data from the Facebook API, in JSON format;
• data from the Twitter API, in JSON format; and
• generic JSON data.

The corpus constructor command corpus() works directly on:

• a vector of character objects, for instance that you have already loaded into the workspace using other tools;
• a VCorpus corpus object from the tm package.
• a data.frame containing a text column and any other document-level metadata.

### Example: building a corpus from a character vector

The simplest case is to create a corpus from a vector of texts already in memory in R. This gives the advanced R user complete flexbility with his or her choice of text inputs, as there are almost endless ways to get a vector of texts into R.

If we already have the texts in this form, we can call the corpus constructor function directly. We can demonstrate this on the built-in character vector of 57 US president inaugural speeches called data_char_inaugural.

str(data_char_inaugural)  # this gives us some information about the object
##  Named chr [1:58] "Fellow-Citizens of the Senate and of the House of Representatives:\n\nAmong the vicissitudes incident to life no event could ha"| __truncated__ ...
##  - attr(*, "names")= chr [1:58] "1789-Washington" "1793-Washington" "1797-Adams" "1801-Jefferson" ...
myCorpus <- corpus(data_char_inaugural)  # build the corpus
summary(myCorpus, n = 5)
## Corpus consisting of 58 documents, showing 5 documents.
##
##             Text Types Tokens Sentences
##  1789-Washington   626   1540        23
##  1793-Washington    96    147         4
##   1801-Jefferson   716   1935        41
##   1805-Jefferson   804   2381        45
##
## Source:  /private/var/folders/92/64fddl_57nddq_wwqpjnglwn48rjsn/T/Rtmp6iJH22/Rbuild435d49afd407/quanteda/vignettes/* on x86_64 by kbenoit
## Created: Mon Feb 13 18:57:22 2017
## Notes:

If we wanted, we could add some document-level variables – what quanteda calls docvars – to this corpus.

We can do this using the R’s substring() function to extract characters from a name – in this case, the name of the character vector data_char_inaugural. This works using our fixed starting and ending positions with substring() because these names are a very regular format of YYYY-PresidentName.

docvars(myCorpus, "President") <- substring(names(data_char_inaugural), 6)
docvars(myCorpus, "Year") <- as.integer(substring(names(data_char_inaugural), 1, 4))
summary(myCorpus, n=5)
## Corpus consisting of 58 documents, showing 5 documents.
##
##             Text Types Tokens Sentences  President Year
##  1789-Washington   626   1540        23 Washington 1789
##  1793-Washington    96    147         4 Washington 1793
##   1801-Jefferson   716   1935        41  Jefferson 1801
##   1805-Jefferson   804   2381        45  Jefferson 1805
##
## Source:  /private/var/folders/92/64fddl_57nddq_wwqpjnglwn48rjsn/T/Rtmp6iJH22/Rbuild435d49afd407/quanteda/vignettes/* on x86_64 by kbenoit
## Created: Mon Feb 13 18:57:22 2017
## Notes:

If we wanted to tag each document with additional meta-data not considered a document variable of interest for analysis, but rather something that we need to know as an attribute of the document, we could also add those to our corpus.

metadoc(myCorpus, "language") <- "english"
metadoc(myCorpus, "docsource")  <- paste("data_char_inaugural", 1:ndoc(myCorpus), sep = "_")
summary(myCorpus, n = 5, showmeta = TRUE)
## Corpus consisting of 58 documents, showing 5 documents.
##
##             Text Types Tokens Sentences  President Year _language
##  1789-Washington   626   1540        23 Washington 1789   english
##  1793-Washington    96    147         4 Washington 1793   english
##   1801-Jefferson   716   1935        41  Jefferson 1801   english
##   1805-Jefferson   804   2381        45  Jefferson 1805   english
##             _docsource
##  data_char_inaugural_1
##  data_char_inaugural_2
##  data_char_inaugural_3
##  data_char_inaugural_4
##  data_char_inaugural_5
##
## Source:  /private/var/folders/92/64fddl_57nddq_wwqpjnglwn48rjsn/T/Rtmp6iJH22/Rbuild435d49afd407/quanteda/vignettes/* on x86_64 by kbenoit
## Created: Mon Feb 13 18:57:22 2017
## Notes:

The last command, metadoc, allows you to define your own document meta-data fields. Note that in assiging just the single value of "english", R has recycled the value until it matches the number of documents in the corpus. In creating a simple tag for our custom metadoc field docsource, we used the quanteda function ndoc() to retrieve the number of documents in our corpus. This function is deliberately designed to work in a way similar to functions you may already use in R, such as nrow() and ncol().

require(readtext)

# generic json - needs a textField specifier
textField = "text")
summary(corpus(mytf2), 5)
# text file
mytf3 <- readtext("~/Dropbox/QUANTESS/corpora/project_gutenberg/pg2701.txt", cache = FALSE)
summary(corpus(mytf3), 5)
# multiple text files
mytf4 <- readtext("~/Dropbox/QUANTESS/corpora/inaugural/*.txt", cache = FALSE)
summary(corpus(mytf4), 5)
# multiple text files with docvars from filenames
docvarsfrom="filenames", sep="-", docvarnames=c("Year", "President"))
summary(corpus(mytf5), 5)
# XML data
textField = "COMMON")
summary(corpus(mytf6), 5)
# csv file
write.csv(data.frame(inaugSpeech = texts(data_corpus_inaugural), docvars(data_corpus_inaugural)),
file = "/tmp/inaug_texts.csv", row.names = FALSE)
mytf7 <- readtext("/tmp/inaug_texts.csv", textField = "inaugSpeech")
summary(corpus(mytf7), 5)

## How a quanteda corpus works

### Corpus principles

A corpus is designed to be a “library” of original documents that have been converted to plain, UTF-8 encoded text, and stored along with meta-data at the corpus level and at the document-level. We have a special name for document-level meta-data: docvars. These are variables or features that describe attributes of each document.

A corpus is designed to be a more or less static container of texts with respect to processing and analysis. This means that the texts in corpus are not designed to be changed internally through (for example) cleaning or pre-processing steps, such as stemming or removing punctuation. Rather, texts can be extracted from the corpus as part of processing, and assigned to new objects, but the idea is that the corpus will remain as an original reference copy so that other analyses – for instance those in which stems and punctuation were required, such as analyzing a reading ease index – can be performed on the same corpus.

To extract texts from a a corpus, we use an extractor, called texts().

texts(data_corpus_inaugural)[2]
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              1793-Washington
## "Fellow citizens, I am again called upon by the voice of my country to execute the functions of its Chief Magistrate. When the occasion proper for it shall arrive, I shall endeavor to express the high sense I entertain of this distinguished honor, and of the confidence which has been reposed in me by the people of united America.\n\nPrevious to the execution of any official act of the President the Constitution requires an oath of office. This oath I am now about to take, and in your presence: That if it shall be found during my administration of the Government I have in any instance violated willingly or knowingly the injunctions thereof, I may (besides incurring constitutional punishment) be subject to the upbraidings of all who are now witnesses of the present solemn ceremony.\n\n "

To summarize the texts from a corpus, we can call a summary() method defined for a corpus.

summary(data_corpus_irishbudget2010)
## Corpus consisting of 14 documents.
##
##                                   Text Types Tokens Sentences year debate
##        2010_BUDGET_01_Brian_Lenihan_FF  1949   8733       374 2010 BUDGET
##       2010_BUDGET_02_Richard_Bruton_FG  1042   4478       217 2010 BUDGET
##         2010_BUDGET_03_Joan_Burton_LAB  1621   6429       307 2010 BUDGET
##        2010_BUDGET_04_Arthur_Morgan_SF  1589   7185       343 2010 BUDGET
##          2010_BUDGET_05_Brian_Cowen_FF  1618   6697       250 2010 BUDGET
##           2010_BUDGET_06_Enda_Kenny_FG  1151   4254       153 2010 BUDGET
##      2010_BUDGET_07_Kieran_ODonnell_FG   681   2309       133 2010 BUDGET
##       2010_BUDGET_08_Eamon_Gilmore_LAB  1183   4217       201 2010 BUDGET
##     2010_BUDGET_09_Michael_Higgins_LAB   490   1288        44 2010 BUDGET
##        2010_BUDGET_10_Ruairi_Quinn_LAB   442   1290        59 2010 BUDGET
##      2010_BUDGET_11_John_Gormley_Green   404   1036        49 2010 BUDGET
##        2010_BUDGET_12_Eamon_Ryan_Green   512   1651        90 2010 BUDGET
##      2010_BUDGET_13_Ciaran_Cuffe_Green   444   1248        45 2010 BUDGET
##  2010_BUDGET_14_Caoimhghin_OCaolain_SF  1188   4094       176 2010 BUDGET
##  number      foren     name party
##      01      Brian  Lenihan    FF
##      02    Richard   Bruton    FG
##      03       Joan   Burton   LAB
##      04     Arthur   Morgan    SF
##      05      Brian    Cowen    FF
##      06       Enda    Kenny    FG
##      07     Kieran ODonnell    FG
##      08      Eamon  Gilmore   LAB
##      09    Michael  Higgins   LAB
##      10     Ruairi    Quinn   LAB
##      11       John  Gormley Green
##      12      Eamon     Ryan Green
##      13     Ciaran    Cuffe Green
##      14 Caoimhghin OCaolain    SF
##
## Source:  /home/paul/Dropbox/code/quantedaData/* on x86_64 by paul
## Created: Tue Sep 16 15:58:21 2014
## Notes:

We can save the output from the summary command as a data frame, and plot some basic descriptive statistics with this information:

tokenInfo <- summary(data_corpus_inaugural)
## Corpus consisting of 58 documents.
##
##             Text Types Tokens Sentences Year  President       FirstName
##  1789-Washington   626   1540        23 1789 Washington          George
##  1793-Washington    96    147         4 1793 Washington          George
##   1801-Jefferson   716   1935        41 1801  Jefferson          Thomas
##   1805-Jefferson   804   2381        45 1805  Jefferson          Thomas
##      1817-Monroe  1040   3696       121 1817     Monroe           James
##      1821-Monroe  1262   4898       129 1821     Monroe           James
##     1829-Jackson   517   1210        25 1829    Jackson          Andrew
##     1833-Jackson   499   1271        29 1833    Jackson          Andrew
##    1837-VanBuren  1315   4175        95 1837  Van Buren          Martin
##    1841-Harrison  1893   9178       210 1841   Harrison   William Henry
##        1845-Polk  1330   5211       153 1845       Polk      James Knox
##      1849-Taylor   497   1185        22 1849     Taylor         Zachary
##      1853-Pierce  1166   3657       104 1853     Pierce        Franklin
##    1857-Buchanan   945   3106        89 1857   Buchanan           James
##     1861-Lincoln  1075   4016       135 1861    Lincoln         Abraham
##     1865-Lincoln   362    780        26 1865    Lincoln         Abraham
##       1869-Grant   486   1243        40 1869      Grant      Ulysses S.
##       1873-Grant   552   1479        43 1873      Grant      Ulysses S.
##       1877-Hayes   829   2730        59 1877      Hayes   Rutherford B.
##    1881-Garfield  1018   3240       111 1881   Garfield        James A.
##   1885-Cleveland   674   1828        44 1885  Cleveland          Grover
##    1889-Harrison  1355   4744       157 1889   Harrison        Benjamin
##   1893-Cleveland   823   2135        58 1893  Cleveland          Grover
##    1897-McKinley  1236   4383       130 1897   McKinley         William
##    1901-McKinley   857   2449       100 1901   McKinley         William
##   1905-Roosevelt   404   1089        33 1905  Roosevelt        Theodore
##        1909-Taft  1436   5844       159 1909       Taft  William Howard
##      1913-Wilson   661   1896        68 1913     Wilson         Woodrow
##      1917-Wilson   549   1656        59 1917     Wilson         Woodrow
##     1921-Harding  1172   3743       148 1921    Harding       Warren G.
##    1925-Coolidge  1221   4442       196 1925   Coolidge          Calvin
##      1929-Hoover  1086   3895       158 1929     Hoover         Herbert
##   1933-Roosevelt   744   2064        85 1933  Roosevelt     Franklin D.
##   1937-Roosevelt   729   2027        96 1937  Roosevelt     Franklin D.
##   1941-Roosevelt   527   1552        68 1941  Roosevelt     Franklin D.
##   1945-Roosevelt   276    651        26 1945  Roosevelt     Franklin D.
##      1949-Truman   781   2531       116 1949     Truman        Harry S.
##  1953-Eisenhower   903   2765       119 1953 Eisenhower       Dwight D.
##  1957-Eisenhower   621   1933        92 1957 Eisenhower       Dwight D.
##     1961-Kennedy   566   1568        52 1961    Kennedy         John F.
##     1965-Johnson   569   1725        93 1965    Johnson   Lyndon Baines
##       1969-Nixon   743   2437       103 1969      Nixon Richard Milhous
##       1973-Nixon   545   2018        68 1973      Nixon Richard Milhous
##      1977-Carter   528   1380        52 1977     Carter           Jimmy
##      1981-Reagan   904   2798       128 1981     Reagan          Ronald
##      1985-Reagan   925   2935       123 1985     Reagan          Ronald
##        1989-Bush   795   2683       141 1989       Bush          George
##     1993-Clinton   644   1837        81 1993    Clinton            Bill
##     1997-Clinton   773   2451       111 1997    Clinton            Bill
##        2001-Bush   622   1810        97 2001       Bush       George W.
##        2005-Bush   772   2325       100 2005       Bush       George W.
##       2009-Obama   939   2729       110 2009      Obama          Barack
##       2013-Obama   814   2335        88 2013      Obama          Barack
##       2017-Trump   582   1662        88 2017      Trump       Donald J.
##
## Source:  /home/paul/Dropbox/code/quanteda/* on x86_64 by paul
## Created: Fri Sep 12 12:41:17 2014
## Notes:
if (require(ggplot2))
ggplot(data=tokenInfo, aes(x=Year, y=Tokens, group=1)) + geom_line() + geom_point() +
scale_x_discrete(labels=c(seq(1789,2012,12)), breaks=seq(1789,2012,12) ) 


# Longest inaugural address: William Henry Harrison
tokenInfo[which.max(tokenInfo$Tokens),] ## Text Types Tokens Sentences Year President ## 1841-Harrison 1841-Harrison 1893 9178 210 1841 Harrison ## FirstName ## 1841-Harrison William Henry ## Tools for handling corpus objects ### Adding two corpus objects together The + operator provides a simple method for concatenating two corpus objects. If they contain different sets of document-level variables, these will be stitched together in a fashion that guarantees that no information is lost. Corpus-level medata data is also concatenated. library(quanteda) mycorpus1 <- corpus(data_char_inaugural[1:5], note = "First five inaug speeches.") ## Warning in corpus.character(data_char_inaugural[1:5], note = "First five ## inaug speeches."): Argument note not used. mycorpus2 <- corpus(data_char_inaugural[53:57], note = "Last five inaug speeches.") ## Warning in corpus.character(data_char_inaugural[53:57], note = "Last five ## inaug speeches."): Argument note not used. mycorpus3 <- mycorpus1 + mycorpus2 summary(mycorpus3) ## Corpus consisting of 10 documents. ## ## Text Types Tokens Sentences ## 1789-Washington 626 1540 23 ## 1793-Washington 96 147 4 ## 1797-Adams 826 2584 37 ## 1801-Jefferson 716 1935 41 ## 1805-Jefferson 804 2381 45 ## 1997-Clinton 773 2451 111 ## 2001-Bush 622 1810 97 ## 2005-Bush 772 2325 100 ## 2009-Obama 939 2729 110 ## 2013-Obama 814 2335 88 ## ## Source: Combination of corpuses mycorpus1 and mycorpus2 ## Created: Mon Feb 13 18:57:22 2017 ## Notes: ### subsetting corpus objects There is a method of the corpus_subset() function defined for corpus objects, where a new corpus can be extracted based on logical conditions applied to docvars: summary(corpus_subset(data_corpus_inaugural, Year > 1990)) ## Corpus consisting of 7 documents. ## ## Text Types Tokens Sentences Year President FirstName ## 1993-Clinton 644 1837 81 1993 Clinton Bill ## 1997-Clinton 773 2451 111 1997 Clinton Bill ## 2001-Bush 622 1810 97 2001 Bush George W. ## 2005-Bush 772 2325 100 2005 Bush George W. ## 2009-Obama 939 2729 110 2009 Obama Barack ## 2013-Obama 814 2335 88 2013 Obama Barack ## 2017-Trump 582 1662 88 2017 Trump Donald J. ## ## Source: /home/paul/Dropbox/code/quanteda/* on x86_64 by paul ## Created: Fri Sep 12 12:41:17 2014 ## Notes: summary(corpus_subset(data_corpus_inaugural, President == "Adams")) ## Corpus consisting of 2 documents. ## ## Text Types Tokens Sentences Year President FirstName ## 1797-Adams 826 2584 37 1797 Adams John ## 1825-Adams 1004 3154 74 1825 Adams John Quincy ## ## Source: /home/paul/Dropbox/code/quanteda/* on x86_64 by paul ## Created: Fri Sep 12 12:41:17 2014 ## Notes: ## Exploring corpus texts The kwic function (KeyWord In Context) performs a search for a word and allows us to view the contexts in which it occurs: options(width = 200) kwic(data_corpus_inaugural, "terror") ## ## [1797-Adams, 1327] fraud or violence, by | terror | , intrigue, or venality ## [1933-Roosevelt, 112] nameless, unreasoning, unjustified | terror | which paralyzes needed efforts to ## [1941-Roosevelt, 289] seemed frozen by a fatalistic | terror | , we proved that this ## [1961-Kennedy, 868] alter that uncertain balance of | terror | that stays the hand of ## [1981-Reagan, 821] freeing all Americans from the | terror | of runaway living costs. ## [1997-Clinton, 1055] They fuel the fanaticism of | terror | . And they torment the ## [1997-Clinton, 1655] maintain a strong defense against | terror | and destruction. Our children ## [2009-Obama, 1646] advance their aims by inducing | terror | and slaughtering innocents, we kwic(data_corpus_inaugural, "terror", valuetype = "regex") ## ## [1797-Adams, 1327] fraud or violence, by | terror | , intrigue, or venality ## [1933-Roosevelt, 112] nameless, unreasoning, unjustified | terror | which paralyzes needed efforts to ## [1941-Roosevelt, 289] seemed frozen by a fatalistic | terror | , we proved that this ## [1961-Kennedy, 868] alter that uncertain balance of | terror | that stays the hand of ## [1961-Kennedy, 992] of science instead of its | terrors | . Together let us explore ## [1981-Reagan, 821] freeing all Americans from the | terror | of runaway living costs. ## [1981-Reagan, 2204] understood by those who practice | terrorism | and prey upon their neighbors ## [1997-Clinton, 1055] They fuel the fanaticism of | terror | . And they torment the ## [1997-Clinton, 1655] maintain a strong defense against | terror | and destruction. Our children ## [2009-Obama, 1646] advance their aims by inducing | terror | and slaughtering innocents, we ## [2017-Trump, 1119] civilized world against radical Islamic | terrorism | , which we will eradicate kwic(data_corpus_inaugural, "communist*") ## ## [1949-Truman, 838] the actions resulting from the | Communist | philosophy are a threat to ## [1961-Kennedy, 519] -- not because the | Communists | may be doing it, In the above summary, Year and President are variables associated with each document. We can access such variables with the docvars() function. # inspect the document-level variables head(docvars(data_corpus_inaugural)) ## Year President FirstName ## 1789-Washington 1789 Washington George ## 1793-Washington 1793 Washington George ## 1797-Adams 1797 Adams John ## 1801-Jefferson 1801 Jefferson Thomas ## 1805-Jefferson 1805 Jefferson Thomas ## 1809-Madison 1809 Madison James # inspect the corpus-level metadata metacorpus(data_corpus_inaugural) ##$source
## [1] "/home/paul/Dropbox/code/quanteda/* on x86_64 by paul"
##
## $created ## [1] "Fri Sep 12 12:41:17 2014" ## ##$notes
## NULL
##
## $citation ## NULL More corpora are available from the quantedaData package. # Extracting Features from a Corpus In order to perform statistical analysis such as document scaling, we must extract a matrix associating values for certain features with each document. In quanteda, we use the dfm function to produce such a matrix. “dfm” is short for document-feature matrix, and always refers to documents in rows and “features” as columns. We fix this dimensional orientation because is is standard in data analysis to have a unit of analysis as a row, and features or variables pertaining to each unit as columns. We call them “features” rather than terms, because features are more general than terms: they can be defined as raw terms, stemmed terms, the parts of speech of terms, terms after stopwords have been removed, or a dictionary class to which a term belongs. Features can be entirely general, such as ngrams or syntactic dependencies, and we leave this open-ended. ## Tokenizing texts To simply tokenize a text, quanteda provides a powerful command called tokenize(). This produces an intermediate object, consisting of a list of tokens in the form of character vectors, where each element of the list corresponds to an input document. tokenize() is deliberately conservative, meaning that it does not remove anything from the text unless told to do so. txt <- c(text1 = "This is$10 in 999 different ways,\n up and down; left and right!",
text2 = "@kenbenoit working: on #quanteda 2day\t4ever, http://textasdata.com?page=123.")
tokenize(txt)
## tokenizedTexts from 2 documents.
## text1 :
##  [1] "This"      "is"        "$" "10" "in" "999" "different" "ways" "," "up" "and" "down" ";" "left" "and" "right" ## [17] "!" ## ## text2 : ## [1] "@kenbenoit" "working" ":" "on" "#quanteda" "2day" "4ever" "," "http" ":" "/" ## [12] "/" "textasdata.com" "?" "page" "=" "123" "." tokenize(txt, removeNumbers = TRUE, removePunct = TRUE) ## tokenizedTexts from 2 documents. ## text1 : ## [1] "This" "is" "in" "different" "ways" "up" "and" "down" "left" "and" "right" ## ## text2 : ## [1] "@kenbenoit" "working" "on" "#quanteda" "2day" "4ever" "http" "textasdata.com" "page" tokenize(txt, removeNumbers = FALSE, removePunct = TRUE) ## tokenizedTexts from 2 documents. ## text1 : ## [1] "This" "is" "10" "in" "999" "different" "ways" "up" "and" "down" "left" "and" "right" ## ## text2 : ## [1] "@kenbenoit" "working" "on" "#quanteda" "2day" "4ever" "http" "textasdata.com" "page" "123" tokenize(txt, removeNumbers = TRUE, removePunct = FALSE) ## tokenizedTexts from 2 documents. ## text1 : ## [1] "This" "is" "$"         "in"        "different" "ways"      ","         "up"        "and"       "down"      ";"         "left"      "and"       "right"     "!"
##
## text2 :
##  [1] "@kenbenoit"     "working"        ":"              "on"             "#quanteda"      "2day"           "4ever"          ","              "http"           ":"              "/"
## [12] "/"              "textasdata.com" "?"              "page"           "="              "."
tokenize(txt, removeNumbers = FALSE, removePunct = FALSE)
## tokenizedTexts from 2 documents.
## text1 :
##  [1] "This"      "is"        "$" "10" "in" "999" "different" "ways" "," "up" "and" "down" ";" "left" "and" "right" ## [17] "!" ## ## text2 : ## [1] "@kenbenoit" "working" ":" "on" "#quanteda" "2day" "4ever" "," "http" ":" "/" ## [12] "/" "textasdata.com" "?" "page" "=" "123" "." tokenize(txt, removeNumbers = FALSE, removePunct = FALSE, removeSeparators = FALSE) ## tokenizedTexts from 2 documents. ## text1 : ## [1] "This" " " "is" " " "$"         "10"        " "         "in"        " "         "999"       " "         "different" " "         "ways"      ","         "\n"
## [17] " "         "up"        " "         "and"       " "         "down"      ";"         " "         "left"      " "         "and"       " "         "right"     "!"
##
## text2 :
##  [1] "@kenbenoit"     " "              "working"        ":"              " "              "on"             " "              "#quanteda"      " "              "2day"           "\t"
## [12] "4ever"          ","              " "              "http"           ":"              "/"              "/"              "textasdata.com" "?"              "page"           "="
## [23] "123"            "."

We also have the option to tokenize characters:

tokenize("Great website: http://textasdata.com?page=123.", what = "character")
## tokenizedTexts from 1 document.
## Component 1 :
##  [1] "G" "r" "e" "a" "t" "w" "e" "b" "s" "i" "t" "e" ":" "h" "t" "t" "p" ":" "/" "/" "t" "e" "x" "t" "a" "s" "d" "a" "t" "a" "." "c" "o" "m" "?" "p" "a" "g" "e" "=" "1" "2" "3" "."
tokenize("Great website: http://textasdata.com?page=123.", what = "character",
removeSeparators = FALSE)
## tokenizedTexts from 1 document.
## Component 1 :
##  [1] "G" "r" "e" "a" "t" " " "w" "e" "b" "s" "i" "t" "e" ":" " " "h" "t" "t" "p" ":" "/" "/" "t" "e" "x" "t" "a" "s" "d" "a" "t" "a" "." "c" "o" "m" "?" "p" "a" "g" "e" "=" "1" "2" "3" "."

and sentences:

# sentence level
tokenize(c("Kurt Vongeut said; only assholes use semi-colons.",
"Today is Thursday in Canberra:  It is yesterday in London.",
"En el caso de que no puedas ir con ellos, ¿quieres ir con nosotros?"),
what = "sentence")
## tokenizedTexts from 3 documents.
## Component 1 :
## [1] "Kurt Vongeut said; only assholes use semi-colons."
##
## Component 2 :
## [1] "Today is Thursday in Canberra:  It is yesterday in London."
##
## Component 3 :
## [1] "En el caso de que no puedas ir con ellos, ¿quieres ir con nosotros?"

## Constructing a document-frequency matrix

Tokenizing texts is an intermediate option, and most users will want to skip straight to constructing a document-feature matrix. For this, we have a Swiss-army knife function, called dfm(), which performs tokenization and tabulates the extracted features into a matrix of documents by features. Unlike the conservative approach taken by tokenize(), the dfm() function applies certain options by default, such as toLower() – a separate function for lower-casing texts – and removes punctuation. All of the options to tokenize() can be passed to dfm(), however.

myCorpus <- corpus_subset(data_corpus_inaugural, Year > 1990)

# make a dfm
myDfm <- dfm(myCorpus)
myDfm[, 1:5]
## Document-feature matrix of: 7 documents, 5 features (0% sparse).
## 7 x 5 sparse Matrix of class "dfmSparse"
##               features
## docs           my fellow citizens   , today
##   1993-Clinton  7      5        2 139    10
##   1997-Clinton  6      7        7 131     5
##   2001-Bush     3      1        9 110     2
##   2005-Bush     2      3        6 120     3
##   2009-Obama    2      1        1 130     6
##   2013-Obama    3      3        6  99     4
##   2017-Trump    1      1        4  96     4

Other options for a dfm() include removing stopwords, and stemming the tokens.

# make a dfm, removing stopwords and applying stemming
myStemMat <- dfm(myCorpus, remove = stopwords("english"), stem = TRUE, removePunct = TRUE)
myStemMat[, 1:5]
## Document-feature matrix of: 7 documents, 5 features (17.1% sparse).
## 7 x 5 sparse Matrix of class "dfmSparse"
##               features
## docs           fellow citizen today celebr mysteri
##   1993-Clinton      5       2    10      4       1
##   1997-Clinton      7       8     6      1       0
##   2001-Bush         1      10     2      0       0
##   2005-Bush         3       7     3      2       0
##   2009-Obama        1       1     6      2       0
##   2013-Obama        3       8     6      1       0
##   2017-Trump        1       4     5      3       1

The option remove provides a list of tokens to be ignored. Most users will supply a list of pre-defined “stop words”, defined for numerous languages, accessed through the stopwords() function:

head(stopwords("english"), 20)
##  [1] "i"          "me"         "my"         "myself"     "we"         "our"        "ours"       "ourselves"  "you"        "your"       "yours"      "yourself"   "yourselves" "he"         "him"
## [16] "his"        "himself"    "she"        "her"        "hers"
##  [1] "и"   "в"   "во"  "не"  "что" "он"  "на"  "я"   "с"   "со"
##  [1] "فى"  "في"  "كل"  "لم"  "لن"  "له"  "من"  "هو"  "هي"  "قوة"

### Viewing the document-frequency matrix

The dfm can be inspected in the Enviroment pane in RStudio, or by calling R’s View function. Calling plot on a dfm will display a wordcloud using the wordcloud package

mydfm <- dfm(data_char_ukimmig2010, remove = c("will", stopwords("english")),
removePunct = TRUE)
mydfm
## Document-feature matrix of: 9 documents, 1,547 features (83.8% sparse).

To access a list of the most frequently occurring features, we can use topfeatures():

topfeatures(mydfm, 20)  # 20 top words
## immigration     british      people      asylum     britain          uk      system  population     country         new  immigrants      ensure       shall citizenship      social    national
##          66          37          35          29          28          27          27          21          20          19          17          17          17          16          14          14
##         bnp     illegal        work     percent
##          13          13          13          12

Plotting a word cloud is done using textplot_wordcloud(), for a dfm class object:

set.seed(20)
textplot_wordcloud(dfm_trim(mydfm, min_count = 6))

The plot.dfm() method passes arguments through to wordcloud() from the wordcloud package, and can prettify the plot using the same arguments:

set.seed(100)
textplot_wordcloud(mydfm, min.freq = 6, random.order = FALSE,
rot.per = .25,
colors = RColorBrewer::brewer.pal(8,"Dark2"))

### Grouping documents by document variable

Often, we are interested in analysing how texts differ according to substantive factors which may be encoded in the document variables, rather than simply by the boundaries of the document files. We can group documents which share the same value for a document variable when creating a dfm:

byPartyDfm <- dfm(data_corpus_irishbudget2010, groups = "party", remove = stopwords("english"), removePunct = TRUE)

We can sort this dfm, and inspect it:

sort(byPartyDfm)[, 1:10]
## Warning: 'sort.dfm' is deprecated.
## See help("Deprecated")
## Document-feature matrix of: 5 documents, 10 features (0% sparse).
## 5 x 10 sparse Matrix of class "dfmSparse"
##        features
## docs    will people budget government public minister tax economy pay jobs
##   FF     212     23     44         47     65       11  60      37  41   41
##   FG      93     78     71         61     47       62  11      20  29   17
##   Green   59     15     26         19      4        4  11      16   4   15
##   LAB     89     69     66         36     32       54  47      37  24   20
##   SF     104     81     53         73     31       39  34      50  24   27

Note that the most frequently occurring feature is “will”, a word usually on English stop lists, but one that is not included in quanteda’s built-in English stopword list.

### Grouping words by dictionary or equivalence class

For some applications we have prior knowledge of sets of words that are indicative of traits we would like to measure from the text. For example, a general list of positive words might indicate positive sentiment in a movie review, or we might have a dictionary of political terms which are associated with a particular ideological stance. In these cases, it is sometimes useful to treat these groups of words as equivalent for the purposes of analysis, and sum their counts into classes.

For example, let’s look at how words associated with terrorism and words associated with the economy vary by President in the inaugural speeches corpus. From the original corpus, we select Presidents since Clinton:

recentCorpus <- corpus_subset(data_corpus_inaugural, Year > 1991)

Now we define a demonstration dictionary:

myDict <- dictionary(list(terror = c("terrorism", "terrorists", "threat"),
economy = c("jobs", "business", "grow", "work")))

We can use the dictionary when making the dfm:

byPresMat <- dfm(recentCorpus, dictionary = myDict)
byPresMat
## Document-feature matrix of: 7 documents, 2 features (0% sparse).
## 7 x 2 sparse Matrix of class "dfmSparse"
##               features
## docs           terror economy
##   1993-Clinton      0       8
##   1997-Clinton      1       8
##   2001-Bush         0       4
##   2005-Bush         1       6
##   2009-Obama        1      10
##   2013-Obama        1       6
##   2017-Trump        1       5

The constructor function dictionary() also works with two common “foreign” dictionary formats: the LIWC and Provalis Research’s Wordstat format. For instance, we can load the LIWC and apply this to the Presidential inaugural speech corpus:

liwcdict <- dictionary(file = "~/Dropbox/QUANTESS/dictionaries/LIWC/LIWC2001_English.dic",
format = "LIWC")
liwcdfm <- dfm(data_char_inaugural[52:57], dictionary = liwcdict, verbose = FALSE)
liwcdfm[, 1:10]

# Further Examples

## Similarities between texts

presDfm <- dfm(corpus_subset(data_corpus_inaugural, Year>1980),
remove = stopwords("english"),
stem = TRUE, removePunct = TRUE)
obamaSimil <- textstat_simil(presDfm, c("2009-Obama" , "2013-Obama"), n = NULL,
margin = "documents", method = "cosine")
dotchart(as.list(obamaSimil)$"2009-Obama", xlab = "Cosine similarity") We can use these distances to plot a dendrogram, clustering presidents: data(SOTUCorpus, package="quantedaData") presDfm <- dfm(corpus_subset(SOTUCorpus, Date > as.Date("1960-01-01")), verbose = FALSE, stem = TRUE, remove = stopwords("english"), removePunct = TRUE) presDfm <- dfm_trim(presDfm, min_count=5, min_docfreq=3) # hierarchical clustering - get distances on normalized dfm presDistMat <- dist(as.matrix(weight(presDfm, "relFreq"))) # hiarchical clustering the distance object presCluster <- hclust(presDistMat) # label with document names presCluster$labels <- docnames(presDfm)
# plot as a dendrogram
plot(presCluster, xlab = "", sub = "", main = "Euclidean Distance on Normalized Token Frequency")

We can also look at term similarities:

sim <- similarity(presDfm, c("fair", "health", "terror"), method = "cosine", margin = "features", n = 10)
print(sim, digits = 2)
## similarity Matrix:
## $fair ## economi begin jefferson author faith call struggl best creat courag ## 0.91 0.91 0.90 0.89 0.89 0.86 0.85 0.84 0.83 0.83 ## ##$terror
##    potenti  adversari commonplac     miracl     racial     bounti     martin      dream      polit   guarante
##       0.90       0.90       0.89       0.89       0.89       0.89       0.89       0.86       0.85       0.85
##
## $health ## shape generat wrong common knowledg planet task demand eye defin ## 0.90 0.90 0.89 0.89 0.89 0.88 0.87 0.87 0.87 0.86 ## Scaling document positions We have a lot of development work to do on the textmodel() function, but here is a demonstration of unsupervised document scaling comparing the “wordfish” model to scaling from correspondence analysis: # make prettier document names docnames(data_corpus_irishbudget2010) <- paste(docvars(data_corpus_irishbudget2010, "name"), docvars(data_corpus_irishbudget2010, "party")) ieDfm <- dfm(data_corpus_irishbudget2010, verbose = FALSE) wf <- textmodel(ieDfm, model = "wordfish", dir=c(2,1)) wca <- textmodel(ieDfm, model = "ca") # plot the results plot(wf@theta, -1*wca$rowcoord[,1],
xlab="Wordfish theta-hat", ylab="CA dim 1 coordinate", pch=19)
text(wf@theta, -1*wca$rowcoord[,1], docnames(ieDfm), cex=.8, pos=1) abline(lm(-1*wca$rowcoord[,1] ~ wf@theta), col="grey50", lty="dotted")

## Topic models

quantdfm <- dfm(data_corpus_irishbudget2010, verbose = FALSE,
remove = c("will", stopwords("english")))

if (require(topicmodels)) {
myLDAfit20 <- LDA(convert(quantdfm, to = "topicmodels"), k = 20)
get_terms(myLDAfit20, 5)
topics(myLDAfit20, 3)
}
## [3,]         14        19         12         7       11       11          11          15          18        12             5         10          19          11