Variable Types

Win-Vector LLC

2017-10-16

‘vtreat’ is a package that prepares arbitrary data frames into clean data frames that are ready for analysis (usually supervised learning). A clean data frame:

To effect this encoding ‘vtreat’ replaces original variables or columns with new derived variables. In this note we will use variables and columns as interchangeable concepts. This note describes the current family of ‘vtreat’ derived variable types.

‘vtreat’ usage splits into three main cases:

In all cases vtreat variable names are built by appending a notation onto the original user supplied column name. In all cases the easiest way to examine the derived variables is to look at the scoreFrame component of the returned treatment plan.

We will outline each of these situations below:

When the target to predict is categorical

An example categorical variable treatment is demonstrated below:

library(vtreat)
dTrainC <- data.frame(x=c('a','a','a','b','b',NA),
   z=c(1,2,3,4,NA,6),y=c(FALSE,FALSE,TRUE,FALSE,TRUE,TRUE),
   stringsAsFactors = FALSE)
treatmentsC <- designTreatmentsC(dTrainC,colnames(dTrainC),'y',TRUE)
## [1] "designing treatments Mon Oct 16 19:21:08 2017"
## [1] "designing treatments Mon Oct 16 19:21:08 2017"
## [1] " have level statistics Mon Oct 16 19:21:08 2017"
## [1] "design var x Mon Oct 16 19:21:08 2017"
## [1] "design var z Mon Oct 16 19:21:08 2017"
## [1] " scoring treatments Mon Oct 16 19:21:08 2017"
## [1] "have treatment plan Mon Oct 16 19:21:08 2017"
## [1] "rescoring complex variables Mon Oct 16 19:21:08 2017"
## [1] "done rescoring complex variables Mon Oct 16 19:21:08 2017"
scoreColsToPrint <- c('origName','varName','code','rsq','sig','extraModelDegrees')
print(treatmentsC$scoreFrame[,scoreColsToPrint])
##   origName   varName  code        rsq       sig extraModelDegrees
## 1        x  x_lev_NA   lev 0.19087450 0.2076623                 0
## 2        x x_lev_x.a   lev 0.08170417 0.4097258                 0
## 3        x x_lev_x.b   lev 0.00000000 1.0000000                 0
## 4        x    x_catP  catP 0.24340634 0.1547700                 2
## 5        x    x_catB  catB 0.05070201 0.5160763                 2
## 6        z   z_clean clean 0.25792985 0.1429977                 0
## 7        z   z_isBAD isBAD 0.19087450 0.2076623                 0

For each user supplied variable or column (in this case x and z) ‘vtreat’ proposes derived or treated variables (in this case x_lev_x.a, x_lev_x.b, x_catP, x_catB, z_clean, and z_isBAD). The mapping from original variable name to derived variable name is given by comparing the columns origName and varName. One can map facts about the new variables back to the original variables as follows:

# Build a map from vtreat names back to reasonable display names
vmap <- as.list(treatmentsC$scoreFrame$origName)
names(vmap) <- treatmentsC$scoreFrame$varName
print(vmap['x_catB'])
## $x_catB
## [1] "x"
# Map significances back to original variables
aggregate(sig~origName,data=treatmentsC$scoreFrame,FUN=min)
##   origName       sig
## 1        x 0.1547700
## 2        z 0.1429977

In the scoreFrame the sig column is the significance of the single variable logistic regression using the named variable (plus a constant term), and the rsq column is the “pseudo-rsquared” or portion of deviance explained (please see here for some notes).

Essentially a derived variable name is built by concatenating an original variable name and a treatment type (also recorded in the code column for convenience). The codes give the different ‘vtreat’ variable types (or really meanings, as all derived variables are numeric).

For categorical targets the possible variable types are as follows:

When the target to predict is numeric

An example numeric variable treatment is demonstrated below:

library(vtreat)
dTrainN <- data.frame(x=c('a','a','a','b','b',NA),
   z=c(1,2,3,4,NA,6),y=as.numeric(c(FALSE,FALSE,TRUE,FALSE,TRUE,TRUE)),
   stringsAsFactors = FALSE)
treatmentsN <- designTreatmentsN(dTrainN,colnames(dTrainN),'y')
## [1] "designing treatments Mon Oct 16 19:21:08 2017"
## [1] "designing treatments Mon Oct 16 19:21:08 2017"
## [1] " have level statistics Mon Oct 16 19:21:08 2017"
## [1] "design var x Mon Oct 16 19:21:08 2017"
## [1] "design var z Mon Oct 16 19:21:08 2017"
## [1] " scoring treatments Mon Oct 16 19:21:08 2017"
## [1] "have treatment plan Mon Oct 16 19:21:08 2017"
## [1] "rescoring complex variables Mon Oct 16 19:21:08 2017"
## [1] "done rescoring complex variables Mon Oct 16 19:21:08 2017"
print(treatmentsN$scoreFrame[,scoreColsToPrint])
##   origName   varName  code       rsq       sig extraModelDegrees
## 1        x  x_lev_NA   lev 0.2000000 0.3739010                 0
## 2        x x_lev_x.a   lev 0.1111111 0.5185185                 0
## 3        x x_lev_x.b   lev 0.0000000 1.0000000                 0
## 4        x    x_catP  catP 0.2857143 0.2745766                 2
## 5        x    x_catN  catN 0.1052632 0.5304117                 2
## 6        x    x_catD  catD 0.1111111 0.5185185                 2
## 7        z   z_clean clean 0.3045045 0.2562868                 0
## 8        z   z_isBAD isBAD 0.2000000 0.3739010                 0

The treatment of numeric targets is similar to that of categorical targets. In the numeric case the possible derived variable types are:

Note: for categorical targets we don’t need cat\_D variables as this information is already encoded in cat\_B variables.

In the scoreFrame the sig column is the significance of the single variable linear regression using the named variable (plus a constant term), and the rsq column is the “rsquared” or portion of variance explained (please see here) for some notes).

When there is no supplied target to predict

An example “no target” variable treatment is demonstrated below:

library(vtreat)
dTrainZ <- data.frame(x=c('a','a','a','b','b',NA),
   z=c(1,2,3,4,NA,6),
   stringsAsFactors = FALSE)
treatmentsZ <- designTreatmentsZ(dTrainZ,colnames(dTrainZ))
## [1] "designing treatments Mon Oct 16 19:21:08 2017"
## [1] "designing treatments Mon Oct 16 19:21:08 2017"
## [1] " have level statistics Mon Oct 16 19:21:08 2017"
## [1] "design var x Mon Oct 16 19:21:08 2017"
## [1] "design var z Mon Oct 16 19:21:08 2017"
## [1] " scoring treatments Mon Oct 16 19:21:08 2017"
## [1] "have treatment plan Mon Oct 16 19:21:08 2017"
print(treatmentsZ$scoreFrame[, c('origName','varName','code','extraModelDegrees')])
##   origName   varName  code extraModelDegrees
## 1        x  x_lev_NA   lev                 0
## 2        x x_lev_x.a   lev                 0
## 3        x x_lev_x.b   lev                 0
## 4        x    x_catP  catP                 2
## 5        z   z_clean clean                 0
## 6        z   z_isBAD isBAD                 0

Note: because there is no user supplied target the scoreFrame significance columns are not meaningful, and are populated only for regularity of code interface. Also indicator variables are only formed by designTreatmentsZ for vtreat 0.5.28 or newer. Beyond that the no-target treatments are similar to the earlier treatments. Possible derived variable types in this case are:

Restricting to Specific Variable Types

Both designTreatmentsX and prepare functions take an argument called codeRestriction that restricts the type of variable that is created. For example, you may not want to create catP and catD variables for a regression problem.

dTrainN <- data.frame(x=c('a','a','a','b','b',NA),
   z=c(1,2,3,4,NA,6),y=as.numeric(c(FALSE,FALSE,TRUE,FALSE,TRUE,TRUE)),
   stringsAsFactors = FALSE)

treatmentsN <- designTreatmentsN(dTrainN,colnames(dTrainN),'y',
                                 codeRestriction = c('lev', 
                                                      'catN',
                                                      'clean',
                                                      'isBAD'),
                                 verbose=FALSE)

# no catP or catD variables
print(treatmentsN$scoreFrame[,scoreColsToPrint])
##   origName   varName  code       rsq       sig extraModelDegrees
## 1        x  x_lev_NA   lev 0.2000000 0.3739010                 0
## 2        x x_lev_x.a   lev 0.1111111 0.5185185                 0
## 3        x x_lev_x.b   lev 0.0000000 1.0000000                 0
## 4        x    x_catN  catN 0.1052632 0.5304117                 2
## 5        z   z_clean clean 0.3045045 0.2562868                 0
## 6        z   z_isBAD isBAD 0.2000000 0.3739010                 0

Conversely, even if you have created a treatment plan for a particular type of variable, you may subsequently decide not to use it. For example, perhaps you only want to use indicator variables and not the catN variable for modeling. You can use codeRestriction in prepare().

dTreated = prepare(treatmentsN, dTrainN, 
                   codeRestriction = c('lev','clean', 'isBAD'))

# no catN variables
head(dTreated)
##   x_lev_NA x_lev_x.a x_lev_x.b z_clean z_isBAD y
## 1        0         1         0     1.0       0 0
## 2        0         1         0     2.0       0 0
## 3        0         1         0     3.0       0 1
## 4        0         0         1     4.0       0 0
## 5        0         0         1     3.2       1 1
## 6        1         0         0     6.0       0 1

varRestriction works similarly, only you must list the explicit variables to use. See the example below.

Overall

Variables that “do not move” (don’t take on at least two values during treatment design) or don’t achieve at least a minimal significance are suppressed. The catB/catN variables are essentially single variable models and are very useful for re-encoding categorical variables that take on a very large number of values (such as zip-codes).

The intended use of ‘vtreat’ is as follows:

‘vtreat’ attempts to compute “out of sample” significances for each variable effect ( the sig column in scoreFrame) through cross-validation techniques.

‘vtreat’ is primarily intended to be “y-aware” processing. Of particular interest is using vtreat::prepare() with scale=TRUE which tries to put most columns in ‘y-effect’ units. This can be an important pre-processing step before attempting dimension reduction (such as principal components methods).

The vtreat user should pick which sorts of variables they are want and also filter on estimated significance. Doing this looks like the following:

dTrainN <- data.frame(x=c('a','a','a','b','b',NA),
   z=c(1,2,3,4,NA,6),y=as.numeric(c(FALSE,FALSE,TRUE,FALSE,TRUE,TRUE)),
   stringsAsFactors = FALSE)
treatmentsN <- designTreatmentsN(dTrainN,colnames(dTrainN),'y',
                                  codeRestriction = c('lev', 
                                                      'catN',
                                                      'clean',
                                                      'isBAD'),
                                 verbose=FALSE)
print(treatmentsN$scoreFrame[,scoreColsToPrint])
##   origName   varName  code       rsq       sig extraModelDegrees
## 1        x  x_lev_NA   lev 0.2000000 0.3739010                 0
## 2        x x_lev_x.a   lev 0.1111111 0.5185185                 0
## 3        x x_lev_x.b   lev 0.0000000 1.0000000                 0
## 4        x    x_catN  catN 0.1131222 0.5145190                 2
## 5        z   z_clean clean 0.3045045 0.2562868                 0
## 6        z   z_isBAD isBAD 0.2000000 0.3739010                 0
pruneSig <- 1.0 # don't filter on significance for this tiny example
vScoreFrame <- treatmentsN$scoreFrame
varsToUse <- vScoreFrame$varName[(vScoreFrame$sig<=pruneSig)]
print(varsToUse)
## [1] "x_lev_NA"  "x_lev_x.a" "x_lev_x.b" "x_catN"    "z_clean"   "z_isBAD"
origVarNames <- sort(unique(vScoreFrame$origName[vScoreFrame$varName %in% varsToUse]))
print(origVarNames)
## [1] "x" "z"
# prepare a treated data frame using only the "significant" variables
dTreated = prepare(treatmentsN, dTrainN, 
                   varRestriction = varsToUse)

head(dTreated)
##   x_lev_NA x_lev_x.a x_lev_x.b     x_catN z_clean z_isBAD y
## 1        0         1         0 -0.1666667     1.0       0 0
## 2        0         1         0 -0.1666667     2.0       0 0
## 3        0         1         0 -0.1666667     3.0       0 1
## 4        0         0         1  0.0000000     4.0       0 0
## 5        0         0         1  0.0000000     3.2       1 1
## 6        1         0         0  0.5000000     6.0       0 1

We strongly suggest using the standard variables coded as ‘lev’, ‘clean’, and ‘isBad’; and the “y aware” variables coded as ‘catN’ and ‘catB’. The non sub-model variables (‘catP’ and ‘catD’) can be useful (possibly as interactions or guards on the corresponding ‘catN’ and ‘catB’ variables) but also encode distributional facts about the data that may or may not be appropriate depending on your problem domain.

When displaying variables to end users we suggest using the original names and the min significance seen on any derived variable:

origVarNames <- sort(unique(vScoreFrame$origName[vScoreFrame$varName %in% varsToUse]))
print(origVarNames)
## [1] "x" "z"
origVarSigs <- vScoreFrame[vScoreFrame$varName %in% varsToUse,]
aggregate(sig~origName,data=origVarSigs,FUN=min)
##   origName       sig
## 1        x 0.3739010
## 2        z 0.2562868