This vignette describes various ways of summarizing `emmGrid`

objects.

`emmGrid`

objects, described`summary()`

,`confint()`

, and`test()`

- Back-transforming to response scale (See also the “transformations” vignette)
- Multiplicity adjustments
- Using “by” variables
- Joint (omnibus) tests
- Testing equivalence, noninferiority, nonsuperiority
- Graphics (in “basics” vignette)

The `ref_grid()`

and `emmeans()`

functions are introduced in
the “Basics” vignette. These functions, and a few related ones,
return an object of class `emmGrid`

:

```
pigs.lm <- lm(log(conc) ~ source + factor(percent), data = pigs)
pigs.rg <- ref_grid(pigs.lm)
class(pigs.rg)
```

```
## [1] "emmGrid"
## attr(,"package")
## [1] "emmeans"
```

```
pigs.emm.s <- emmeans(pigs.rg, "source")
class(pigs.emm.s)
```

```
## [1] "emmGrid"
## attr(,"package")
## [1] "emmeans"
```

If you simply show these objects, you get different-looking results:

```
pigs.rg
```

```
## 'emmGrid' object with variables:
## source = fish, soy, skim
## percent = 9, 12, 15, 18
## Transformation: "log"
```

```
pigs.emm.s
```

```
## source emmean SE df lower.CL upper.CL
## fish 3.39 0.0367 23 3.32 3.47
## soy 3.67 0.0374 23 3.59 3.74
## skim 3.80 0.0394 23 3.72 3.88
##
## Results are averaged over the levels of: percent
## Results are given on the log (not the response) scale.
## Confidence level used: 0.95
```

This is based on guessing what users most need to see when displaying the object. You can override these defaults; for example to just see a quick summary of what is there, do

```
str(pigs.emm.s)
```

```
## 'emmGrid' object with variables:
## source = fish, soy, skim
## Transformation: "log"
```

The most important method for `emmGrid`

objects is `summary()`

. It is used as the
default for displaying an `emmeans()`

result like `pigs.emm.s`

. This `summary()`

method for `emmGrid`

objects) actually produces a `data.frame`

, but with extra bells
and whistles:

```
class(summary(pigs.emm.s))
```

```
## [1] "summary_emm" "data.frame"
```

This can be useful to know because if you want to actually *use* `emmeans()`

results
in other computations, you should save its summary, and then you can access those
results just like you would access data in a data frame. The `emmGrid`

object itself
is not so accessible. There is a `print.summary_emm()`

function that is what
actually produces the output you see above – a data frame with extra
annotations.

`summary()`

and its relatives {#summary}As you may have gathered, the most important method for `emmGrid`

objects is `summary()`

.
It has a lot of options, and the detailed documentation via
`help("summary.emm")`

is worth a look.

Just `summary(<object>)`

by itself will produce a summary that varies somewhat
according to context. It does this by setting different defaults for the `infer`

argument, which consists of two logical values, specifying confidence intervals
and tests, respectively.
The summary of a newly made reference grid will show just estimates and
standard errors, but not confidence intervals or tests (that is, ```
infer =
c(FALSE, FALSE)
```

). The summary of an `emmeans()`

result, as we see above, will
have intervals, but no tests (i.e., `infer = c(TRUE, FALSE)`

); and the result of
a `contrast()`

call (see comparisons and contrasts) will
show test statistics and *P* values, but not intervals (i.e., ```
infer = c(FALSE,
TRUE)
```

). There are courtesy methods `confint()`

and `test()`

that just call
`summary()`

with the appropriate `infer`

setting; for example,

```
test(pigs.emm.s)
```

```
## source emmean SE df t.ratio p.value
## fish 3.39 0.0367 23 92.540 <.0001
## soy 3.67 0.0374 23 97.929 <.0001
## skim 3.80 0.0394 23 96.407 <.0001
##
## Results are averaged over the levels of: percent
## Results are given on the log (not the response) scale.
```

It is not particularly useful, though, to test these EMMs against the default of zero – which is why tests are not usually shown. It makes a lot more sense to test them against some target concentration, say 40. And suppose we want to do a one-sided test to see if the concentration is greater than 40. Remembering that the response is log-transformed in this model,

```
test(pigs.emm.s, null = log(40), side = ">")
```

```
## source emmean SE df null t.ratio p.value
## fish 3.39 0.0367 23 3.69 -8.026 1.0000
## soy 3.67 0.0374 23 3.69 -0.577 0.7153
## skim 3.80 0.0394 23 3.69 2.740 0.0058
##
## Results are averaged over the levels of: percent
## Results are given on the log (not the response) scale.
## P values are right-tailed
```

Transformations and link functions are supported an several ways in **emmeans**,
making this a complex topic worthy of its own vignette.
Here, we show just the most basic approach. Namely, specifying the argument
`type = "response"`

will cause the displayed results to be back-transformed
to the response scale, when a transformation or link function is incorporated
in the model. For example, let's try the preceding `test()`

call again:

```
test(pigs.emm.s, null = log(40), side = ">", type = "response")
```

```
## source response SE df null t.ratio p.value
## fish 29.8 1.09 23 40 -8.026 1.0000
## soy 39.1 1.47 23 40 -0.577 0.7153
## skim 44.6 1.75 23 40 2.740 0.0058
##
## Results are averaged over the levels of: percent
## P values are right-tailed
## Tests are performed on the log scale
```

Note what changes and what doesn't change. In the `test()`

call, we *still* use
the log of 40 as the null value; `null`

must always be specified on the
linear-prediction scale, in this case the log. In the output, the displayed
estimates, as well as the `null`

value, are shown back-transformed. As well, the
standard errors are altered (using the delta method). However, the *t* ratios
and *P* values are identical to the preceding results. That is, the tests
themselves are still conducted on the linear-predictor scale (as is noted in the
output).

Similar statements apply to confidence intervals on the response scale:

```
confint(pigs.emm.s, side = ">", level = .90, type = "response")
```

```
## source response SE df lower.CL upper.CL
## fish 29.8 1.09 23 28.4 Inf
## soy 39.1 1.47 23 37.3 Inf
## skim 44.6 1.75 23 42.3 Inf
##
## Results are averaged over the levels of: percent
## Confidence level used: 0.9
## Intervals are back-transformed from the log scale
```

With `side = ">"`

, a *lower* confidence limit is computed on the log scale, then
that limit is back-transformed to the response scale. (We have also illustrated
how to change the confidence level.)

Both tests and confidence intervals may be adjusted for simultaneous inference.
Such adjustments ensure that the confidence coefficient for a whole set of
intervals is at least the specified level, or to control for multiplicity
in a whole family of tests. This is done via the `adjust`

argument. For `ref_grid()`

and `emmeans()`

results, the default is ```
adjust =
"none"
```

. For most `contrast()`

results, `adjust`

is often something else,
depending on what type of contrasts are created. For example, pairwise
comparisons default to `adjust = "tukey"`

, i.e., the Tukey HSD method.
The `summary()`

function sometimes *changes* `adjust`

if it is inappropriate.
For example, with

```
confint(pigs.emm.s, adjust = "tukey")
```

```
## source emmean SE df lower.CL upper.CL
## fish 3.39 0.0367 23 3.30 3.49
## soy 3.67 0.0374 23 3.57 3.76
## skim 3.80 0.0394 23 3.70 3.90
##
## Results are averaged over the levels of: percent
## Results are given on the log (not the response) scale.
## Confidence level used: 0.95
## Conf-level adjustment: sidak method for 3 estimates
```

the adjustment is changed to the Sidak method because the Tukey adjustment is inappropriate unless you are doing pairwise comparisons.

####### {#adjmore}

An adjustment method that is usually appropriate is Bonferroni; however, it can
be quite conservative. Using `adjust = "mvt"`

is the closest to being the
“exact” all-around method “single-step” method, as it uses the multivariate *t*
distribution (and the **mvtnorm** package) with the same covariance structure as
the estimates to determine the adjustment. However, this comes at high
computational expense as the computations are done using simulation techniques.
For a large set of tests (and especially confidence intervals), the
computational lag becomes noticeable if not intolerable.

For tests, `adjust`

increases the *P* values over those otherwise obtained with
`adjust = "none"`

. Compare the following adjusted tests with the unadjusted ones
previously computed.

```
test(pigs.emm.s, null = log(40), side = ">", adjust = "bonferroni")
```

```
## source emmean SE df null t.ratio p.value
## fish 3.39 0.0367 23 3.69 -8.026 1.0000
## soy 3.67 0.0374 23 3.69 -0.577 1.0000
## skim 3.80 0.0394 23 3.69 2.740 0.0175
##
## Results are averaged over the levels of: percent
## Results are given on the log (not the response) scale.
## P value adjustment: bonferroni method for 3 tests
## P values are right-tailed
```

Sometimes you want to break a summary down into smaller pieces; for this
purpose, the `by`

argument in `summary()`

is useful. For example,

```
confint(pigs.rg, by = "source")
```

```
## source = fish:
## percent prediction SE df lower.CL upper.CL
## 9 3.22 0.0536 23 3.11 3.33
## 12 3.40 0.0493 23 3.30 3.50
## 15 3.44 0.0548 23 3.32 3.55
## 18 3.52 0.0547 23 3.41 3.63
##
## source = soy:
## percent prediction SE df lower.CL upper.CL
## 9 3.49 0.0498 23 3.39 3.60
## 12 3.67 0.0489 23 3.57 3.77
## 15 3.71 0.0507 23 3.61 3.82
## 18 3.79 0.0640 23 3.66 3.93
##
## source = skim:
## percent prediction SE df lower.CL upper.CL
## 9 3.62 0.0501 23 3.52 3.73
## 12 3.80 0.0494 23 3.70 3.90
## 15 3.84 0.0549 23 3.73 3.95
## 18 3.92 0.0646 23 3.79 4.06
##
## Results are given on the log (not the response) scale.
## Confidence level used: 0.95
```

If there is also an `adjust`

in force when `by`

variables are used, the
adjustment is made *separately* on each `by`

group; e.g., in the above, we would
be adjusting for sets of 4 intervals, not all 12 together.

There can be a `by`

specification in `emmeans()`

(or equivalently, a `|`

in the
formula); and if so, it is passed on to `summary()`

and used unless overridden
by another `by`

. Here are examples, not run:

```
emmeans(pigs.lm, ~ percent | source) ### same results as above
summary(.Last.value, by = percent) ### grouped the other way
```

Specifying `by = NULL`

will remove all grouping.

There is also a `simple`

argument for `contrast()`

that is in essence
the inverse of `by`

; the contrasts are run using everything *except* the
specified variables as `by`

variables. To illustrate, let's consider
the model for `pigs`

that includes the interaction (so that the levels
of one factor compare differently at levels of the other factor).

```
pigsint.lm <- lm(log(conc) ~ source * factor(percent), data = pigs)
pigsint.rg <- ref_grid(pigsint.lm)
contrast(pigsint.rg, "consec", simple = "percent")
```

```
## source = fish:
## contrast estimate SE df t.ratio p.value
## 12 - 9 0.1849 0.1061 17 1.742 0.2359
## 15 - 12 0.0045 0.1061 17 0.042 0.9999
## 18 - 15 0.0407 0.1061 17 0.383 0.9626
##
## source = soy:
## contrast estimate SE df t.ratio p.value
## 12 - 9 0.1412 0.0949 17 1.487 0.3591
## 15 - 12 -0.0102 0.0949 17 -0.108 0.9992
## 18 - 15 0.0895 0.1342 17 0.666 0.8571
##
## source = skim:
## contrast estimate SE df t.ratio p.value
## 12 - 9 0.2043 0.0949 17 2.152 0.1184
## 15 - 12 0.1398 0.1061 17 1.317 0.4519
## 18 - 15 0.1864 0.1424 17 1.309 0.4570
##
## Results are given on the log (not the response) scale.
## P value adjustment: mvt method for 3 tests
```

In fact, we may do *all* one-factor comparisons by specifying `simple = "each"`

.
This typically produces a lot of output, so use it with care.

From the above, we already know how to test individual results. For pairwise comparisons (details in the “comparisons” vignette), we might do

```
pigs.prs.s <- pairs(pigs.emm.s)
pigs.prs.s
```

```
## contrast estimate SE df t.ratio p.value
## fish - soy -0.273 0.0529 23 -5.153 0.0001
## fish - skim -0.402 0.0542 23 -7.428 <.0001
## soy - skim -0.130 0.0530 23 -2.442 0.0570
##
## Results are averaged over the levels of: percent
## Results are given on the log (not the response) scale.
## P value adjustment: tukey method for comparing a family of 3 estimates
```

But suppose we want an *omnibus* test that all these comparisons are zero.
Easy enough, using the `joint`

argument in `test`

(note: the `joint`

argument
is *not* available in `summary()`

; only in `test()`

):

```
test(pigs.prs.s, joint = TRUE)
```

```
## df1 df2 F.ratio p.value note
## 2 23 28.849 <.0001 d
##
## d: df1 reduced due to linear dependence
```

Notice that there are three comparisons, but only 2 d.f. for the test, as cautioned in the message.

The test produced with `joint = TRUE`

is a “type III” test (assuming the default
equal weights are used to obtain the EMMs). See more on these types of tests for
higher-order effects in the “interactions” vignette section on
contrasts.

####### {#joint_tests}

For convenience, there is also a `joint_tests()`

function that performs
joint tests of contrasts among each term in a model or `emmGrid`

object.

```
joint_tests(pigsint.rg)
```

```
## model term df1 df2 F.ratio p.value
## source 2 17 30.256 <.0001
## percent 3 17 8.214 0.0013
## source:percent 6 17 0.926 0.5011
```

The tests of main effects are of families of contrasts; those for interaction
effects are for interaction contrasts. These results are essentially the same as
a “Type-III ANOVA”, but may differ in situations where there are empty cells or
other non-estimability issues, or if generalizations are present such as unequal
weighting. (Another distinction is that sums of squares and mean squares are not
shown; that is because these really are tests of contrasts among predictions,
and they may or may not corresopond to model sums of squares.) One may use `by`

variables to obtain separate tables of joint tests.

The `delta`

argument in `summary()`

or `test()`

allows the user to
specify a threshold value to use in a test of equivalence, noninferiority,
or nonsuperiority. An equivalence test is kind of a backwards significance
test, where *P* values *decrease* as differences decrease below a specified
threshold value `delta`

.
The help page for `summary.emmGrid`

gives the details of
these tests.
Suppose in the present example, we consider two sources to be equivalent if they
are within 25% of each other. We can test this as follows:

```
test(pigs.prs.s, delta = log(1.25), adjust = "none")
```

```
## contrast estimate SE df t.ratio p.value
## fish - soy -0.273 0.0529 23 0.937 0.8209
## fish - skim -0.402 0.0542 23 3.308 0.9985
## soy - skim -0.130 0.0530 23 -1.765 0.0454
##
## Results are averaged over the levels of: percent
## Results are given on the log (not the response) scale.
## Statistics are tests of equivalence with a threshold of 0.22314
## P values are left-tailed
```

By our 25% standard, the *P* value is quite small for comparing soy and skim,
providing statistical evidence that their difference is enough smaller than the
threshold to consider them equivalent.

Graphical displays of `emmGrid`

objects are described in the
“basics” vignette