Post-run diagnostics and analyses

2021-09-24

In this tutorial, we assume that you successfully ran an MCMC on a network model, and that it is now time to have a critical look at the output of the run.

This tutorial will present how to check that the MCMC run went fine, but it will explain only very briefly how to check that the model is compatible with the observed data. More details about this important step is presented in the next vignette about posterior predictive checks.

This vignette is using the MCMC run from the Quick start tutorial. Please go back and run the code of that vignette to generate the MCMC data if you haven't already!

Reminder: what is the result format returned by run_mcmc()?

In the Quick Start tutorial, we generated the run object by running run <- run_mcmc(m, iter = 1000). The output from run_mcmc() is a well-behaved mcmc.list object as implemented in the coda package:

is(run, "mcmc.list")
## [1] TRUE

It makes it very easy to use the tools already available for this class of objects, such as those implemented in the packages coda, bayesplot, ggmcmc or MCMCvis.

In addition to all those existing tools, the isotracer package also adds some extra methods to easily calculate derived parameters from an mcmc.list. You will learn more about how to do this in the vignette Derived parameters.

General diagnostics

Trace plot

You should always run several chains when performing a Bayesian MCMC. Trace plots allow to get a feeling for:

If you are not satisfied with the traces, you need to run a longer run or maybe to tweak the settings of the Stan run.

This is the trace plot we obtained from the previous vignette:

plot(run)
# Note: the figure below only shows a few of the traceplots for vignette concision

plot of chunk unnamed-chunk-6

In this case, the chains have converged towards the same region, and the mixing looks good for all of them. There is no obvious problem visible in those traces.

Gelman and Rubin's convergence diagnostic

It can be useful to have a more formal test of the convergence of the chains. The Gelman and Rubin's convergence diagnostic is implemented in the coda package. We can run it with the coda::gelman.diag() function. See the coda documentation ?gelman.diag for more details about this diagnostic.

Let's have a look at the diagnostic for our chains:

library(coda)
run %>% gelman.diag()
## Potential scale reduction factors:
## 
##                          Point est. Upper C.I.
## eta                           1.005      1.006
## lambda_algae                  1.011      1.056
## lambda_daphnia                1.018      1.040
## lambda_NH4                    1.005      1.014
## upsilon_algae_to_daphnia      1.006      1.007
## upsilon_daphnia_to_NH4        1.000      1.002
## upsilon_NH4_to_algae          1.007      1.007
## zeta                          0.999      0.999
## 
## Multivariate psrf
## 
## 1.02

The diagnostic values should be very, very close to 1: it looks good in this case!

If some values were e.g. \(>1.05\), this would already be enough to cause us to wonder about the sampling quality.

Predicted trajectories

In order to check the quality of the model fit, the consistency between the parameter posteriors and the observed data can be checked by plotting the credible envelopes for the estimated trajectories along with the observed data points. This is called a posterior predictive check and is very important to check that the model can actually predict the observed data reasonably well. If the observed data cannot be satisfactorily predicted from the fitted model, then our model is not a good model of the data!

To do a posterior predictive check, the first step is to generate predictions for the model based on the MCMC posteriors with predict():

# From the Quick Start tutorial:
# 'm' is the network model we used when calling 'run <- run_mcmc(m, iter = 1000)'
predictions <- predict(m, run, probs = 0.95)

We can then visualize the predictions along with the observations with the plot() method:

plot(predictions)

plot of chunk unnamed-chunk-10

This plot enables to compare both the size and the proportion observations with the predictions.

Post-run analyses

Parameter estimates

The quickest way to get parameter estimates is to use the summary() function on the posterior:

run %>% summary()
## 
## Iterations = 501:1000
## Thinning interval = 1 
## Number of chains = 2 
## Sample size per chain = 500 
## 
## 1. Empirical mean and standard deviation for each variable,
##    plus standard error of the mean:
## 
##                             Mean       SD  Naive SE Time-series SE
## eta                      0.11406 0.035547 0.0011241      0.0015571
## lambda_algae             0.06508 0.041661 0.0013174      0.0015855
## lambda_daphnia           0.01615 0.017135 0.0005418      0.0006609
## lambda_NH4               0.04738 0.034692 0.0010971      0.0012590
## upsilon_algae_to_daphnia 0.07751 0.020345 0.0006434      0.0008842
## upsilon_daphnia_to_NH4   0.04938 0.005573 0.0001762      0.0001933
## upsilon_NH4_to_algae     0.32981 0.037233 0.0011774      0.0016169
## zeta                     0.27835 0.093675 0.0029622      0.0043836
## 
## 2. Quantiles for each variable:
## 
##                               2.5%      25%     50%     75%   97.5%
## eta                      0.0657811 0.089369 0.10718 0.13032 0.20284
## lambda_algae             0.0050176 0.031595 0.06048 0.09120 0.15554
## lambda_daphnia           0.0005427 0.004773 0.01107 0.02102 0.06422
## lambda_NH4               0.0034780 0.019911 0.03963 0.06652 0.13103
## upsilon_algae_to_daphnia 0.0478212 0.063323 0.07412 0.08671 0.12600
## upsilon_daphnia_to_NH4   0.0390770 0.045554 0.04922 0.05298 0.06127
## upsilon_NH4_to_algae     0.2520988 0.306209 0.32943 0.35404 0.40647
## zeta                     0.1552088 0.216713 0.25943 0.31607 0.52576

If you need to store those values, for example to plot them, you can assign the output of summary() to an object:

estimates <- run %>% summary()
names(estimates)
## [1] "statistics" "quantiles"  "start"      "end"        "thin"       "nchain"

The means and standard deviations are accessible in $statistics:

estimates$statistics
##                                Mean          SD     Naive SE Time-series SE
## eta                      0.11406312 0.035547223 0.0011241019   0.0015570816
## lambda_algae             0.06507779 0.041660899 0.0013174333   0.0015855312
## lambda_daphnia           0.01614987 0.017134558 0.0005418423   0.0006608822
## lambda_NH4               0.04737599 0.034691906 0.0010970544   0.0012589519
## upsilon_algae_to_daphnia 0.07751274 0.020345459 0.0006433799   0.0008841544
## upsilon_daphnia_to_NH4   0.04938313 0.005573325 0.0001762440   0.0001933378
## upsilon_NH4_to_algae     0.32981040 0.037233077 0.0011774133   0.0016169176
## zeta                     0.27835452 0.093674537 0.0029622489   0.0043836326

and the quantiles are in $quantiles:

estimates$quantiles
##                                  2.5%         25%        50%        75%      97.5%
## eta                      0.0657811165 0.089368605 0.10718424 0.13031932 0.20284455
## lambda_algae             0.0050175876 0.031594803 0.06048331 0.09120007 0.15554090
## lambda_daphnia           0.0005426863 0.004773352 0.01106789 0.02101735 0.06421725
## lambda_NH4               0.0034780041 0.019910945 0.03963001 0.06651784 0.13102830
## upsilon_algae_to_daphnia 0.0478212076 0.063322513 0.07411591 0.08670898 0.12600446
## upsilon_daphnia_to_NH4   0.0390770453 0.045554196 0.04921562 0.05297624 0.06127194
## upsilon_NH4_to_algae     0.2520987830 0.306208697 0.32943406 0.35403875 0.40647388
## zeta                     0.1552087708 0.216712947 0.25942604 0.31606758 0.52575689

Parameter correlations

The dependencies between your model parameters might be of interest to you. If you would like to analyse the correlations between parameters during the MCMC run, you can use a few ready-made functions to get a quick overview of the correlation structure.

The isotracer package comes with the minimalist function mcmc_heatmap() to draw the strength of parameter correlations:

mcmc_heatmap(run)

plot of chunk unnamed-chunk-15

But of course you could use other functions provided by other packages, such as:

Extracting parameters, trajectories and flows

If you are interested in getting detailed tables containing all the samples of the parameter posteriors, you can use the tidy_mcmc() function:

tidy_mcmc(run)
## # A tibble: 1,000 × 3
##    mcmc.chain mcmc.iteration mcmc.parameters
##         <int>          <int> <list>         
##  1          1              1 <dbl [8]>      
##  2          1              2 <dbl [8]>      
##  3          1              3 <dbl [8]>      
##  4          1              4 <dbl [8]>      
##  5          1              5 <dbl [8]>      
##  6          1              6 <dbl [8]>      
##  7          1              7 <dbl [8]>      
##  8          1              8 <dbl [8]>      
##  9          1              9 <dbl [8]>      
## 10          1             10 <dbl [8]>      
## # … with 990 more rows

By default the parameter values are nested into a list column, but you can also get a flat table with spread = TRUE:

tidy_mcmc(run, spread = TRUE)
## # A tibble: 1,000 × 10
##    mcmc.chain mcmc.iteration    eta lambda_algae lambda_daphnia lambda_NH4
##         <int>          <int>  <dbl>        <dbl>          <dbl>      <dbl>
##  1          1              1 0.151        0.0365        0.00267    0.0396 
##  2          1              2 0.122        0.0612        0.00478    0.0435 
##  3          1              3 0.0980       0.0500        0.0211     0.00770
##  4          1              4 0.118        0.146         0.00510    0.170  
##  5          1              5 0.110        0.0101        0.0120     0.00572
##  6          1              6 0.169        0.0706        0.0168     0.0131 
##  7          1              7 0.165        0.0215        0.0186     0.00253
##  8          1              8 0.207        0.0155        0.0145     0.0610 
##  9          1              9 0.0780       0.0417        0.00443    0.0288 
## 10          1             10 0.0905       0.0486        0.0598     0.103  
## # … with 990 more rows, and 4 more variables: upsilon_algae_to_daphnia <dbl>,
## #   upsilon_daphnia_to_NH4 <dbl>, upsilon_NH4_to_algae <dbl>, zeta <dbl>

The above table only contains the primary parameters. If you are interested in getting the predicted trajectories for individual MCMC samples, you can use the tidy_trajectories() function:

# We have to also provide the original network model `m`
tt <- tidy_trajectories(m, run, n = 200)
tt
## # A tibble: 200 × 4
##    mcmc.chain mcmc.iteration mcmc.parameters trajectories    
##         <int>          <int> <list>          <list>          
##  1          2              4 <dbl [8]>       <tibble [1 × 5]>
##  2          2             87 <dbl [8]>       <tibble [1 × 5]>
##  3          2            319 <dbl [8]>       <tibble [1 × 5]>
##  4          2            271 <dbl [8]>       <tibble [1 × 5]>
##  5          1             71 <dbl [8]>       <tibble [1 × 5]>
##  6          2            184 <dbl [8]>       <tibble [1 × 5]>
##  7          1            371 <dbl [8]>       <tibble [1 × 5]>
##  8          2            257 <dbl [8]>       <tibble [1 × 5]>
##  9          2            198 <dbl [8]>       <tibble [1 × 5]>
## 10          1            307 <dbl [8]>       <tibble [1 × 5]>
## # … with 190 more rows

As you can see, the tt object is a tidy table which contains the parameter values and the corresponding trajectories calculated for 200 randomly selected MCMC samples. The calculated trajectories are stored in the trajectories column and provide the quantities of unmarked and marked tracer (e.g. light and heavy isotope) for each compartment at each time step:

tt$trajectories[[1]]
## # A tibble: 1 × 5
##   timepoints  unmarked        marked          sizes           proportions    
##   <list>      <list>          <list>          <list>          <list>         
## 1 <dbl [260]> <dbl [260 × 3]> <dbl [260 × 3]> <dbl [260 × 3]> <dbl [260 × 3]>

Because each trajectory is itself a table containing a time series for each compartment, the output of tidy_trajectories() has several levels of nesting. This makes it a bit cumbersome to manipulate. Note that the output format of this function might change in the future.

Here is an example of what can be done using the predicted trajectories:

algae <- tt %>%
  mutate(prop_algae = map(trajectories, function(tr) {
    tr[["proportions"]][[1]][, "algae"]
  })) %>%
  pull(prop_algae) %>%
  do.call(rbind, .)
time <- tt$trajectories[[1]]$timepoints[[1]]
plot(0, type = "n", xlim = range(time), ylim = range(algae), las = 1,
     xlab = "Time", ylab = "Proportion of marked tracer (algae)",
     main = "Posterior sample of trajectories (for 15N prop. in algae)")
invisible(sapply(seq_len(nrow(algae)), function(i) {
  lines(time, algae[i,], col = adjustcolor("seagreen3", alpha.f = 0.2))
}))

plot of chunk unnamed-chunk-21

Finally, if what you are interested in are not the trajectories per se but the actual flows of nutrient during the experiment, you can use the tidy_flows() function to extract flows in a similar way:

#' Again, note that we also provide the original network model `m`
tf <- tidy_flows(m, run, n = 200)
tf
## # A tibble: 200 × 4
##    mcmc.chain mcmc.iteration mcmc.parameters flows               
##  *      <int>          <int> <list>          <list>              
##  1          2            110 <dbl [8]>       <grouped_df [6 × 3]>
##  2          1            395 <dbl [8]>       <grouped_df [6 × 3]>
##  3          1            401 <dbl [8]>       <grouped_df [6 × 3]>
##  4          2             66 <dbl [8]>       <grouped_df [6 × 3]>
##  5          2            134 <dbl [8]>       <grouped_df [6 × 3]>
##  6          1             46 <dbl [8]>       <grouped_df [6 × 3]>
##  7          2            193 <dbl [8]>       <grouped_df [6 × 3]>
##  8          1            240 <dbl [8]>       <grouped_df [6 × 3]>
##  9          1             98 <dbl [8]>       <grouped_df [6 × 3]>
## 10          1             48 <dbl [8]>       <grouped_df [6 × 3]>
## # … with 190 more rows

The returned object is very similar to the output of tidy_trajectories(), except that the trajectories column is replaced by a flows column:

tf$flows[[1]]
## # A tibble: 6 × 3
## # Groups:   from [3]
##   from    to      average_flow
##   <chr>   <chr>          <dbl>
## 1 algae   daphnia      0.0667 
## 2 algae   <NA>         0.0718 
## 3 daphnia NH4          0.0794 
## 4 daphnia <NA>         0.0623 
## 5 NH4     algae        0.0745 
## 6 NH4     <NA>         0.00770

The average flow values are given in flow per unit of time. See ?tidy_flows() for more details, including the possibility of calculating steady state flows for network systems that admits a steady state equilibrium.

The tidy_trajectories() and tidy_flows() functions are especially useful when you want to do some calculations related to some specific properties of the trajectories or of the nutrient flows over the whole MCMC posterior.