NEWS | R Documentation |

A faster-fitting procedure is implemented (through fitme()) for fitting a class of LMMs (without heteroscedasticity, without fixed random-effect variances...).

The restriction, to models with a single random effect, of the computation of the component of prediction variance for uncertainty in dispersion parameters, has been removed.

simulate.HLfit() has new arguments for controlling prior weights and simulation from fits with structured dispersion.

EarthChord distance: see help("EarthChord").

Models with 'geostatistical' random-effect terms (Matern, Cauchy) can now be fitted by sparse-precision methods (although this is not necessarily very efficient).

Possible improvements in numerical accuracy, notably by sparse-precision methods when (1) fitting models with AR1 terms, that are numerically unstable for extreme AR1 correlation values; (2) in REML fits with low residual variance.

Improvement in numerical accuracy of computation of prediction variance when residual variance is low or spatial correlation matrices are nearly-singular.

New function MSFDR() implements the multiple-stage false discovery rate method of Benjamini and Gavrilov 2009.

Improved .LRT() function (called through fixedLRT()) which makes it more suitable for programming, and which better handles generic-optimization issues.

fixedLRT(), LRT() and anova() now call by default spaMM_boot() for the bootstrap. They handle new arguments to control the spaMM_boot() call, allowing finer control of poorly- or non-informative bootstrap samples.

is_separated() convenience function to check for (quasi-)separation in binomial-response models (this was previously implemented but not exported nor documented).

Possibly improved initial value for lambda may affect speed and results of some fits.

New extractors formula(), terms(), nobs(), and extract.AIC(), being methods for 'HLfit' objects of generics defined by 'stats'.

The syntax '.' can now be used in formulas, e.g. HLfit(y ~ (.)^2, ...)

Automatic conversion of correlation matrices to precision factor matrices when sparse-precsion algorithm is requested; new as_precision() function for such conversion.

More transparent control of the residual variance parameter of a residual dispersion model. However, this affects the default results if the residual dispersion model is a mixed-effect one. Use 'fixed=list(phi=NA)' in the 'resid.model' (forcing estimation of its phi) to recover past results.

More efficient computation of conditional AIC. Users can also control whether it should be computed along with other information criteria.

New extractor how() to cleanly extract information about how a model fit was obtained.

An hidden memory-dependent constraint on the size of data that could be fit by fixed-effect models by ML has been alleviated.

Numerous improvements for residual dispersion models with random effects makes fitting of the joint residual and main-response models more likely to converge in reasonable time when both models contain spatial terms.

The Predictor() function is deprecated. See help("Predictor") for details.

The designL.from.Corr() function is superseded by mat_sqrt().

The structure of the return value of sym_eigen() is modified.

'resid.formula' argument of fitting functions is formally deprecated (it has been declared obsolete since v1.8.0).

The computation of prediction variance (specifically its component for uncertainty in dispersion parameters) could be incorrect when several random effects were considered. Fixed as noted in 'New Features'.

The 'random-slope' (Intercept+slope) code was OK, but the more general 'random-coefficients' code (>2 parameters) was not.

Bug in initialization of adjacency correlation parameter, which prevented at least some bootstrap computations.

Fixed a (previously documented) issue in spaMM's update() method, stemming from the distinct interpretation of formulas without explicit fixed-effect terms by spaMM vs. stats. Formula updates should still be carefully checked, as getting them fluent is not on the priority list.

Fits failed when spaMM.options(sparse_precision=TRUE) but a corrMatrix was not specified as a precision matrix.

Prediction failed from models involving poly(., raw=FALSE) terms.

Prediction should use contrast information contained in the fitted object rather than in the session's options().

confint() for fixed-effect models has various problems.

An horrendous bug was found in the implementation of Gamma family for inverse link. Fortunately, no reasonable people would use Gamma(inverse)...

Stricter control of numerical accuracy of COMPoisson-family fits, which does not affect high-nu fits (which were already accurate), but which improve accuracy particularly around nu=1.

Use of the 'lpSolveAPI' package is reintroduced to check (quasi-)separation.

New extractor get_ranPars() (with limited functionality so far).

Parallel computation of some bootstrap computations by foreach+doSNOW has been reinstated.

plot.HLfit has a new argument 'ask', and no longer has a side effect on subsequent plots.

Bug in COMPoisson-family fits, which impacted most clearly the results when some response values were zero.

Crashing bug could occurs in zero-truncated models (for various cases of the Laplace approximation).

Fixed some bootstrap code which could fail (because the original try() statement had disappeared through mis-editing :-( ).

simulate(., type = "(ranef|response)") failed, or returned incorrectly formatted results.

Non-trivial re.form argument now works for models including a 'corrMatrix' term.

Precision matrices with more levels than there are in the data are now handled.

COMPoisson-family fits could crash in some cases (bug introduced in version 2.4.0 in code for initial values).

Calls not requesting any form of least-squares fit, such as fitme(y ~ 0, data = <.>), failed.

Prediction interval computation failed for zero-truncated distributions.

Prediction variance computation could fail for large newdata ('slicing' algorithm).

Cauchy spatial correlation family has been implemented.

For binomial family the response can now be specified as a factor.

Better default selection of methods for autoregressive fits with many random-effect levels may lead to much faster fits.

fitme() now selects estimation methods for phi and lambda by default according to nrow(data), which may change details of the results, To overcome the new defaults, fitme() now allows better control of estimation method and of initial values for phi and lambda.

get_predCov_var_fix() may now handle a non-trivial 're.form'.

New 'max.print' argument for summary.HLfit().

New function spaMM_boot() for parametric bootstrap.

New 'resp_testfn' argument for simulate.HLfit(), to simulate samples that satisfy a given condition.

Revised summary of random-effect parameters.

revised format of ranef() screen output for random-coefficient terms, using new method 'print.ranef'.

Imports package 'pbapply' to draw progress bars, including in parallel computations; no longer Suggests the package 'foreach' (and optionally 'doSNOW') for the same effect.

An error was produced when model formulas included a nested random effect (.|./.) followed by a correlated random effect such as Matern(.).

Incorrect prediction variance with non-trivial 're.form' with random-coefficient term(s) or permutations of the random effects.

Errors when attempting to fit models with correlated random effects and 'negbin' response family.

A wrong algorithm could sometimes be selected for 'inner' estimation of the correlation parameter of CAR models.

Wrong documentation for ranef().

anova() failed when a fit had non-estimable coefficients because of collinearity.

Revised 'help(ranFix)' to clarify the usage of argument for fixing parameters in the different fitting functions.

Revised default initial values of lambda may slightly change results.

Revised default 'fittingFunction' for fixedLRT() may slightly change results, or may call for a user-provided explicit value of 'fittingFunction'.

Corrected the poorly written documentation for COMPoisson.

Zero-truncated variants of Poisson and negative-binomial families are now implemented.

Models with first-order autoregressive (AR1) random-effects terms are now fitted.

fitme() now fits models with random-coefficient terms, using by default a different strategy than the other fitting functions.

Prediction variance computations extended to most models handled by spaMM, including random-slope models.

The 'ranFix' argument of HLfit(), and similar 'ranPars' or 'fixed' arguments of other functions, allow a finer control of variance parameters of random effect than before.

predict.HLfit() should be substantially faster in most practical cases, particularly when the 'newdata' have many rows.

Better approximations are implemented for the infinite sums required for fitting COMPoisson models, in particular for low 'nu'.

New extractor get_rankinfo().

Some gains in speed, particularly for fit by sparse-precision algorithms, which also benefit from better memory management for large model matrices.

Prediction variance computation now take into account uncertainty in the random effect variance for adjacency random effects.

Control of optimizers has been revised, with distinct 'optimizer1D' and 'optimizer' in the spaMM.options(); lower case is now used for 'optimizer' keyword in all user-level code; and minqa::bobyqa() can now be called. See ?spaMM.options for further information.

Argument 'ZALMatrix' of predictor() is deprecated.

Argument 'as_matrix' of get_ZALMatrix() is deprecated.

predict.HLfit() gets a new argument 'blockSize'.

predict.HLfit( . , binding=NA) newly defined to return a vector instead of a one-column matrix. Attributes will differ too.

Removed dependency on package 'ff'.

Since version 2.0.0, the ranef() extractor did not return "correlated" effects except for random-coefficient models.

Prediction variance computation failed for models involving an adjacency random effect and other random effects.

Revision of 'help(autoregressive)' following introduction of AR1 models.

Revision of 'help(fitme)' to account for its new features.

New 'help(fixed)' gathering information on how to fix parameter values.

Simplification and updating of documentation pages for fitting functions.

New or updated documentation for 'negbin', 'Tnegbin', 'Poisson', 'Tpoisson' following implementation of the truncated variants.

Reimplementation of corMatern (forced by effectively backward-incompatible changes in Rcpp) might very slightly change results.

Added '...' argument to LRT(). The function does not yet make anything of it.

Widespread improvement in speed for large data sets, thanks to some of the following changes:

Sparse precision matrix algorithm extended to all HGLMs whose linear predictor involves only a restricted set of random effect terms (excluding mainly Matern and random-coefficient terms). It is used by default for conditional autoregressive models, or when the new argument 'covStruct' is used to provide a sparse precision matrix: see help(covStruct). See also help(pedigree) for a type of applications where the latter syntax may be of particular interest, and help(sparse_precision) for more information.

Significantly faster algorithm for fitting models with large dense correlation matrices (as in Matern correlation models).

new extractor function get_boot_replicate() to get the response vector for a given bootstrap replicate

‘spaMM_glm()’ and ‘spaMM_glm.fit()’ get the same ‘singular.ok=TRUE’ argument that ‘glm()’ and ‘glm.fit()’ got in R-devel r73380 (2017-09-28).

More reliable version of Levenberg-Marquardt algorithm for mixed models (see 'help("LevenbergM")'). Revised rules for using it, so that users typically do not need to think about it.

fixedLRT() results might be modified in particular when the likelihood has local maxima.

Changes in the way the generic optimizers are used may lead to changes in the fitme() and corrHLfit() results.

Usage of 'control.corrHLfit' argument has been modified.

Fitted values always get their names from the row names of the data.

Fixed numerical issues for binomial(cloglog) family with large values of linear predictor.

Random-coefficient models could not be properly fitted by Levenberg-Marquardt algorithm.

simulate.HLfit() can now simulate samples from a fitted GLMM under the distribution of random effects conditional on the data and given the fixed-effect estimates (using a Gaussian approximation in the GLMM case).

predict.HLfit() could no longer return covariance matrices as a result of a bug introduced in version 2.1.0.

'respVar' computation by predict.HLfit() did not take into account non-identity links, although the documentation suggested it did.

fitme() can now use a fast sparse matrix algorithm for CAR (adjacency) (G)LMMs. See revised help(adjacency) for further information.

confint.HLfit() now computes confidence intervals for GLMs.

spaMM now handles cases where nloptr::nloptr() can be trapped in an infinite loop.

residuals.HLfit() has been redefined to bring it closer to stats::residuals.glm(). Up to now it only returned response residuals, now it returns (unscaled) deviance residuals.

'method' argument of spaMM_glm() redefined to allow more experiments (default behaviour is unchanged).

simulate.HLfit() now also simulates GLMs.

predict(<formula ith random-coefficient term>, <newdata>) is now handled.

binomial(cauchit) is now handled.

simulate() member function added to COMPoisson().

simulate() member function modified in negbin() and in spaMM's version of Gamma() to return matrices when nsim>1, as described in help(family).

simulate.HLfit() now simulates COMPoisson and negbin (mixed) models.

HLCor() no longer fitted adjacency models in Version 2.0.0.

spaMM.options(<new values>) did not return old values.

For GLMs, confint.HLfit() (computed no interval but) did return an empty interval.

COMPoisson$linkfun sometimes returned theta=0, causing failure of fit.

Several problems corrected in degrees of freedom computation for information criteria retruned by get_any_IC().

Several problems corrected in prediction variance calculation with non-null 're.form'.

Core functions have been extensively rewritten to manage expensive matrix computations more efficiently. As a result, version 2.0.0 is 2-3 times faster when correlation matrices of random effects are dense (as typically the case for Matern correlation matrices), and can be much faster when correlation matrices are sparse.

Simple interface for using orthodromic distances: see help(Earth).

bootstrap computations can now use parallelisation; see new argument 'nb_cores' for fixedLRT() and related functions.

The term for uncertainty in dispersion parameters is now included in computation of prediction variance (1) by default from objects returned by fitme() (special default 'control' argument no longer needed); and (2) for models with multiple random effects.

New extractors get_predCov_var_fix() and preprocess_fix_corr() for more efficient computation of specific blocks of prediction covariance matrix.

get_ZALMatrix() gets a new argument 'as_matrix' to control the return value.

confint() now works for HLCor fits.

nloptr() call replaces optim() call in random-slope code, with notable improvements in some toy examples. Random-slope models with >1 predictor variables, such as (x+y|z), are now fitted.

New function stripHLfit() to remove some large objects from a fit object before saving it to file.

Time to fit a model is now recorded in the fit object (as element 'fit_time').

Structure of bootstrap output in return value of fixedLRT() modified.

Structure of ranef() result modified.

Several efforts to reduce the size of fitted objects.

Likelihood-ratio chi-square statistic renamed 'chi2_LR' (was 'LR2') in return value of anova() and some bootstrap outputs.

Raster.Rd removed to reduced package dependencies. A link to a web page is instead provided.

spaMM now provides a slightly less informative diagnostic for (quasi-)separation in binary response models, but no longer 'imports' package lpSolveAPI, instead 'suggesting' e1071.

Tracing stuff, such as 'trace' argument of some functions, has been removed.

Possible values of 'control.corrHLfit$Optimizer' are (as before) 'L-BFGS-B' and (new) 'NLOPT_LN_BOBYQA'. nlminb can no longer be selected.

get_ZALMatrix() is now a standard function rather than part of the object resulting from a fit.

fitme() now fits CAR model parameters using its own optimization procedure rather than through HLfit's procedures.

Message added to warn users about the interpretation of formulas without fixed effects.

More meaningful display of results for random-coefficients models; new argument 'details' for summary.HLfit().

ML fit summaries no longer display a 'p_bv' value, to prevent confusion about what this value means.

Version 1.10.0 failed on R < 3.2.0 because cBind() and rBind() calls from earlier spaMM versions have been replaced by cbind() and rbind() calls (see help(cBind)). The Depends field is now corrected to R (>= 3.2.0).

A few stopping bugs in predict() have been corrected.

LRT() did not work on HLCor fit objects. This is corrected.

spaMM now fits models with random effects in the residual dispersion model, and thus implements a class of “double hierarchical GLMs”.

fitme() now fits models with offset in the dispersion model.

Modified convergence criteria for the iterative algorithm in HLfit().

Examples run generally faster by about 30%.

New extractor function get_intervals() and get_ZALMatrix().

Intervals returned by predict.HLfit() (and hence by get_intervals()) use Student's t distribution when the fitted model is a simple linear model.

Prediction variance can now be computed from fits of models with a corrMatern term.

An HLfit object's $APHLs$p_bv for a *fixed-effect* model now returns the restricted likelihood (for estimation of residual dispersion parameters), which it has never done so far (out of disregard for this case). This allows REML fits of fixed-effect models using fitme() (but has no bearing on similar fits using HLfit()).

More informative summary() when non-default spatial distance function, or non-default prior weights, are used.

New 'refit' control in fitme().

New extractor function get_RLRTSim_args() for use with the LRTSim package to perform small-sample LRT of the presence of a random effect.

Summary() outlook revised, in particular to more accurately display the meaning of phi for Gamma response.

The objects returned by the fitting functions should be of much smaller size than before.

Better reporting of terms that have to be ignored in the computation of prediction variance.

New 'inverse.Gamma', 'spaMM-conventions' and 'member-extractor' documentation pages. HLfit() documentation revised.

Some numerical issues that could affect GLM fitting with HLfit() have been fixed.

More accurate detection of unidentifiable random effect components, when residual variance formulas involve a pure offset term.

confint() on a LMM fitted by fitme() was returning incorrect results. This is corrected.

Computation of prediction variance has been reimplemented so that it requires storing smaller matrices than before. Further, some terms required for this computation are computed only when prediction variance is requested, rather than being always returned by HLfit(). This allows some gains in computation time for large data sets.

simulate.HLfit() gets new arguments and prints some information, in anticipation of future developments.

spaMM.colors() gets new argument 'adjustcolor_args' for finer control.

Filled plot functions gets new argument 'plot.title'.

spaMM_glm(), which has been designed to 'rescue' stats::glm() when the latter fails, did not rescue it for some Gamma(log) GLMs. This is corrected.

Computation of AIC (and other information criteria) failed in many cases. This is corrected.

confint() did not correctly handle models with several random effects. This is corrected.

fitme() does not fit random-slope models, nor models with offset in the dispersion model, but this was not documented. This is corrected.

C++ code has been fixed for compatibility with recent standards.

New fitme() function, intended to become the main fitting function in spaMM, can be substantially faster than HLfit() or corrHLfit(), particularly for large data sets, but can occasionally be slower.

Some other moderate but noticeable improvements in speed.

Users no longer need to explicitly request AIC computation when calling the fit of a model. The new extractor function get_any_IC() (or its more conventionnally named alias, AIC()) can be used to compute them from a fitted model object.

Negative binomial response models can be fitted by 'family=negbin()'.

The syntax 'family=COMPoisson()', with unknown 'nu' parameter, works. This means that COMpoisson() models can now be fitted with unknown values of this parameter, by corrHLfit() or by the new fitme() function. The same is true for the parameter of the 'negbin' family.

Initial or fixed values of the variance of random effects can be controlled independently when there are several random effects.

spaMM_glm() gains a new argument 'strict' for finer control. spaMM_glm.fit() has increased default 'control$maxit'.

filled.mapMM() now returns a predictor of the response surface. It also gains a new argument 'smoothObject' that takes such a predictor, allowing for faster redrawing.

fixedLRT() gets a new argument 'fittingFunction' to select between corrHLfit() and fitme().

Revised usage of 'corrMatrix' argument of HLCor().

'spaMM', as well as 'RcppEigen', had become broken for Windows 32bit. See https://github.com/RcppCore/RcppEigen/issues/34 for details. A suitable compilation flag is now used to overcome this issue.

simulate.HLfit() could fail when a design matrix was stored as a _M_atrix. This is corrected.

predict.HLfit() can return prediction intervals.

New spaMM_glm() and spaMM_glm.fit() functions provide a robust alternative (using the Levenberg-Marquardt algorithm) to glm.fit() when the latter diverges. They also circumvent the initial-value issues of glm().

Slight general increase in speed.

The Conway-Maxwell-Poisson family is implemented.

New extractor functions get_fixefVar(), get_predVar(), get_residVar() and get_respVar().

Subsetting the result of a predict() call also subsets its variance attributes, thanks to a new internal 'Extract' method '[.predictions'.

spaMM is protected from disturbing numerical problems of the Gamma()$dev.resids() function, thanks to an internal redefinition of the Gamma family.

New 'control.glm' argument of HLfit(), to control calls to spaMM_glm() within HLfit().

Once-again revamped 'variances' argument and documentation of predict.HLfit(). The 'variances' argument of filled.mapMM() is replaced by 'variance'.

New argument 'resid.model' replaces 'resid.formula' and 'resid.family' specifications in the main fitting function.

The default behaviour of mapMM() for argument 'map.asp=NULL' has been modified.

The obscure control parameters 'RHOMAX' and 'NUMAX' previously accessible through spaMM.options() are no longer needed and have been removed.

Better control of some plot margins.

Fitting random-slope models may be substantially less slow (but still slow).

Printing bug for non-standard REML formula in summary.HLfit() corrected.

Conditions where predict() return an object of class 'predictions' corrected to avoid a printing bug.

The 'xrange' and 'yrange' arguments of spaMMplot2D() were not effective. This is corrected.

Printing bug in summary.fixedLRT() corrected.

spaMM.options(maxLambda=. . .) did not fully control extreme lambda estimates. The revised code is more effective, and also better bypasses some numerical problems that can occur for such values.

Fitting phi using the corrHLfit() outer optimization mechanism was inefficiently implemented and could return some confusing output. This is corrected.

Less inefficient code for fixed-effect models, faster and handling larger data sets.

Changed default value of 'upper' argument of optim call() within corrHLfit(), and better reporting of issues with this call.

Argument 'which' replaces arguments 'REML' in logLik().

New control 'break_conv_logL' in argument control.HLfit of function HLfit(). Mainly for development purposes.

Improved reporting of convergence issues in HLfit().

More comfortable print method for vcov() value.

This new version includes several improvements of predict(), the most notable being a correction of prediction variances for new values of predictor variables, which were incorrect in CRAN version 1.6.2 (but not in CRAN version 1.5.1). Many other more trivial bugs or features of predict() have been corrected.

summary.fixedLRT() failed to display correctly the bootstrap results. This is corrected.

A bug in C++ code, introduced in version 1.7.1, has been corrected in version 1.7.2; and code applying some Eigen functions on a matrix with a dimension equal to zero (generating an UBSAN runtime error but a correct retrun value in other conditions) has been fixed.

Prediction variance can now account for uncertainty is dispersion parameters (though not for all models; see Details of 'disp' in the 'predict' documentation).

Now allows prediction for a large number of spatial positions by storing some large correlation matrices on disk rather than in memory (using the ff package, and as controlled by the spaMM option 'ff_threshold').

Convenience extractor functions 'dev_resids' and 'deviance' have been added.

New data objects 'worldcountries' and 'oceanmask' added for better plots of spatial predictions (see the new documentation '?raster' for plot of spatial predictions using rasters and geographical projections).

More efficient implementation of a conditional autoregressive (CAR) model.

're.form' argument of 'predict' function has been extended to handle values of class 'formula'.

etaFix$beta usage has been extensively revised to provide a convenient interface for fixing (some of) the fixed effects (beta coefficients) values.

If anyone on Earth ever tried inverse.Gamma("-1/mu") random effects, s-he might have been confronted with negative estimates of u_h. This problem is inherent to the algorithms described in the HGLM literature, but spaMM now brings a solution, which allows one to fit more securely models with v ~ Gamma( lambda/(lambda+1), lambda+1). A new link identity is also allowed for Gamma random effects [i.e., v=u ~ Gamma(1/lambda,lambda) ], albeit with the restriction that lambda is < 1.

New 'symSVD' argument for designL.from.Corr function.

New keyword 'predict' for 'which' argument of 'plot.HLfit', for easier plotting of prediction diagnostic plot.

'make.scaled.dist' has been renamed 'make_scaled_dist', and gets a new 'return_matrix' argument for programming purposes.

Substantial gains in speed for moderately large data sets (hundreds of lines), in particular by minimizing the use of the QR module from Eigen.

Users no longer have to think about pre-computing prediction coefficients. 'predictionCoeffs' function has been removed. 'coeffs' argument of predict.HLfit() has been removed.

More compact printing of predict() results.

The output format of information criterias computed by HLfit, and related documentation, have been revised.

confint() did not correctly handle PQL fits. This is corrected. In addition, it better handles some non-convergence issues.

Gamma response models with non-unit 'prior.weights' produced an inaccurate correction from EQL to 'HL(1,1)'. This is corrected.

anova() and LRT() declared some nested models as non-nested, and sometimes stopped. This is corrected.

HLfit could stop on some fits involving moderately large sparse design matrices (Z) for random effects. This is corrected.

A printing bug for ranef(), which prevented the display of its result when lambda was fixed, has been corrected

New example_maxtime package option for controlling execution of examples.

Tests are now controlled by a local environment variable _LOCAL_TESTS_ so that they are not run on CRAN.

Faster code for (some) sparse models with a single random effect on large data sets. It is also possible (by using

`spaMM.options(QRmethod=Matrix::qr)`

) to use sparse matrix algorithms in other models, but this will be slow in many applications. Various other changes were made to speed up computation in dense models (by about 10% in the tests).The non-Euclidean distance functions implemented in the

`proxy`

package (e.g. Geodesic) can now be used (although only Geodesic was tested, and the Matern correlation function with arbitrary rho and nu is not mathematically feasible for all distance functions). See`control.dist`

argument of`HLCor`

and`corrHLfit`

.

Reorganized graphic functions, with new

`spaMMplot2D`

function and new`envir`

argument for`mapMM.HLfit`

.-
`predict.HLfit`

modified for better consistency with other widely used predict methods: added`newdata`

argument and modified default behaviour of`binding`

argument. New

`getDistMat`

extractor for extracting the distance matrix used in the Matérn modelThis NEWS.Rd file is created

Correction of a bug that caused incorrect prediction on new data for 1D (only) spatial model.