After a decade on CRAN, merTools 1.0.0 is out today. It has been a long-term goal of mine to bring merTools to a stable state, fix outstanding bugs, and rework the `predictInterval()` function at its core to be more modular and easy to maintain. This release achieves that and marks the package as feature complete and the start of a long-term-support (LTS) phase. This release hits the goal of 1.0.0 of a stable API, outstanding issues closed, a correctness fix at the core of predictInterval(), and modularity in the core prediction interval functionality. The documentation is also now available [via a pkgdown site as well!](https://jknowles.github.io/merTools)

install.packages("merTools")

What merTools is for #

merTools has been on CRAN since 2015, with over 457,000 downloads and about 8,000 a month. It exists to get the most out of large mixed-effects models fit with lme4 . Its headline function, predictInterval(), produces prediction intervals for lmer/glmer models by simulating from the joint distribution of the fixed and random effects — the approach from Gelman and Hill (2007),¹ the same machinery behind arm::sim().

The motivation is practical. lme4::bootMer() and full MCMC give you principled uncertainty, but on models with tens of thousands of groups they can be impractical or simply too slow to use interactively.

predictInterval() is built to give you honest intervals on exactly those models, in about a second.

Try it out and validate it against brms! #

The obvious question for a simulation-based approximation is how close it is to doing the full Bayesian thing. When we first wrote merTools brms was in its infancy, but since then it has become my go-to for most Bayesian inference and, frankly, made much of merTools obsolete. I used an LLM to assist me with refactoring the code and responding to open issues, so I wanted to create a new set of tests and benchmarks to validate that the package still worked.

So for 1.0 I created a new short vignette that compares brms and merTools results side by side: the same data and the same model — the bundled hsb data (7,185 students in 160 schools), modeling math achievement from student SES, school-mean SES, and a school-varying SES slope. We hold out 20% of students to test on.

data(hsb)
hsb$schid <- factor(hsb$schid)
new_schools <- sample(levels(hsb$schid), 6)
hsb$.set <- "train"
hsb$.set[hsb$schid %in% new_schools] <- "test_new"
seen <- which(hsb$.set == "train")
hsb$.set[sample(seen, round(0.2 * length(seen)))] <- "test_seen"
train     <- droplevels(hsb[hsb$.set == "train", ])
test_seen <- hsb[hsb$.set == "test_seen" & hsb$schid %in% levels(train$schid), ]

f1 <- mathach ~ ses + meanses + (ses | schid)
m_lme <- lmer(f1, data = train)

# Fit brms once and cache it. Persist the genuine from-scratch fit time to a
# file, so the timing table below reports the real compile+sample cost even on
# later renders that just read the cached fit (which take ~2s, not ~7 min).
brms_cache <- file.path(outdir, "brms_hsb_meanses.rds")
time_cache <- file.path(outdir, "brms_hsb_meanses_seconds.rds")
fresh_fit <- !file.exists(brms_cache)
elapsed <- system.time(
  m_brm <- fit_brm(f1, train, gaussian(), "brms_hsb_meanses"))[["elapsed"]]
if (fresh_fit) saveRDS(elapsed, time_cache)
brms_fit_seconds <- if (file.exists(time_cache)) readRDS(time_cache) else elapsed

Point estimates are essentially identical #

Both methods condition on the same estimated fixed effects and school BLUPs, so the conditional-mean predictions agree almost exactly.

lme_mean <- predict(m_lme, newdata = test_seen)
brm_mean <- colMeans(posterior_epred(m_brm, newdata = test_seen, allow_new_levels = TRUE))
c(correlation = cor(lme_mean, brm_mean),
  mean_abs_diff = mean(abs(lme_mean - brm_mean)),
  response_sd = sd(test_seen$mathach))

#>   correlation mean_abs_diff   response_sd 
#>    0.99984045    0.03670134    6.71745425

ggplot(data.frame(merTools = lme_mean, brms = brm_mean), aes(merTools, brms)) +
  geom_abline(slope = 1, intercept = 0, linetype = 2, color = "grey50") +
  geom_point(alpha = 0.3, size = 0.8, color = "#1B9E77") + coord_equal() +
  labs(x = "lme4 predict()", y = "brms posterior_epred()") +
  theme_civilytics()

And they are equally well calibrated #

The real test is out-of-sample: do nominal intervals cover held-out scores at the nominal rate? We draw a full predictive distribution for each held-out student from each method — predictInterval(returnSims = TRUE) and brms::posterior_predict() — and compare empirical coverage. Both track the nominal level, and each other.

The mt_draws() helper used below — saved alongside the analysis — just wraps predictInterval() to hand back the raw simulation matrix:

# One column of posterior-style draws per row of `newdata`.
mt_draws <- function(fit, newdata, ...) {
  pi <- predictInterval(fit, newdata = newdata, returnSims = TRUE, ...)
  attr(pi, "sim.results")
}

mt_seen <- mt_draws(m_lme, test_seen)
pp_seen <- posterior_predict(m_brm, newdata = test_seen, allow_new_levels = TRUE)
data.frame(nominal = NOMINAL,
           merTools = coverage(mt_seen, test_seen$mathach, NOMINAL),
           brms     = coverage(pp_seen, test_seen$mathach, NOMINAL))

#>   nominal  merTools      brms
#> 1    0.50 0.4569776 0.4598698
#> 2    0.80 0.7895879 0.7932032
#> 3    0.90 0.9168474 0.9168474
#> 4    0.95 0.9660159 0.9703543

At a fraction of the cost #

t_lme <- system.time(lmer(f1, data = train))[["elapsed"]]
t_mt  <- system.time(mt_draws(m_lme, test_seen))[["elapsed"]]
t_pp  <- system.time(posterior_predict(m_brm, newdata = test_seen,
                                       allow_new_levels = TRUE))[["elapsed"]]
data.frame(
  step = c("lme4 fit", "predictInterval", "brms fit (compile+sample)", "posterior_predict"),
  seconds = round(c(t_lme, t_mt, brms_fit_seconds, t_pp), 2))

#>                        step seconds
#> 1                  lme4 fit    0.07
#> 2           predictInterval    0.66
#> 3 brms fit (compile+sample)  409.13
#> 4         posterior_predict    2.38

The entire lme4 + predictInterval() path runs in about a second; the single brms fit takes several minutes — a couple of orders of magnitude, and the gap only widens on the large models predictInterval() was really built for. The point isn’t to stop using brms: if you want the full posterior, use it. The point is that when full MCMC is impractical, predictInterval() is a fast, well-calibrated stand-in. The full story — new-group handling and a binomial GLMM — is in the validation vignette .

When full MCMC is impractical, predictInterval() is a fast, well-calibrated stand-in.

Visualizing group effects with plotREimpact() #

New in 1.0, plotREimpact() visualizes REimpact() output — the fitted outcome across the distribution of a grouping factor’s effect. Pass a named list and it overlays grouping factors on shared axes, so you can compare their influence directly. Here, the instructor and student effects in the InstEval ratings data:

m1 <- lmer(y ~ service + lectage + studage + (1 | d) + (1 | s), data = InstEval)
imp_d <- REimpact(m1, InstEval[7, ], groupFctr = "d", breaks = 5, n.sims = 300, level = 0.9)
imp_s <- REimpact(m1, InstEval[7, ], groupFctr = "s", breaks = 5, n.sims = 300, level = 0.9)
plotREimpact(list("Instructor (d)" = imp_d, "Student (s)" = imp_s)) + theme_civilytics()

Moving a case from the bottom to the top of the instructor-effect distribution shifts the predicted rating substantially more than the same move across the student distribution — the instructor grouping factor carries more of the signal.

What else is new in 1.0 #

The release notes have the full list; the highlights:

• Correctness fix for nested random effects (#124). predictInterval() was returning seed-dependent point estimates when a prediction frame mixed observed and unobserved levels of an interaction grouping factor (e.g. (1 | a/b)). A max.col() tie-break was silently letting an unobserved level borrow a random observed level’s effect. Unobserved levels now correctly fall back to the fixed effects; observed-level predictions are bit-for-bit unchanged.

• New new.levels = "draw". For a group the model never saw, the default ("zero") drops its random effect; "draw" instead samples the effect from the estimated random-effect covariance — the direct analogue of brms::posterior_predict(allow_new_levels = TRUE). The default is unchanged.

• plotFEsim() now highlights significant terms, and shinyMer() is revived and extended with a Model Summary tab and subset-based draws.

Why LTS / maintenance mode #

merTools does what it set out to do, and the API has been stable for years. Going forward, 1.0.x releases will be about keeping it healthy — bug fixes, CRAN and dependency compatibility, and documentation — rather than churning the interface.

Get it / cite it / file issues #

install.packages("merTools")
citation("merTools")

• Package site & vignettes: https://jknowles.github.io/merTools/
• Source & issues: https://github.com/jknowles/merTools

Thanks to everyone who has contributed over the years — Carl Frederick, Alex Whitworth, Ben Bolker (@bbolker), Davis Vaughan (@DavisVaughan) — and to the issue reporters who made this release better, including @dotPiano, whose report led to the #124 correctness fix.

References #

New book covers the ins and outs of doing education data analysis well in public education agencies from the perspective of three analysts with decades of experience in the field.

Boston, MA – October 7, 2019 – Wendy Geller, Dorothyjean Cratty, and Jared Knowles – three data analysts with expertise in public education agencies – have teamed up to write a new book which covers the missing elements that are critical to success in building data capacity in education agencies. The book is intended for education agency data analysts, teams of analyst, and data managers, strategists, and leaders seeking to improve how their agency operates.

Many education agency data analysts come from a social science research background and the transition to work inside agencies can come with a lot of new challenges. This book is a guide through those challenges covering topics such as metadata, data requests, how to work with IT, politics, and descriptive data analysis.

The book covers these topics with wit and humor and a perspective only possible from authors who’ve been in the trenches and gotten the work done. Each chapter was reviewed by another expert in the field who gave valuable outside perspective and broadened the horizon of the book to ensure its relevance for agencies across the country.

The book is accompanied by a website where analysts across the country can get in touch and suggest contributions for planned future volumes. On the website you can also learn more about the biographies of the authors and each of the contributors.

Education Data Done Right (EDDR) is available now digitally on LeanPub with a suggested price of $15: www.leanpub.com/eddatadoneright Print copies available at Amazon .

About the Authors

Dr. Wendy Geller is currently the Director of the Data Management & Analysis Division. There, she leads a team that serves as a centralized resource to the Vermont Agency of Education. Her crew collects, stewards, and leverages the institution’s critical data assets to create and share data products that enable empirically-based practice and policy decision-making.

Dr. Jared Knowles is formerly a research analyst at the Wisconsin Department of Public Instruction (2011-2016) and is currently the president of Civilytics Consulting LLC, which provides training, analytic services, and strategy to education agencies across the country.

Ms. Dorothyjean Cratty is formerly a research associate at the U.S. Department of Education. She is currently the founder of DJC Applied Research, which provides high-quality in-depth data analysis of administrative education data.

Media contacts:

Jared E. Knowles

617.393.1939

jared@civilytics.com

Table of Contents

Introduction

Metadata and Business Rules

An Analyst’s Guide to IT

Data Requests: You Can Make them Useful (we swear)

Politics and Data Driven Decision Making

Moments of Truth: Why Calculating Descriptive Statistics is Important

Applying Tools of the Trade: Descriptive Data Commands in Context

Conclusions

I’m off on a new professional adventure where I am doing public facing work for the first time in years. While working at home the other day I thought it would be a great idea to keep myself on-task and document my decisions if I recorded myself working with my webcam. Then, I thought, why stop there – why not livestream my work?

And thus, the R Data Science Livestream was born. The idea is that every day for an hour or two I will livestream myself doing some data science tasks related to my current project – which is to analyze dozens of years of FBI Uniform Crime reports (read more ). I haven’t done much R coding in the last 4 months, so it’s also a good way to shake off the rust of being out of the game for so long.

So if you are at all interested or curious why someone would do this, check out the landing page I put up to document the project and if you are really curious, maybe even tune in or watch the archives on YouTube !

Update 2: Development version also supports models from the blme package.

By far the most popular content on this blog are my two tutorials on how to fit and use multilevel models in R. Since publishing those tutorials I have received numerous questions, comments, and hits to this blog looking for more information about multilevel models in R. Since my day job involves fitting and exploring multilevel models, as well as explaining them to a non-technical audience, I began working with my colleague Carl Frederick on an R package to make these tasks easier. Today, I’m happy to announce that our solution, merTools , is now available. Below, I reproduce the package README file, but you can find out more on GitHub. There are two extensive vignettes that describe how to make use of the package, as well as a shiny app that allows interactive model exploration. The package should be available on CRAN within the next few days.

Working with generalized linear mixed models (GLMM) and linear mixed models (LMM) has become increasingly easy with advances in the lme4 package. As we have found ourselves using these models more and more within our work, we, the authors, have developed a set of tools for simplifying and speeding up common tasks for interacting with merMod objects from lme4. This package provides those tools.

Installation #

# development version
library(devtools)
install_github("jknowles/merTools")

# CRAN version -- coming soon
install.packages("merTools")

Shiny App and Demo #

The easiest way to demo the features of this application is to use the bundled Shiny application which launches a number of the metrics here to aide in exploring the model. To do this:

devtools::install_github("jknowles/merTools")
library(merTools)
m1 <- lmer(y ~ service + lectage + studage + (1|d) + (1|s), data=InstEval)
shinyMer(m1, simData = InstEval[1:100, ]) # just try the first 100 rows of data

On the first tab, the function presents the prediction intervals for the data selected by user which are calculated using the predictInterval function within the package. This function calculates prediction intervals quickly by sampling from the simulated distribution of the fixed effect and random effect terms and combining these simulated estimates to produce a distribution of predictions for each observation. This allows prediction intervals to be generated from very large models where the use of bootMer would not be feasible computationally.

On the next tab the distribution of the fixed effect and group-level effects is depicted on confidence interval plots. These are useful for diagnostics and provide a way to inspect the relative magnitudes of various parameters. This tab makes use of four related functions in merTools: FEsim, plotFEsim, REsim and plotREsim which are available to be used on their own as well.

On the third tab are some convenient ways to show the influence or magnitude of effects by leveraging the power of predictInterval. For each case, up to 12, in the selected data type, the user can view the impact of changing either one of the fixed effect or one of the grouping level terms. Using the REimpact function, each case is simulated with the model’s prediction if all else was held equal, but the observation was moved through the distribution of the fixed effect or the random effect term. This is plotted on the scale of the dependent variable, which allows the user to compare the magnitude of effects across variables, and also between models on the same data.

Predicting #

Standard prediction looks like so.

predict(m1, newdata = InstEval[1:10, ])
#>        1        2        3        4        5        6        7        8
#> 3.146336 3.165211 3.398499 3.114248 3.320686 3.252670 4.180896 3.845218
#>        9       10
#> 3.779336 3.331012

With predictInterval we obtain predictions that are more like the standard objects produced by lm and glm:

#predictInterval(m1, newdata = InstEval[1:10, ]) # all other parameters are optional
predictInterval(m1, newdata = InstEval[1:10, ], n.sims = 500, level = 0.9,
                stat = 'median')
#>         fit      lwr      upr
#> 1  3.074148 1.112255 4.903116
#> 2  3.243587 1.271725 5.200187
#> 3  3.529055 1.409372 5.304214
#> 4  3.072788 1.079944 5.142912
#> 5  3.395598 1.268169 5.327549
#> 6  3.262092 1.333713 5.304931
#> 7  4.215371 2.136654 6.078790
#> 8  3.816399 1.860071 5.769248
#> 9  3.811090 1.697161 5.775237
#> 10 3.337685 1.417322 5.341484

Note that predictInterval is slower because it is computing simulations. It can also return all of the simulated yhat values as an attribute to the predict object itself.

predictInterval uses the sim function from the arm package heavily to draw the distributions of the parameters of the model. It then combines these simulated values to create a distribution of the yhat for each observation.

Plotting #

merTools also provides functionality for inspecting merMod objects visually. The easiest are getting the posterior distributions of both fixed and random effect parameters.

feSims <- FEsim(m1, n.sims = 100)
head(feSims)
#>          term        mean      median         sd
#> 1 (Intercept)  3.22673524  3.22793168 0.01798444
#> 2    service1 -0.07331857 -0.07482390 0.01304097
#> 3   lectage.L -0.18419526 -0.18451731 0.01726253
#> 4   lectage.Q  0.02287717  0.02187172 0.01328641
#> 5   lectage.C -0.02282755 -0.02117014 0.01324410
#> 6   lectage^4 -0.01940499 -0.02041036 0.01196718

And we can also plot this:

plotFEsim(FEsim(m1, n.sims = 100), level = 0.9, stat = 'median', intercept = FALSE)

We can also quickly make caterpillar plots for the random-effect terms:

reSims <- REsim(m1, n.sims = 100)
head(reSims)
#>   groupFctr groupID        term        mean      median        sd
#> 1         s       1 (Intercept)  0.15317316  0.11665654 0.3255914
#> 2         s       2 (Intercept) -0.08744824 -0.03964493 0.2940082
#> 3         s       3 (Intercept)  0.29063126  0.30065450 0.2882751
#> 4         s       4 (Intercept)  0.26176515  0.26428522 0.2972536
#> 5         s       5 (Intercept)  0.06069458  0.06518977 0.3105805
#> 6         s       6 (Intercept)  0.08055309  0.05872426 0.2182059

plotREsim(REsim(m1, n.sims = 100), stat = 'median', sd = TRUE)

Note that plotREsim highlights group levels that have a simulated distribution that does not overlap 0 – these appear darker. The lighter bars represent grouping levels that are not distinguishable from 0 in the data.

Sometimes the random effects can be hard to interpret and not all of them are meaningfully different from zero. To help with this merTools provides the expectedRank function, which provides the percentile ranks for the observed groups in the random effect distribution taking into account both the magnitude and uncertainty of the estimated effect for each group.

ranks <- expectedRank(m1, groupFctr = "d")
head(ranks)
#>      d (Intercept) (Intercept)_var       ER pctER
#> 1 1866   1.2553613     0.012755634 1123.806   100
#> 2 1258   1.1674852     0.034291228 1115.766    99
#> 3  240   1.0933372     0.008761218 1115.090    99
#> 4   79   1.0998653     0.023095979 1112.315    99
#> 5  676   1.0169070     0.026562174 1101.553    98
#> 6   66   0.9568607     0.008602823 1098.049    97

Effect Simulation #

It can still be difficult to interpret the results of LMM and GLMM models, especially the relative influence of varying parameters on the predicted outcome. This is where the REimpact and the wiggle functions in merTools can be handy.

impSim <- REimpact(m1, InstEval[7, ], groupFctr = "d", breaks = 5,
                   n.sims = 300, level = 0.9)
impSim
#>   case bin   AvgFit     AvgFitSE nobs
#> 1    1   1 2.787033 2.801368e-04  193
#> 2    1   2 3.260565 5.389196e-05  240
#> 3    1   3 3.561137 5.976653e-05  254
#> 4    1   4 3.840941 6.266748e-05  265
#> 5    1   5 4.235376 1.881360e-04  176

The result of REimpact shows the change in the yhat as the case we supplied to newdata is moved from the first to the fifth quintile in terms of the magnitude of the group factor coefficient. We can see here that the individual professor effect has a strong impact on the outcome variable. This can be shown graphically as well:

library(ggplot2)
ggplot(impSim, aes(x = factor(bin), y = AvgFit, ymin = AvgFit - 1.96*AvgFitSE,
                   ymax = AvgFit + 1.96*AvgFitSE)) +
  geom_pointrange() + theme_bw() + labs(x = "Bin of `d` term", y = "Predicted Fit")

Here the standard error is a bit different – it is the weighted standard error of the mean effect within the bin. It does not take into account the variability within the effects of each observation in the bin – accounting for this variation will be a future addition to merTools.

This is a major update including removing little used functions and renaming and restructuring functions.

New Functionality #

• A new package vignette is now included
• nth_max function for finding the nth highest value in a vector
• retained_calc now accepts user specified values for sid and grade
• destring function deprecated and renamed to makenum to better reflect the use of the function
• crosstabs function exported to allow the user to generate the data behind crosstabplot but not draw the plot

Deprecated #

• dropbox_source deprecated, use the rdrop2 package
• plotForWord function deprecated in favor of packages like knitr and rmarkdown
• mapmerge2 has been deprecated in favor of a tested mapmerge
• mosaictabs.labels has been deprecated in favor of crosstabplot

Bug Fixes #

• nsims in gelmansim was renamed to n.sims to align with the arm package
• Fixed bug in retained_calc where user specified sid resulted in wrong ids being returned
• Inserted a meaningful error in age_calc when the enddate is before the date of birth
• Fixed issue with age_calc which lead to wrong fraction of age during leap years
• lag_data now can do leads and lags and includes proper error messages
• fix major bugs for statamode including faulty default to method and returning objects of the wrong class
• add unit tests and continuous integration support for better package updating
• fix behavior of max_mis in cases when it is passed an empty vector or a vector of NA
• leading_zero function made robust to negative values
• added NA handling options to cutoff and thresh
• Codebase is now tested with lintr to improve readability

From the vignette:

"caretEnsemble has 3 primary functions: caretList, caretEnsemble and caretStack. caretList is used to build lists of caret models on the same training data, with the same re-sampling parameters. caretEnsemble andcaretStack are used to create ensemble models from such lists of caret models. caretEnsemble uses greedy optimization to create a simple linear blend of models and caretStack uses a caret model to combine the outputs from several component caret models."

I am excited about this package because the ensembling features in caretEnsemble are used to provide additional predictive power in the Wisconsin Dropout Early Warning System (DEWS). I’ve written about this system before , but it is a large-scale machine learning system used to provide schools with a prediction on the likely graduation of their middle grade students. It is easy to implement and provides additional predictive power for the cost of some CPU cycles.

Additionally, Zach and I have worked hard to make ensembling models *easy*. For example, you can automatically build lists of models – a library of models – for ensembling using the caretList function. This caretList can then be used directly in either the caretEnsemble or caretStack mode, depending on how you want to combine the predictions from the submodels. These new caret objects also come with their own S3 methods (adding more in future releases) to allow you to interact with them and explore the results of ensembling – including summary, print, plot, and variable importance calculations. They also include the all important predict method allowing you to generate predictions for use elsewhere.

Zach has written a great vignette that should give you a feel for how caretEnsemble works. And, we are actively improving caretEnsemble over on GitHub . Drop by and let us know if you find a bug, have a feature request, or want to let us know how it is working for you!