This post picks up with Fisher’s idea of the plug-in principle. As with the last post, the point of emphasis is on the connection between maximum likelihood estimates and the bootstrap.

The plug-in principle

Part of Fisher’s genius is his practicality, and the plug-in principle is a prime example of his problem solving tactics. Let’s say that we’re doing a linear regression problem

$$y_i = \beta^T x_i + \varepsilon_i \stackrel{\textsf{notation}}{=} y = X\beta + \varepsilon \iff \E [y | x] = X\beta \stackrel{\textsf{notation}}{=} \E [y_i | x_i] = \beta^T x_i,$$

where $\beta \in \reals^d$ is a vector of parameters to be estimated based off of $n$ observations.

Fisher style

We’re interested in the standard error of the $k$th component of the estimated coefficient vector $\hat{\beta}$

$$\mathsf{se} \big( \hat{\beta}_k \big) = \sigma_\varepsilon \sqrt{(X^T X)^{-1}_{kk}}.$$

Now, here’s where we go Fisherian on this problem, use our plug-in estimates, and make sure that everybody who should be wearing a hat is wearing a hat

$$\widehat{\mathsf{se}} \big( \hat{\beta}_k \big) = \hat{\sigma}_\varepsilon \sqrt{(X^T X)^{-1}_{kk}}.$$

All these quantities can be estimated based on the data $(X,y)$, which means that we’re in business.

Efron style

Now, let’s look at the bootstrap version of these estimates:

$$\widehat{\mathsf{se}}_\infty \big( \hat{\beta}_k \big) = \hat{\sigma}_\varepsilon \sqrt{(X^T X)^{-1}_{kk}},$$

where we perform $B = \infty$ bootstrap samples. This results follows the fact that the variance of a bootstrap estimate $\hat{\beta}^\star$ is

$$\Var\left( \hat{\beta}^\star \right) = (X^T X)^{-1} X^T \Var(y^\star) X (X^T X)^{-1} = \hat{\sigma}^2_\varepsilon (X^T X)^{-1}.$$

After we take the square root, we’re done.

Take-aways

The bootstrap gives us the same standard error estimates as Fisher’s plug-in estimates, but it does so by reversing the order of operations.

MLE world: compute, then plug in.

• Compute an approximate formula for the standard error as a function of the unknown parameters.
• Plug in the estimates.

Bootstrap world: plug in, then compute.

• Plug in estimates for the unknown parameters.
• Compute standard errors via bootstrap sampling (i.e, simulation).

So, where Fisher was clever in his mathematics, the bootstrap is clever in exploiting cheap computation. Moreover, the bootstrap allows us to side step some (potentially nasty) mathematics via (lots of) computation.

Next up

… confidence intervals!

References

This post is related to material from:

• “R.A. Fisher in the 21st Century” by Bradley Efron.
• Computer Age Statistical Inference: Algorithms, Evidence, and Data Science by Bradley Efron and Trevor Hastie. A digital copy lives here: CASI.
• An Introduction to the Bootstrap by Bradley Efron and Robert J. Tibshirani.