⧼exchistory⧽
3 exercise(s) shown, 0 hidden
Jun 25'23

Consider the linear regression model $Y_i = X_i \beta + \varepsilon_i$ with the $\varepsilon_i$ i.i.d. following a standard normal law $\mathcal{N}(0, 1)$. Data on the response and covariate are available: $\{(y_i, x_i)\}_{i=1}^8 = \{ (-5, -2), (0, -1), \\ (-4, -1), (-2, -1), (0, 0), (3,1), (5,2), (3,2) \}$.

• Assume a zero-centered normal prior on $\beta$. What variance, i.e. which $\sigma_{\beta}^2 \in \mathbb{R}_{\gt0}$, of this prior yields a mean posterior $\mathbb{E}(\beta \, | \, \{(y_i, x_i)\}_{i=1}^8, \sigma_{\beta}^2)$ equal to $1.4$?
• Assume a non-zero centered normal prior. What (mean, variance)-combinations for the prior will yield a mean posterior estimate $\hat{\beta} = 2$?
Consider the Bayesian linear regression model $\mathbf{Y} = \mathbf{X} \bbeta + \vvarepsilon$ with $\vvarepsilon \sim \mathcal{N} ( \mathbf{0}_n, \sigma^2 \mathbf{I}_{nn})$ and priors $\bbeta \, | \, \sigma^2 \sim \mathcal{N} ( \mathbf{0}_p, \sigma_{\beta}^{2} \mathbf{I}_{pp})$ and $\sigma^2 \sim \mathcal{IG}(a_0, b_0)$ where $\sigma_{\beta}^{2} = c \sigma^{2}$ for some $c \gt 0$ and $a_0$ and $b_0$ are the shape and scale parameters, respectively, of the inverse Gamma distribution. This model is fitted to data from a study where the response is explained by a single covariate, and henceforth $\bbeta$ is replaced by $\beta$, with the following relevant summary statistics: $\mathbf{X}^{\top} \mathbf{X} = 2$ and $\mathbf{X}^{\top} \mathbf{Y} = 5$.
• Suppose $\mathbb{E}( \beta \, | \, \sigma^2=1, c, \mathbf{X}, \mathbf{Y}) = 2$. What amount of regularization should be used such that the ridge regression estimate $\hat{\beta}(\lambda_2)$ coincides with the aforementioned posterior (conditional) mean?
• Give the (posterior) distribution of $\beta \, | \, \{ \sigma^2=2, c=2, \mathbf{X}, \mathbf{Y} \}$.
• Discuss how a different prior of $\sigma^2$ affects the correspondence between $\mathbb{E} (\beta \, | \, \sigma^2, c, \mathbf{X}, \mathbf{Y})$ and the ridge regression estimator.