I would really like to understand how Spike-and-Slab Priors work in relation to Linearized Models. Can somebody provide a toy example of a Spike-and-Slab Prior with a Bernoulli spike and a Gaussian slab?
Here's my generic Linear Model: $$ y = \mathbf{\beta*x} + \epsilon $$ or written as $$ y = \sum_{j=1}^{k}\beta_j*x_j + \epsilon $$
I understand in Bayesian Inference, one has a posterior distribution that $$\mathbf{\theta} = [\beta_1, \beta_2, ..., \beta_j]$$ $$P(\theta|x) = \frac{P(x|\theta)*P(\theta)}{P(x)}$$
which is essentially Likelihood * Prior / Normalizer.
In this, each parameter (i.e. each $\beta_j$) has it's own distribution and I would like to fit the parameters ($\theta$) to the data ($x$) while the data is fixed and the parameters are random variables of their own distributions (btw, is that the same as $P(\theta_j)=P(x|\theta_j)*P(\theta_j)/P(x)$ with indicies?).
If any of this is incorrect, please correct me :) It would be greatly appreciated
My reference:
The paper I am reading uses Bayesian Model Averaging with a Spike-and-Slab Prior. The Spike-and-Slab Prior uses "consists of a Bernoulli distribution that specifies whether or not a variable is selected as influential. The slab prior is a Gaussian distribution that models the effect sizes of the variables, conditional on their being chosen as influential" Shankar et al. BMC Bioinformatics (2015) 16:31 DOI 10.1186/s12859-015-0467-6. To figure out what a Spike-and-Slab Prior was, so I can use this w/o any black-box R package, I checked out: https://www2.stat.duke.edu/courses/Fall05/sta395/joelucas1.pdf where the description of the algorithm is below.
My confusion:
I don't understand how this is related to Bernoulli and Gaussian distributions. Is the $h_{0j}$ a random variable following Bernoulli distribution and $h_{1j}$ following Gaussian ? If $f$ has to be extremely large, does the value of it matter and why does their need to be a separate value for every index $f_j$? I found this resource on the stats stackexchange (https://stats.stackexchange.com/questions/142818/is-a-spike-and-slab-prior-a-proper-prior) but I'm not sure what $F(x)$ is in the figure; it's definitely not a p.d.f. since it doesn't integrate to 1.
My apologies if this is long, but I've been looking up resources for over a week and it's explained a little differently everywhere I've looked; it's difficult to piece together the pieces w/o a toy example.
