How does bayesian regression differs from maximum likelihood regression?

Question

I know that the basic difference between the two is that in maximum likelihood approach, parameter vector (say $w$) is considered a constant but in bayesian we make use of prior probability which also helps us illustrate the uncertainty in the predicted value of $w$. But I don't understand the general difference between the two as I am a beginner in the field of machine learning and stats. Like in maximum likelihood approach , we solve for $w$ and predict result of new inputs (say $x$) as $x^* w$. What do we solve for in bayesian approach and how new values are predicted ? Also how does bayesian regression automatically chooses the model comolexity and avoids the problem of overfitting ?

Answer 1

I think when you said 'prior probability', you meant 'posterior probability'.

In a Bayesian approach, you infer a posterior probability distribution for the parameters $w$, so your prediction is an average $E(x*w)$ where $E$ is taken over the posterior distribution. In the maximum likelihood approach you infer a point estimate $\hat w$ and your prediction is $x*\hat w .$

(This distinction isn't as important when everything's linear like this... In the Bayesian case, since $E(x*w) = x*E(w),$ we might as well say that we have a point estimate $\hat w = E(w)$ and that our prediction is $x*\hat w.$ For nonlinear models, the distinction is important.)

You're right that the posterior probability distribution gives a sense of the uncertainty in the predicted value of the parameters. However, in the maximum likelihood method, you can also get a sense of uncertainty in the prediction by confidence intervals/regions.