Difference between optimization and estimation

Question

I fear this question may be stupid, but yesterday I got an email from an engineer with these sentences in it:

Subset simulation is a sampling method for estimating failure probabilities. The task of estimating a failure probability is not an optimization problem; it is an estimation problem.

I as an old idiotic mathematician always thought that every statistical estimation problem (excluding pathological cases) can be formulated as an optimization problem.

Maybe I am wrong?

Can anybody of you enlighten me?

Thanks

Answer 1

In optimization you need an error function to minimize. Depending on the estimation problem it may not be possible to formulate an error function, due to for example fundamentally incomplete information.

E.g. without any further context, assuming you're estimating some ground truth probability $p$, then your squared error function for a guess $x$ would be $f(x) = (p-x)^2$, but to compute this function you need to know $p$ in the first place.

Answer 2

Estimators can be constructed without optimization when you have some framework which lets you analytically derive, rather than search for, your estimator.

Some examples:

• The German tank problem https://en.wikipedia.org/wiki/Estimation_theory#Maximum_of_a_uniform_distribution
• Problems where Monte Carlo methods are the best available solution.

Whereas, you'd call it an optimisation problem if the best known approach is to search for a solution, rather than calculate a solution.

Optimization algorithms often think about the “landscape” of possible solutions. Estimation techniques may not.