I just noticed that my pythons scipy library gave me another solution to an exponential curve fit than my pocket calculator.
If you plot both solutions both match the data (visually) very well. This lead me to the question of what can be said about existence and uniqueness of least square fits.
So consider a set of finite points $(x_i,y_i), i \in \{1,\dots,n\}, n \in \mathbb{N}$. And a function $f\colon D \supset \mathbb{R} \to \mathbb{R}$ which depends on parameters $a_j, j \in \mathbb\{1,\dots,k\}, k \in \mathbb{N}$.
Now one wants to find the parameters $(a_j)$ such that $\sum_i (f(x_i) -y_i)^2$ is minimal.
What results are there about existence and uniqueness of $f$?
I am interested in particular in the case that $f$ is an elementary function such as
- polynomial function of the form $f(x) = \sum_l a_l x^l$
- exponential function of the form $f(x) = a_0 \cdot a_1^x$
- something reciprocial like $f(x) = a_0 + a_1/x^m$
- logarithmic function like $f(x) = a_0 + a_1 \ln(x)$
If in some of those cases the solution doesn't exist or is not unique, I am looking for simple counterexamples. I am also interested in good references (which includes the proofs) about this special topic.
The dependence of $f$ on $x$ is not very important, it's the dependence on the parameters $a_j$ for each fixed $x$ that counts. In particular, in each of your cases except the second $f$ is linear in the $a_j$, and then $\sum_i (f(x_i)-y_i)^2$ is convex in the vector $\bf a$. Moreover, unless the $f(x_i)$ are linearly dependent, it is strictly convex, which implies that a minimum is unique, and $\sum_i (f(x_i)-y_i)^2 \to \infty$ as $\|\bf a\| \to \infty$, which implies that the minimum exists.