I have a fundamental reasoning error, which I simply can't solve.
Given the Problem to find the best solution for a nonlinear least square Problem, given m functions $f_{1...m}(\bf{x})$ with n parameters $\bf{x}=x_{1...n}$ where m>n. So, we have the two column vectors $f$ with dimension $(1,m)$ and x $(1,n)$.
Now, we want to find the best solution for x by iteratively improving our parameter guess at iteration step (s+1): $x^{(s+1)}$ using the jacobian matrix $J$:
$$x^{(s+1)} = x^{(s)} +J^*f(x^{(s)})$$ $$J^* = (J^TJ)^{-1}J^T$$
Now, the result of that equation should be dimension $(1,n)$ as we have n parameters. However, we have $f(x^{(s)})$, which is $(1,m)$. And for the Jacobian $J$ we have dimension $(m,n)$. Making $J^TJ$ to $(n,n)$ and $J^* = (J^TJ)^{-1}J^T$ has $(n,m)$. Resulting in $(1,m)$ where we should have $(1,n)$.
I certainly misunderstood something, but what?
You seem to be using a convention where the first index denotes the number of columns and the second index denotes the number of rows. In this convention the Jacobian $J$ is a $(n,m)$ matrix apposed to $(m,n)$ as written. See for example Wiki:Jacobian and remember that the Wiki-page uses the opposite convention so when it's written $m\times n$ that means $(n,m)$ in your convention. If you redo the calculation using this you should find that it results in a dimension of $(1,n)$ for $(J^TJ)^{-1}J^T$ which agrees with that of $x$ (a column vector).
In the alternative convention $n\times m$ where $n$ is the number of rows and $m$ the number of columns we have that $f$ is a $m\times 1$ matrix, $x$ is a $n\times 1$ matrix and $J$ is a $m\times n$ matrix making $J^TJ$ a $n\times n$ matrix and $(J^TJ)^{-1}J^T$ a $n\times m$ matrix which finally leads to $(J^TJ)^{-1}J^Tf$ being a $n\times 1$ matrix (a column vector) which agrees with the dimensions of $x$.