I am trying to study least square and linear regression and I understand the solution for $Ax = b$ when x is the unknown and the LS solution is given by $(A^TA)^{-1}A^TA$. Now, I was wondering if something similar exists when A is unknown. I was wondering if I do the corresponding:
$$ Ax=b $$ $$ x^TA^T = b $$
Therefore, $A^T = (xx^T)^{-1}xb$ or $A = b^Tx^T ((xx^T)^{-1})^T$
On
You are basically asking for a linear transformation that maps one vector $x$ to another vector $b$. With no other constraints, there are infinitely many answers except in the special case of one dimension.
A linear mapping $A:\mathbb{R}^{n} \rightarrow \mathbb{R}^{n}$ requires $n$ linearly independent equations in order to be fully specified. This means you would need $n$ equations $Ax_1=b_1,Ax_2=b_2,...,Ax_n=b_n$ (or more simply $AX=B$ where $X$ and $B$ are $n\times n$ matrices) in order to get a unique answer.
On
If $x$ is zero, an $A$ exists (if and) only if $b=0$ and in this case, $A$ can be chosen arbitrarily. If $x$ is nonzero, then any $A$ such that $Ax=b$ is given by $$\tag{1} A=bx^\dagger+Z(I-xx^\dagger), $$ where $Z$ is any matrix of the same size as $A$ and $x^\dagger=x^*/\|x\|_2^2$. You may verify that $$ Ax=b(x^\dagger x)+Z(I-xx^\dagger)x=b+Z(x-x(x^\dagger x))=b+Z(x-x)=b $$ using $x^\dagger x=x^*x/\|x\|_2^2=1$. The form (1) follows simply from recasting $Ax=b$ as a linear system for the matrix $A$ and from the fact that any solution to $Bz=c$ is given by $z=z_0+w$, where $z_0$ is any solution to $Bz=c$ and $w$ is in the kernel of $B$.
Since $xx^T$ is a rank one matrix, it is not invertible and $(xx^T)^{-1}$ does not exist. Your system is therefore underdetermined.
The best way to obtain a (pseudo-)unique solution is either to replace $(xx^T)^{-1}$ by the Moore-Penrose pseudoinverse of $x^Tx$ or to use a regularization procedure (like the Tikhonov one, where you would replace $(xx^T)^{-1}$ by $(xx^T+\lambda I)^{-1}$ with $\lambda > 0$).