Let's say I have the following equation $$AX=B$$ where $A$ is a $8\times 3$ matrix (known), $X$ is a $3\times3$ "diagonal" matrix which represents the coefficients (unknown) and $B$ is a $8\times 3$ matrix (known).
Whenever I solve for $X$ using least squares $X= (A^TA)^{-1} A^T B$, I get a square matrix which is not diagonal.
Any idea how can I force least square solution matrix to be diagonal?
Here is the thing. If $X=diag(x_1,x_2,x_3)$ then the elements of first row of $AX$, i.e. $a_{11}x_1, a_{12}x_2, a_{13}x_3$, are equal to $b_{11}$, $b_{12}$ and $b_{13}$ respectively. Hence $x_1=b_{11}/a_{11}$ and $x_2=b_{12}/a_{12}$...further, the second row gives then $x_1=b_{21}/a_{21}$,...and so forth until the 8th row. So in order for $X$ to be a diagonal it must be that the first column of $B$ is a multiple of the first column of $A$, the second column of $B$ a multiple of the second column of $A$ and likewise for the third (that is $b_{.1}=x_1a_{.1}$, $b_{.2}=x_2a_{.2}$,...). I guess your known matrices do not satisfy that.
I guess what you want to do is to minimize $(AX-B)'(AX-B)$ with respect to $X$ under the necessary side conditions (being 'off diagonal elements of $X$ are zero')
Edit: actually, $X$ being diagonal gives you three equations of the form $b_{.i}=a_{.i}x_i$ for each column $i$. The least square solution is $$x_i=(a_{.i}'a_{.i})^{-1}a_{.i}'b_{.i} $$ for each $i$. Now, solving this way means you are treating the columns as separate equations. If they are related by correlated error terms, google for SUR (seemingly unrelated) equations, to find efficient estimators...