Partial derivative of a matrix multiplied by a vector wrt matrix

Question

Given a matrix $A$ and and two vectors x and b, what is the gradient of $(A\cdot x-b)^2$ with respect to $A$? (I am trying to find the matrix which best sustains a given linear equation using gradient descent)

Answer 1

Don't get what you try to do, but probably that is solution what are you looking for:

$F = (A\cdot x-b)^2$

$\frac{\partial F}{\partial A} = 2 \cdot (A\cdot x - b) \cdot x^T$

That is extremely look like backpropagation algorithm in Artificial Neural Networks, where $b$ target, $A$ neuron synapse, $x$ is your input, $F$ is a square error function and you differentiation is a gradient for backpropagation algorithm for network without hidden layer and linear function ($f(x) = x$) as activation one.

Backpropagation algorithm in classic interpretation use gradient descent rule and your question similar to this one. One difference that you try differentiate with respect to $A$, not to $x$ as at Wikipedia example.