I have tried to calculate the gradient of the LMS cost function as follows but have a problem.
$$J(\theta) = \frac12(y - X'\theta)^2$$ where $y$ is a scalar, theta and $X$ is a $n$ dimensional vector and $X'$ denotes tranpose of $X$.
gradient wrt $\theta$ according to me should be $(y - X'\theta)\theta$.
Is this correct? I have a doubt because opening the bracket results in an exression which involves multiplication of $X'\theta \,\theta$ which is not possible. Where have I made the mistake ?
The gradient of $J(\theta)=\frac12 (y-X'\theta)^2$ is given by
$$\text{grad }J(\theta)=(y-X'\theta)X$$
It's of course a vector because $\theta$ is a vector too. If you don't see how it works, just compute the element-wise derivatives first.