Suppose that $\beta$ is scalar, $x,y$ are column vectors $(m\times 1)$ and $A$ is an $(m\times m)$ matrix. We are looking to differentiate the expression $$\big(x^{T}A - \beta y^{T}\big)\big(x^{T}A - \beta y^{T}\big)^{T}$$ with respect to $x$. My idea:
Let $f(x)=\big(x^{T}A - \beta y\big)\big(x^{T}A - \beta y\big)^{T}$ defined on some appropriate finite dimensional space over $\mathbb{R}.$ We have:
\begin{align} f(x+h) &= \big((x+h)^{T}A - \beta y^{T}\big)\big((x+h)^{T}A - \beta y^{T}\big)^{T} \\ &= \big(\big(x^{T}+h^{T}\big)A - \beta y^{T}\big)\big(\big(x^{T}+h^{T}\big)A - \beta y^{T}\big)^{T} \\ &= \big(\big(x^{T}A -\beta y^{T}\big)+h^{T}A\big)\big(\big(x^{T}A -\beta y^{T}\big)+h^{T}A\big)^{T} \\ &=\big(x^{T}A - \beta y^{T}\big)\big(x^{T}A - \beta y^{T}\big)^{T} + \big(x^{T}A - \beta y^{T}\big)hA^{T} + \big(x^{T}A - \beta y^{T}\big)h^{T}A + hh^{T}A \\ &=f(x) + \big(x^{T}A - \beta y^{T}\big)hA^{T} + \big(x^{T}A - \beta y^{T}\big)h^{T}A + hh^{T}AA^{T}. \end{align}
Hence,
\begin{align} f(x+h)-f(x) &= \big(x^{T}A - \beta y^{T}\big)hA^{T} + \big(x^{T}A - \beta y^{T}\big)h^{T}A + hh^{T}AA^{T} \\ &= \big(x^{T}A - \beta y^{T}\big)hA^{T} + \big(x^{T}A - \beta y^{T}\big)h^{T}A + |h|^{2}AA^{T} \\ &= 2\big(x^{T}A - \beta y^{T}\big)hA^{T} + |h|^{2}AA^{T}, \end{align}
which finally yields (by dividing by $h$ and letting $h\rightarrow 0$) that the derivative is $2\big(x^{T}A - \beta y^{T}\big)A^{T}$. What do you think ?
Your approach is correct. A few errors appear: $$ f(x+h) = f(x) + (x^T A - \beta y^T ) A^T h + h^T A (x^T A - \beta y^T )^T + h^T A A^T h $$ $$ f(x+h) = f(x) + h^T A (x^T A - \beta y^T )^T + h^T A (x^T A - \beta y^T )^T + h^T A A^T h $$ $$ f(x+h) = f(x) + 2 h^T A (A^T x - \beta y) + h^T A A^T h $$ The gradient is $$ \nabla f(x) = 2 A (A^T x - \beta y) $$