If $\langle A,B\rangle=\sum_{i=1}^n\sum_{j=1}^n a_{ij}b_{ij}$, how do I solve
$$\langle xx^T,A^TA\rangle$$?
Is there a shortcut for doing this? or should I expand each entry and do the inner product?
I got $\|Ax\|^2_2$, but wondering if there is any other ways to do it.
You're not going to find a cleaner formula than $\|Ax\|_2^2.$
You can derive it in a coordinate-free manner by using the formula for Frobenius product in terms of trace: $$A:B = \operatorname{tr}(A^TB).$$ So then $$xx^T : A^TA = \operatorname{tr}(xx^TA^TA) = \operatorname{tr}(x^TA^TAx) = (Ax) : (Ax) = \|Ax\|^2_2$$ where the key second step uses invariance of trace under cyclic permutations.