Within this AoPS thread there was the following question asked
Let $\phi(X,Y)=X^TAY$ be a scalar product on $\mathbb R^3,$ and let $f:\mathbb R^3\to \mathbb R^3, X\mapsto BX$, where $$A=\begin{pmatrix} 1 & 0 & 0 \\ 0 & 2 & t^2-2 \\ 0 & t^2-2 & 2 \end{pmatrix},~B=\begin{pmatrix} 0 & t^2 & t^2 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \end{pmatrix},~t\in \mathbb{R}$$ Prove that $\phi(f(X),Y)=\phi(X,f(Y))~\forall X,Y\in\mathbb R^3.$
Utilizing basic properties of the transpose it can be deduced that on the one hand we got
$$\phi(f(X),Y)=(BX)^TAY=X^TB^TAY=X^T(B^TA)Y$$
whereas on the other hand one got that
$$\phi(X,f(Y))=X^TA(BY)=X^T(AB)Y$$
so it remains to show that $B^TA=AB$. Noting that $A=A^T$ we can conclude that $B^TA=B^TA^T=(AB)^T$ and thus the problem boils down to showing that $(AB)^T=AB$, i.e. the product matrix of $A$ and $B$ is symmetric. The product can be computed fairly easy and so we finally obtain
$$A\cdot B=\begin{pmatrix}1 & 0 & 0 \\ 0 & 2 & t^2-2 \\ 0 & t^2-2 & 2\end{pmatrix}\cdot \begin{pmatrix}0 & t^2 & t^2 \\ 1 & 0 & 0 \\ 1 & 0 & 0\end{pmatrix}=\begin{pmatrix}0 & t^2 & t^2 \\ t^2 & 0 & 0 \\ t^2 & 0 & 0\end{pmatrix}$$
The latter matrix is indeed symmetric which proves the given statement.
However, I am curious whether there exists a method not demanding to actually compute the product of $A$ and $B$, maybe only relying on the fact that $A=A^T$. So far I was only able to deduce that $B^T=ABA^{-1}$ or similiar $B=A^{-1}B^TA$ but I am not sure whether that is helpful at all.
Thanks in advance!
Axioms of inner product:
Let $V$ be a $\mathbb C$-vector space (or $\mathbb R$, but it's only necessary to define it on $\mathbb C$). $\phi:V\times V \to \mathbb C$ is an inner product, if $$ \forall(x,y_1, y_2 \in V, \alpha_!, \alpha_2\in \mathbb C) \phi(x, \alpha_1y_1+\alpha_2y_2) = \alpha_1\phi(x,y_1) + \alpha_2\phi(x, y_2) $$ $$ \forall (v \in V-\{0\}) 0 < \phi(v, v) \in \mathbb R $$ $$ \forall (x, y \in V) \phi(x, y) = \overline{\phi(y, x)} $$
Suppose that $\phi(x,y) = x^TAy$ is an inner product - it satisfies first axiom (easy to check), satisfies third axiom (we're working with real numbers here, and matrix $A$ is symmetric, but depending on $t$, second axiom might be violated, ex. - for $t=0$ and vector $v=(0, 1, 1)$, $\phi(v,v)=0$. Therefore, suppose that we're working with $t$ that gives valid inner product.
Let $$ A' = \left(\begin{array}{rrr} 1 & 0 & 0 \\ 0 & 2 & t^{2} - 2 \\ 0 & t^{2} - 2 & 3 \end{array}\right) $$ If $x^TAy$ is inner product, then $x^TA'y$ is an inner product too - axioms second and third remain unchanged. For such $t$, that $x^TAy$ was valid inner product, it can be verified that $x^TA'y$ is valid inner product too (applying Sylvester's criterion).
The proof you mentioned doesn't depend on the elements of matrices $A$ and $B$ (it just uses the fact that $A$ is symmetric, and explois some properties of inner product). So it should remain valid for product given by $x^TA'y$. This however is false, because $A'B$ is not symmetric, as can be checked by computation. This means that you need to actually look at the elements of both matrices (and probably do the multiplication).