I am proving that $A$ is singular matrix without assuming that $B$ is orthogonal, following is my method, but I am not sure it is correct and complete.
$A$ and $B$ are real non-zero $3\times3$ matrices and satisfy the equation $$(AB)^T+B^{-1}A=0$$
Questions: Without assuming that $B$ is orthogonal, prove that $A$ is singular.
Second try: $$B(AB)^T=-A$$ $$BB^TA^T=-A=-IA$$ $$det(B)det(B^T)det(A^T)=-det(I)det(A)$$
Since $$det(B)=det(B^T), det(A^T)=det(A)$$
but $$[det(B)]^2 \neq -det(I)$$
therefore, det(A)=0 and A is singular.
(First wrong proof):
Assume $A$ is a non-singular matrix. Then the inverse of $A$ exists.
Let $B=A$, then $$(AA)^T=-A^{-1}A=-I=(-I)^T\Longrightarrow AA=-I,$$ and$$\det(AA)=\det(-I)\Longrightarrow\det(A)\det(A)=-\det(I).$$ From the symbol, two negatives and two positive all make a positive. Therefore, $A$ cannot be a non-singular matrix.
Your question is ambiguous. It is not clear whether $B$ is a given matrix or $(AB)^T+B^{-1}A=0$ for every invertible $B$.
If $B$ is a given matrix, then your proof is wrong because you cannot specify $B$ at will. In fact, the problem statement is false in this case.
Even if the question means that $(AB)^T+B^{-1}A=0$ for every invertible matrix $B$, your proof is correct only when the size of $A$ is odd, because $\det(-I)\ne-\det(I)$ when the size of the matrix is even.
To prove the problem statement, note that if we put $B=I$ into the condition $(AB)^T+B^{-1}A=0$, we get $A^T=-A$. Next, put $B=2I$ instead to get $-2A+\frac12A=0$. Hence $A=0$.
If you want to deliberately avoid using any orthogonal $B$, you may modify the proof as follows. First, rewrite the condition as $(BB^T)A^T=-A$. Now, pick any invertible matrix $P$ that is not orthogonal (such as the upper triangular matrix of ones). Put $B=P$ to get $(PP^T)A^T=-A$. Next, put $B=2P$ to get $4(PP^T)A^T=-A$. Subtract the second equation by the first, we get $3(PP^T)A^T=0$. Since $P$ is invertible, we conclude that $A=0$.