Whenever I read some use of the term “orthogonal”, I have been able to find some way in which it is at least metaphorically similar to the idea of two orthogonal lines in euclidean space.
E.g. orthogonal random variables, etc.
But I cannot see how $A^{-1}=A^T$ captures the idea of “orthogonality”. What is “orthogonal” about a matrix that satisfies this property?
On
Beacuase a matrix has that property if and only if all columns are orthonormal. This is also equivalent to the assertion that all rows are orthonormal.
On
If a matrix is orthogonal, the column of the matrix will form an orthonormal system. That is, $\langle v_i,v_j\rangle=\delta_{i,j}$ for two column vectors $v_i$ and $v_j$. Now this implies rather than it is a definition that $AA^T=I$.
On
When this happens, the determinant is $1$ or $-1$, which you can verify explicitly. Not only is the determinant a unit, the transformation preserves orthogonal vectors.
Let $u,v$ be orthogonal vectors. Then the inner product $\langle Au , Av\rangle = \langle u, A^TAv \rangle$. This is true because of the nature of the inner product; I encourage you to explicitly verify this. However, $A^TA = I$, so $$\langle Au, Av \rangle = \langle u, v \rangle$$ Thus, orthogonality has been preserved. This is the real reason why we need $A^T = A^{-1}$ for a matrix to be orthogonal.
On
With $A \in \mathbb{R}^{n\times n}$ and $A^{–1} = A^\top$ we get $$ I = A^{–1} A = A^\top A = (a_1\dotsb a_n)^\top (a_1\dotsb a_n) \iff \\ \delta_{ij} = a_i^\top a_j = a_i \cdot a_j \quad (i, j \in \{1, \dotsc, n \}) $$ So this property is equivalent to the column vectors (or row vectors) forming an orthonormal basis.
Note: The complex case will involve complex conjugation and result in orthogonality regarding the complex scalar product.
On
When you multiply two matrices, the result consists of the dot products of the rows of the first matrix with the columns of the second. If you have $A^{-1}=A^T$, then $A^TA=I$, but this product consists of all of the pairwise dot products of the columns of $A$. All of the products that correspond to dot products of different columns are zero, which is precisely the condition for their orthogonality. Moreover, all of the dot product of columns with themselves are equal to $1$, which means that all of the columns are unit vectors—the columns of $A$ are an orthonormal set of vectors.
On
To expand on Akiva Weinberger's comment - an orthogonal matrix describes a transformation that preserves the geometry. Distances and angles in inner-product spaces are defined by the inner product, and so to say a matrix $A$ defines a geometry-preserving transformation (a "rigid" or "orthogonal" transformation) is to require that for any two vectors $v_1 , v$2 the following holds:
$$(Av_1) \cdot (Av_2) = v_1 \cdot v_2$$ since in matrix-multiplication notation $v \cdot w = v^T w$, and using the fact that $(AB)^T=B^T A^T$ the requirement is:
$$v_1^Tv_2 = (Av_1)^T(Av_2) = v_1^TA^TAv_2$$
which holds true for any $v_1,v_2$ iff $A^T=A^{-1}$.
$A^{-1} = A^T \iff A^T A = I$, so let's explore what this latter property means. (Assume that we are working in a real vector space, specifically $\mathbb{R}^n$.) Write
$$A = \begin{pmatrix} \uparrow & \uparrow & \dots&\uparrow \\ v_1 &v_2&\dots&v_n \\ \downarrow&\downarrow&\dots&\downarrow \end{pmatrix} $$ then $$A^TA = \begin{pmatrix}\leftarrow & v_1 & \rightarrow \\ \leftarrow & v_2 & \rightarrow \\ \vdots & \vdots & \vdots \\ \leftarrow & v_n & \rightarrow \end{pmatrix} \begin{pmatrix} \uparrow & \uparrow & \dots&\uparrow \\ v_1 &v_2&\dots&v_n \\ \downarrow&\downarrow&\dots&\downarrow \end{pmatrix} =\begin{pmatrix} v_1\cdot v_1 & v_1\cdot v_2&\dots&v_1 \cdot v_n \\ v_2\cdot v_1 & v_2\cdot v_2&\dots&v_2 \cdot v_n \\ \vdots & \vdots & \ddots & \vdots \\ v_n\cdot v_1 & v_n\cdot v_2&\dots&v_n \cdot v_n \\ \end{pmatrix}. $$ So, $A^T A = I$ corresponds exactly to $v_i \cdot v_i = 1$ and $v_i \cdot v_j = 0$ for $i \neq j$, i.e. $\{v_i\}$ are an orthonormal system. So, if a matrix is orthogonal then its columns are orthogonal. Similarly, you can see that its rows are also orthogonal, since if $A$ is orthogonal then $A^T$ is also.