I am not sure if i'm understanding this correctly but if I have a map from $T:X \rightarrow Y$ then the adjoint $T^*: Y \rightarrow X$
Is the inverse not defined in the same way? I figure I am missing some assumption or restriction.
I would really appreciate someone clarifying this.
The adjoint $T^*$ is defined by $\langle x,T^*(y)\rangle_X = \langle T(x),y\rangle_Y$ for all $x\in X$ and $y\in Y$.
To answer your follow-up question, you can check, for example, that for $X=Y=\Bbb R^n$, you'll have $T^* = T^{-1}$ if and only if the matrix of $T$ with respect to an orthonormal basis is an orthogonal matrix (i.e., the columns form an orthonormal basis for $\Bbb R^n$).