Given that a set of orthonormal basis $\psi$ is constructed by a set of non-orthogonal basis $\chi$ by the following relation: $$\psi_{\mu}=\sum^{\mu}_{i=1}t_{i\mu}\chi_{i} \tag{1}$$
The matrix representation of an operator in the orthonormal bases: $$A=\sum_{\mu \nu} |\psi_{\mu}\rangle a_{\mu \nu}\langle\psi_{\nu}| \tag{2}$$ Is the the following relation correct? $$\bar{a}_{ij}=\sum_{\mu \nu}t_{i \mu }a_{\mu \nu}t^{\dagger}_{\nu j} \tag{3}$$ where $a_{\mu \nu}$, $\bar{a}_{ij}$ are components of the matrix representation of the operator $A$ in the orthonormal basis and Non-orthogonal basis respectively.
Proof:(?)
From (2): $$A=\sum_{\mu \nu}|\sum^{\mu}_{i=1}t_{i \mu}\chi_i \rangle a_{\mu \nu} \langle \sum^{\nu}_{j=1}t_{j\nu} \chi_{\nu}|\overbrace{=}^{\sum^{\mu}_{i=1} \to \sum_i \text{since } t_{i \mu}=0 \text{ for } i>\mu}\sum_{\mu \nu}|\sum_{i}t_{i \mu}\chi_i \rangle a_{\mu \nu} \langle \sum_{j}t_{j\nu} \chi_{j}|$$ $$=\sum_{ij}| \chi_i\rangle \underbrace{\left(\sum_{\mu \nu}t_{i \mu}a_{\mu \nu} t^{\dagger}_{\nu j} \right)}_{\bar{a}_{ij}}\langle\chi_j| \tag{4}$$
However, the textbook I'm reading (Arfken Mathematical Methods For Physicists ,Example 5.6.1 pg293 Gram-Schmidt Transformation) states that $$a_{\mu \nu}=\sum_{ij }t^{\dagger}_{\mu i}\bar{a}_{ij}t_{j \nu} \tag{5}$$ and in general $t_{ij} \iff T, T^{\dagger} \neq T^{-1}$ which seems to contradict (3) because (3)$\to$ (5) via the relation $T^{-1}=T^{\dagger}$.
What have I done wrong?
Just for completeness sake, your equation (5) is derived just like you tried to prove equation (3): $$ \langle\psi_\mu,A\psi_\nu\rangle=\Big\langle\sum_it_{i\mu}\chi_i,A\sum_jt_{j\nu}\chi_j\Big\rangle=\sum_{i,j}t_{i\mu}^\dagger\langle\chi_i,A\chi_j\rangle t_{j\nu} $$ As for your actual question: the problem is what you try to read out from equation (4); given a (non-orthonormal basis) $(v_i)_i$ and a linear map $B$ on the inner product space---with representation $(b_{ij})_{i,j}$ w.r.t. $(v_i)_i$---one in general does not have $B=\sum_{i,j}b_{ij}|v_i\rangle\langle v_j|$. This is only the case, in general, if $(v_i)_i$ is orthonormal in which case the transformation matrix $T$ is unitary, that is, $T^{-1}=T^*$, as you pointed out.
Let us make this more clear via a simple example. Consider the linear map $A:\mathbb C^2\to\mathbb C^2$, $(x,y)\mapsto(2x+3y,x)$ so its representation matrix with respect to the standard basis $\mathcal B_1=\{\begin{pmatrix}1\\0\end{pmatrix},\begin{pmatrix}0\\1\end{pmatrix}\}$ reads, as expected, $$ A_{\mathcal B_1}=\begin{pmatrix}2&3\\1&0\end{pmatrix}\,. $$ Now let us change the basis to the (non-orthonormal) one $\mathcal B_2=\{\begin{pmatrix}1\\0\end{pmatrix},\begin{pmatrix}1\\1\end{pmatrix}\}$ meaning the transformation matrix is $T=\begin{pmatrix}1&1\\0&1\end{pmatrix}$. Thus $$ A_{\mathcal B_2}=T^{-1}A_{\mathcal B_1}T=\begin{pmatrix}1&-1\\0&1\end{pmatrix} \begin{pmatrix}2&3\\1&0\end{pmatrix}\begin{pmatrix}1&1\\0&1\end{pmatrix}=\begin{pmatrix}1&4\\1&1\end{pmatrix}\,. $$ One can also verify this by hand, i.e. $Av_1=v_1+v_2$ and $Av_2=4v_1+v_2$. And this is where the problem arises: decomposing $A_{\mathcal B_2}$ in terms of the $|v_i\rangle\langle v_j|$ yields \begin{align} A_{\mathcal B_2}&=(-3)\cdot\begin{pmatrix} 1&0\\0&0\end{pmatrix}+3\cdot\begin{pmatrix} 1&1\\0&0 \end{pmatrix}+0\cdot\begin{pmatrix} 1&0\\1&0 \end{pmatrix}+1\cdot\begin{pmatrix} 1&1\\1&1 \end{pmatrix}\\ &=-3|v_1\rangle\langle v_1|+3|v_1\rangle\langle v_2|+|v_2\rangle\langle v_2|\,. \end{align} These coefficients obviously do not match the entries of $A_{\mathcal B_2}$. In other words, the problem in your equation (4) is that in general there is no (simple) relation between the entries of a representation matrix and its expansion coefficients with respect to a basis---unless of course it is orthonormal.
Edit: While this is not part of your mistake I think it nonetheless is worth pointing out that both the representation matrix and the expansion coefficients are again different from the "usual" matrix elements $\langle v_i,Av_j\rangle$ (in the orthonormal case); again take our example: $$ (\langle v_i,Av_j\rangle)_{i,j}=\begin{pmatrix} 2&5\\3&6 \end{pmatrix}\not\in\Big\{\begin{pmatrix}1&4\\1&1\end{pmatrix},\begin{pmatrix}-3&3\\0&1\end{pmatrix}\Big\} $$