If we have a function $\rho:SL(2,\mathbb{Z})\rightarrow GL(d,\mathbb{C})$ where $m\in SL(2,\mathbb{Z})$ and $\mathcal{M}\in GL(d,\mathbb{C})$ and d is given, for example d=3.
And we know that $\rho:s\rightarrow \mathcal{S}$ and $\rho:t\rightarrow\mathcal{T}$ where $s,t,\mathcal{S},\mathcal{T}$ is also given. More precisely $s$ and $t$ are the generators of the $SL(2,\mathbb{Z})$ group.
\begin{equation} t=\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}\qquad\qquad\qquad s=\begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix} \end{equation}
\begin{gather} m=s^{\alpha_{1}}t^{\beta_{1}}s^{\alpha{2}}t^{\beta_{2}}...s^{\alpha{n}}t^{\beta_{n}}\\ \mathcal{M}=\mathcal{S}^{\alpha_{1}}\mathcal{T}^{\beta_{1}}\mathcal{S}^{\alpha{2}}\mathcal{T}^{\beta_{2}}...\mathcal{S}^{\alpha{n}}\mathcal{T}^{\beta_{n}} \end{gather}
If $H$ is a normal subgroup of $SL(2,\mathbb{Z})$ and it has finite index in $SL(2,\mathbb{Z})$ and it is the kernel of $\rho$ then why does it mean that the image of $\rho$ also will be finite (i.e. the order of the group)?
If I understand your question correctly, it is just because, by the first isomorphism theorem, you will have an isomorphism $$ \bar{\rho} : SL(2,\mathbb{Z})/H \to \text{image of $\rho$}. $$ The first group is finite, thus so is the second.
I am assuming $\rho$ is a homomorphism with $H = \ker(\rho)$.