Let $h$ be a linear mapping of $V_1 \rightarrow V_2$
\begin{gather*} h^0 = \frac{1}{g} \sum_{t \in G}(\rho_t^2)^{-1}h\rho_t^1 \end{gather*}
(1) if $\rho^1$ and $\rho^2$ are not isomorphic, then $h^0 = 0$
(2) if $V_1 = V_2$ and $\rho^1 = \rho^2, h^0$ is a homothety of ratio $(1/n)$Tr$(h)$, $n = \dim(V_1)$
Looking at the matrix forms, we let $\rho_t^1 = (r_{i_1j_1}(t)), \rho_t^2 = (r_{i_2j_2}(t))$, and define $h = (x_{i_2i_1}), h^0 = (x_{i_2i_1}^0)$, so \begin{gather*} x_{i_2i_1}^0 = \frac{1}{g}\sum_{t,j_1,j_2}r_{i_2j_2}(t^{-1})x_{j_2j_1}r_{j_1i_1}(t) \end{gather*}
Serre says "The right hand side is a linear form with respect to $x_{j_2j_1}$; in case (1) this form vanishes for all systems of values of the $x_{j_2j_1}$; thus it's coefficients are zero."
Okay, cool - I can follow you, Serre. Here is where I get confused. He then says, directly by case (1), \begin{gather*} \frac{1}{g}\sum_{t,j_1,j_2}r_{i_2j_2}(t^{-1})r_{j_1i_1}(t) = 0 \end{gather*} Here, I don't see why we lose $x_{j_2j_1}$, especially if it's supposed to have coefficients that are zero.
Later on he says "equating the coefficients of $x_{j_2j_1}$ gives us \begin{gather*} \frac{1}{g}\sum_{t,j_1,j_2}r_{i_2j_2}(t^{-1})r_{j_1i_1}(t) ... \end{gather*} and, again, I don't see why we lost $x_{j_2j_1}$ - or specifically why it takes on value 1?
Since case (1) says $\rho_1$ and $\rho_2$ are not isomorphic, the equality $$ 0 =x^0_{i_2i_1} = \frac{1}{g} \sum_{g, j_1, j_2} r_{i_2j_2}(t^{-1})x_{j_2j_1}r_{j_1i_1}(t) $$ holds for any map $h$, i.e., it holds for any system of values $x_{j_2j_1}$. So let's pick a map $h$ for which $x_{j_2j_1} = 1$ for each pair $(j_2,j_1)$: the matrix with $1$ in each component.
For the second question (now in the case where $h^0 = \lambda$), and he says "equating coefficients of the $x_{j_2j_1}$...", the main question, I think, is "why can we just equate coefficients?" Again we use that the equality $$ \frac{1}{g} \sum_{t, j_1, j_2} r_{i_2j_2}(t^{-1}) x_{j_2j_1}r_{j_1i_1}(t) = x^0_{i_2i_1} = \lambda\delta_{i_2i_1} = \frac{1}{n} \sum_{j_1,j_2} \delta_{i_2i_1}\delta_{j_2 j_1}x_{j_2j_1} \tag{$\ast$}$$ holds for any system of values of $x_{j_2j_1}$, i.e., for any map $h$. Fix $x_{j_{2_0}j_{1_0}}$. First let $h$ be the map that has $x_{j_{2_0}j_{1_0}}= 1$ and $0$ elsewhere. Then the equality ($\ast$) reduces to $$ \frac{1}{g}\sum_{t\in G} r_{i_1j_{2_0}}(t^{-1}) r_{j_{1_0}i_1}(t) = \frac{1}{n} \delta_{i_2i_1}\delta_{j_{2_0} j_{1_0}}.$$ Hence for all $(j_1,j_2)$ $$ \frac{1}{g}\sum_{t\in G} r_{i_1j_2}(t^{-1}) r_{j_1i_1}(t) = \frac{1}{n} \delta_{i_2i_1}\delta_{j_2 j_1}$$