Let $H$ be the multiplicative group of matrices of the following form for $a,b,c \in \mathbb{F}_{p}$ (the finite field of $p$ elements, for $p$ prime)
$$ M(a,b,c) = \begin{bmatrix} 1 & a & c\\ 0 & 1 & b \\ 0 & 0 & 1 \end {bmatrix} $$
Then it is not too difficult to see that $A = M(1,0,0)$ and $B = M(0,1,0)$ generate $H$, and further if $C = ABA^{-1}B^{-1} = M(0,0,1)$, then we can see that we have the general formula:
$$ M(s,t,u) = B^{t}C^{u}A^{s} $$
Then we let $V = \mathbb{C}^{\mathbb{F}_{p}}$ be the vector space of complex valued functions on $\mathbb{F}_{p}$. Then, for $\lambda$ any root of $x^{p}-1$ in $\mathbb{C}$, define $\rho_{\lambda} : H \rightarrow \operatorname{GL}(V)$ to be such that:
$$ \rho_\lambda(A) : f(x) \mapsto f(x-1) \hspace1cm \rho_\lambda(B) : f(x) \mapsto \lambda^{x}f(x) $$
Then $\rho_\lambda$ is a representation of $H$, and I'm concerned about when $\rho_\lambda$ is irreducible.
In what follows I will outline my method and my current progress. I have tried to read around the subject, but since I have only taken an introductory representation theory class, and have not yet studied lie-algebras I am struggling to understand the available material. I would appreciate if someone could verify the validity of my solution, and more importantly the efficacy of my approach, and potentially suggest better ways of tacking the problem.
I can show fairly easily that when $\lambda = 1$ we have the $\rho_1$-invariant subspace of $V$ given by $W = \{ f : \mathbb{F}_{p} \rightarrow \mathbb{C} \mid f(x) \equiv \alpha \ \text{for some} \ \alpha \in \mathbb{C} \} $. Hence when $\lambda = 1$, $\rho_{1}$ is reducible.
My aim then to show that when $\lambda \neq 1$ we have that $\rho_{\lambda}$ is irreducible. To do this I am going to attempt to calculate the inner product of the trace of $\rho_\lambda$, denoted $\chi_\lambda$, with itself and show that it is equal to 1.
To do this we shall use the basis of $V$ given by $\{e_{n} : \mathbb{F}_{p} \rightarrow \mathbb{C} \mid x \mapsto \delta_{n,x}\}_{n=0}^{p-1}$ (where $\delta$ is the Kronecker-delta function).
Then we see that the following relations hold:
\begin{align*} \rho_\lambda(A^{s}) (e_{n}(x)) & = e_{n}(x-s) = \delta_{n,x-s} = \delta_{n+s,x} = e_{n+s}(x) \\ \rho_\lambda(B^{t}) (e_{n} (x))& = (\lambda^{tx})e_{n}(x) \\ \rho_\lambda(C) (e_{n} (x)) & = \rho_\lambda(A) \rho_\lambda(B) \rho_\lambda(A^{-1})\rho_\lambda(B^{-1}) (e_{n} (x)) = \lambda^{-1}e_{n}(x) \\ \rho_\lambda(C^{u}) (e_{n} (x)) & = \lambda^{-u}e_{n}(x) \end{align*}
Hence we see that: $$ \rho_\lambda(M(s,t,u))(e_{n}(x)) = (\lambda^{tx - u})e_{n+s}(x) $$
Hence, if $s \neq 0$ we see that $\chi_{\lambda}(M(s,t,u)) = 0$. Now with the added assumption that $\lambda \neq 1$ we see that for $t\neq 0$, $\chi_{\lambda}(M(0,t,u)) = 0$ since:
$$ \chi_{\lambda}(M(0,t,u)) = \sum_{i=0}^{p-1} \lambda^{ti - u} = \lambda^{-u} \sum_{i=0}^{p-1} (\lambda^{t})^{i} = \lambda^{-u}\sum_{i=0}^{p-1} \lambda^{i} = 0 $$
Where the penultimate equality is true since $p$ is prime we have that $\xi \mapsto \xi^{l}$ is an element of the Galois group of $(x^{p} - 1)$ over $\mathbb{Q}$ for every $l \neq 0$, and the last equality is true since $\lambda$ is a root of $x^{p} - 1$ that is not $1$, and so $\lambda$ is a root of $ (x^{p} - 1)/(x-1) = \sum_{i=0}^{p-1} x^{i}$.
Then we have that $\chi_{\lambda}(M(0,0,u)) = \sum_{i=0}^{p-1} (\lambda^{-u}) = p\lambda^{-u}$.
Finally we see that the pairing of $\chi_\lambda$ with itself is given by:
\begin{align*} \left< \chi_\lambda , \chi_\lambda \right> & = \frac{1}{\left| H \right|} \sum_{g \in H } \chi_\lambda(g) \overline{\chi_\lambda(g)} = \frac{1}{p^{3}} \sum_{0\leq s,t,u\leq p-1} \chi_\lambda(M(s,t,u)) \overline{\chi_\lambda(M(s,t,u))} \\ & = p^{-3} \sum_{u=0}^{p-1} \chi_\lambda(M(0,0,u)) \overline{\chi_\lambda(M(0,0,u))} \\ & = {p^{-3}} \sum_{u=0}^{p-1} (p\lambda^{-u}) \overline{(p\lambda^{-u})}\\ & = {p^{-3}} \sum_{u=0}^{p-1} p^{2}\lambda^{-u} \overline{\lambda^{-u}} \\ & = {p^{-1}} \sum_{u=0}^{p-1} \lambda^{-u} \lambda^{u} = {p^{-1}} \sum_{u=0}^{p-1} (1) = 1 \end{align*}
Where the penultimate line is true since $\lambda$ is a root of unity, so in particular has absolute value of $1$, and so $\overline{\lambda} = \lambda^{-1}$ thence $\overline{\lambda^{-u}} = \lambda^{u}$.
Hence we may conclude that, for $\lambda \neq 1$, $\rho_\lambda$ is irreducible.