Let $p$ be a fixed prime and $\mathbb Z_p$ the ring of $p$-adic integers. Consider the subgroup $G_n\subseteq \operatorname{GL}_n(\mathbb Z_p)$ given by all matrices $(a_{ij})_{ij}$ such that $$ \begin{align*} v(a_{ij}) &\ge 1 &&\text{if $i>j$}\\ v(a_{ii}-1)&\ge1 &&\text{for all $1\le i\le n$}, \end{align*} $$ where $v$ denotes the usual $p$-adic valuation of $\mathbb Z_p$, i. e. $v(x) = r$ if $x\in p^r\mathbb Z_p\setminus p^{r+1} \mathbb Z_p$ and $v(0) = \infty$.
If $n\ge p-1$, find a matrix of order $p$ in $G_n$.
This problem appears as an exercise in Peter Schneider’s book on $p$-adic Lie groups, example 23.3, which shows that $G_n$ carries a $p$-valuation if and only if $n<p-1$.
The condition $n\ge p-1$ is necessary, since a matrix $A$ of order $p$ has only $p$-th roots of unity as its eigenvalues and hence $f(X) = X^{p-1}+X^{p-2}+\dotsb+ X+1$ has to divide the characteristic polynomial of $A$. Hence, the companion matrix of $f(X)\cdot (X-1)^{n-p+1}= (X^p-1)(X-1)^{n-p}$ gives a matrix of order $p$ in $\operatorname{GL}_n(\mathbb Z_p)$ (which even has entries in $\mathbb Z$).
This is, where I got stuck. For $p=3$, I found the matrix $\begin{pmatrix}-2 & -1\\ 3 & 1\end{pmatrix}$ by making the ansatz $\begin{pmatrix}1+3a & b\\ 3c & 1+3d\end{pmatrix}$ and deriving from the characteristic polynomial $X^2+X+1$ the conditions $$ \begin{align*} a+d = -1\\ bc = 3ad-1, \end{align*} $$ which has four solutions in $\mathbb Z$; in $\mathbb Z_3$, $(a,b)$ can be freely chosen from $\mathbb Z_3\times \mathbb Z_3^\times$.
Let us try this following the observations you make in your well-thought out post:
For a ring $R$, it is often useful to think of elements of $GL_n(R)$ as being automorphisms of the $R$-lattice $R^n$. This will help motivate the following construction, which is very standard.
Edit: To get the example that you have worked out, you should take $\theta = \zeta_3 + 1$ instead. I suppose that this $\theta$ is another possible choice in the general scenario too.