Let $A \in GL_n(\mathbb{Z}_p)$. We consider the maximum norm $\| \cdot \|$ on $M_{n \times n}(\mathbb{Q}_p)$, which coincides with the operator norm with respect to the maximum norm on $\mathbb{Q}_p^n$. If $\|A^2 - I\|$ is small enough, can I find some $B \in GL_n(\mathbb{Z}_p)$ such that $B^2 = I$ and $\|A - B\|$ is small? That is, if $A$ is almost of order 2, is $A$ close to an element of order $2$?
This is true for $U(n)$ with the norm induced from $\mathbb{C}^{n^2}$ (notice that $U(n)$ is maximal compact in $GL_n(\mathbb{C})$ just as $GL_n(\mathbb{Z}_p)$ is maximal compact in $GL_n(\mathbb{Q}_p)$). It can be proven by first diagonalizing $A$ over $U(n)$ and then choosing $B$ diagonal with entries $\pm 1$ according to the sign of the real parts of the corresponding entries of $A$. So diagonalization is used, and the order on the reals, both of which do not pass over to the $p$-adics.
The answer is yes, at least for odd $p$. It follows from the following theorem of Brawley from "Similar Involutory Matrices (mod $p^m$)":
Theorem: Let $p$ be an odd prime, $m \geq 1$, and let $A$ be an integral matrix such that $A^2 = I \mod p^m$. Then there exist integral matrices $P, Q$ and $1 \leq t \leq n$ such that $P$ and $Q$ are inverse $\mod p^m$ and $PAQ = J_t \mod p^m$, where $J_t = diag(I_t, -I_{n-t})$.
Now $A^2 = I \mod p^m$ is equivalent to $\|A^2 - I\| \leq p^m$. The fact that $P$ is invertible $\mod p^m$ implies that a representative is in $GL_n(\mathbb{Z}_p)$ and so $\|PAP^{-1} - J_t\| \leq p^m$ too. Choosing $B = P^{-1}J_tP$ and using that the norm is invariant under $GL_n(\mathbb{Z}_p)$, we conclude.
A similar theorem is available for $p = 2$, see the last section of "Involutory matrices over finite commutative rings" by Brawley and Gamble, although the canonical form is more complicated than just $J_t$. I did not spend much time on it (I am happy with $p$ odd for the time being) but I believe it should still work.