I have that I need to start with the number I'm trying to represent $2\pmod {5^n}$ and then keep finding the next solution to $x^4-1=0$ that's equivalent to what I have already mod $5^{n+1}$
So the sequence I have so far is $2,7,57,182,2057,\ldots$
The coefficients of $5^n$ added at each step are in turn $5^0\cdot2+5^1\cdot1+5^2\cdot2+5^3\cdot1+5^4\cdot3\ldots$ so I feel like this could be written as a base $5$ sequence $\ldots31212_5$ which looks a lot like a $5$-adic number.
I wouldn't be able to prove it but Wikipedia says the fact this is a Hensel lifting process guarantees precisely one solution for each successive $n$ - so I presume $w(2)$ is an infinite sequence and a $5$-adic number. What form does it take and to what does it converge?
The answer I'm expecting is that it has infinitely many nonzero terms, it's not eventually periodic and that it converges to some number which can only really be thought of as a $4^{th}$ root of unity, and which is different to a complex $4^{th}$ root of unity in the sense that you could add any choice of complex $4^{th}$ root to $\Bbb Z_5$ and it remain independent from $w(2)$. Does all that sound right and is there anything else of fundamental importance to be aware of? (Apart from the premise I started with, which is that these form an alternative radix to $\{0,1,2,3,4\}$?)
What you are converging to is a square root of $-1$. The $5$-adic integers have a bit of notoriety because $5$ is the smallest $p$ for which $\mathbb{Z}_p$ has such roots.
Because $\mathbb{Z}_5[x]$ has Unique Factorisation, any solution of $x^4-1=0$ which is the polynomial you use to construct this equation must solve one of the factor equations $x-1=0,x+1=0,x^2+1=0$. Clearly your representation with a $\bmod 5$ residue of $2$ misses the first two equations and so must solve the third. The equation $x^2+1=0$ also will have a root ending with $3$ in $5$-adics, which is of course the additive inverse of your root ending with $2$.
We might be tempted to call one root $i$ and the other $-i$. But no such identification is inherently built into the numbers because the $5$-adic roots live in a different domain from the complex numbers. What we can do, however, is use either of these roots in place of $i$ or $-i$ and thus "map" complex number relationships into $5$-adic relationships. For instance:
$(1\pm i)^2=2×(\pm i)$
$[1+(...212)]^2=2×(...212)$ or $[1+(...233)]^2=2×(...233)$
As a final note, the question asks whether the representation for this square root of $-1$ eventually becomes periodic. That answer is "no". The number $-1$ is not a rational square, and only rational numbers have periodic $p$-adic representations.