I tried applying the algorithm from Wikipedia in order to calculate $-133^{-1}\mod 256$
I spent already time myself finding the mistake but no success. This is how I did go about applying the algorithm:
| r | newr | quotient | t | newt |
|---|---|---|---|---|
| 256 | -133 | a | 0 | 1 |
| -133 | 123 | -1 | 1 | 1 |
| 123 | -10 | -1 | 1 | 2 |
| -10 | 3 | -12 | 2 | 25 |
| 3 | -1 | -3 | 25 | 77 |
| -1 | 0 | -3 | 77 | 256 |
The solution should be $-77$
The loop invariant is $\,-133\, t_k\equiv r_k\pmod{\!256},\,$ so at termination, reading that in the 2nd last row this becomes: $\, -133\cdot 77\equiv -1,\,$ so negating $\,-133(-77)\equiv 1,$ i.e. $\,(-133)^{-1}\equiv -77$.
The Wikipedia algorithm is working with the system of least nonnegative residues, so you could have avoided the need for negation by working in this system, starting with $\,-133\equiv 123$.
The key idea of the algorithm is that $\bmod 256,\,$ given $\,r > r'> 0\,$ both multiples of $-133,\,$ then we can find a smaller multiple $\,r'' := r\bmod r' = r-q\,r'.\,$ Indeed if $\,r \equiv -133t,$ $\,r'\equiv -133t'$ then $\,r'' = r-q\,r' \equiv -133(t-qt')$ remains a multiple of $\,-133$.
The algorithm starts with the two obvious multiples $\,r_0\equiv 0\equiv 256,\,$ and $\,r_1 \equiv -133,\,$ then generates a chain of descending multiples till it reaches $\,0.\,$ The row with the last nonzero $\,r\,$ yields the sought result, as explained in the first paragraph above.
If you use negative residues $\,r_i\,$ then we can achieve descent by ensuring that they have decreasing magnitude, i.e. $\,|r_{k+1}| < |r_k|.\,$ Then the algorithm terminates with the least magnitude multiple $\,r\not\equiv 0\,$ and the gcd is $\,|r|,\,$ and the inverse can be obtained by negating (if need be), as above.
To gain better intuition on this Euclidean descent process (sequence of decreasing multiples) you may find it helpful to view the simpler case of Gauss's algorithm.