I used the following formula to answer the question;
$$\frac{((x∗k)+1)}{n},$$
where $x=(1,2,3...N)$. If the result of the formula is an integer then that result is the inverse to n mod k. In this case n=5 and k=26. So I found that when $x=1$ the result is $5.4$. When $x=2$ the result is $10.6$. When $x=3$ the result is $15.8$. When $x=4$ the result is $21$. 21 is an integer so $5∗21=1 \space mod\space 26.$
Can anyone explain whether $N=26$ in this case, because we are dealing with Mod $26$?
On
No, $N \neq 26$ in your example.
Every number $n$ coprime to the modulus $k$ has exactly one unique inverse $i$ in the range $1 \leq i \leq k-1$.
Imposing those bounds, we get:
$$1 \leq \frac{xk + 1}{n} \leq k-1$$
$$\lceil \frac{n-1}{k}\rceil \leq x \leq \lfloor\frac{n(k-1)-1}{k}\rfloor$$
So in your case, you only had to test $x = 1, 2, 3, 4$.
To make things slightly simpler, you can just take the lower bound as $1$ and the upper bound as $n$. This will be fine for small values of $n$, where you don't do too much "extra" work.
I would point you toward the Extended Euclidean Algorithm as a much more efficient way of calculating the modular inverse. A nice implementation is the 'magic box' method.
Yes, $N=26$ in your case, and in general $N=k$ in your algorithm.
On a side remark, you could have quickly noticed that $5\times 5 \equiv 25 \equiv -1$ so $5 \times (-5) \equiv 1$ mod $26$.
On the general case I would recommend using the extended Euclidean algorithm rather than the method you described for calculating inverses as it is significantly faster and deep.