Consider the matrix $X = \alpha I$ where $I$ is the $n \times n$ identity matrix and $\alpha$ is a real number.
If $\alpha \to \pm 0$ does $X^{-1} \to \pm \infty I$? Similarly, if $\alpha \to \infty$ does $X^{-1} \to 0$?
Since the inverse of $X$ (for $n=2$) is given by $$ X^{-1} = \begin{bmatrix} \frac{1}{\alpha} & 0 \\ 0 & \frac{1}{\alpha} \end{bmatrix} $$
This seems to be the case because $$ \lim_{\alpha \to 0^{\pm}} \frac{1}{\alpha} = \pm \infty$$ $$ \lim_{\alpha \to \infty} \frac{1}{\alpha} = 0$$
I am unsure if this statement is true because if $\alpha=0$ or $\alpha=\infty$, the matrix is not invertible, but since we are computing the limit, but this is okay.
As I said in the comments, standard matrix algebra doesn't allow for scaling by $\infty$. It doesn't mean that someone couldn't perform non-standard operations with matrices, but the convention simply doesn't take you where you want to go. And, as is often the case, when you wish to go off the beaten path, you need to be extra careful you don't fall and hurt yourself.
In the standard analysis of matrices, we simply don't permit $\pm \infty$ to be in an of the entries. We will get to why in a second, but let's just accept, for now, that the convention doesn't allow it.
We would say that $\lim_{\alpha \to 0} X^{-1}_\alpha$ doesn't exist, even if the limit were one-sided. This is because there is no limiting matrix $A$ that becomes "arbitrarily close" to $X^{-1}_\alpha$ as $\alpha$ becomes very small. Eventually, the matrix $X^{-1}_\alpha$ is going to grow larger (in norm) than any one given matrix that you nominate. As it blows past every matrix, it has no limit. It would be better to say $\lim_{\alpha \to 0}\|X^{-1}_\alpha\| = \infty$, i.e. the limit of the norms of these matrices tends to $\infty$, making it a limit along the real line, instead of a divergent limit of matrices.
So what, you may ask, is the deal with the real line? The real line allows $\infty$ and $-\infty$ limits. Yes, $\infty$ and $-\infty$ are not strictly real numbers, but we let them slide, right? Why don't we let it slide in matrices?
Well, let's not forget that considering $\pm \infty$ alongside real numbers is not without its costs. Sure, we can define, intuitively, $\infty + 1 = \infty$, or $\infty \div 2 = \infty$, and other such things. But this tends to break all sorts of rules that we take for granted in the real numbers. For example, we have the cancellation law: $$a + c = b + c \implies a = b.$$ But, if we allow $c = \infty$, and $a$ and $b$ are finite numbers, then our intuition leads us to believe that $a + c = \infty$ and $b + c = \infty$, and thus $a + c = b + c$, but there's no reason to believe that $a$ and $b$ are equal (e.g. one could be $0$ and the other could be $1$). So, allowing $\infty$ into our algebra of $+$ and $\times$ can very quickly start breaking our established intuition about real numbers.
Then, of course, we have indeterminate operations with $\infty$ and $-\infty$. What is $\infty \div \infty$? Or $\infty - \infty$? Or $1^\infty$? Suddenly we have to tread lightly when even proposing operations between extended real numbers, as holes in our domain suddenly appear.
This is all compounded when it comes to matrices. What would be the product $$\begin{pmatrix} 1 & \infty \\ 0 & 1 \end{pmatrix}\begin{pmatrix} 1 & -\infty \\ 0 & 1\end{pmatrix}?$$ The only entry that doesn't come up indeterminate is the bottom left entry!
We put up with $\pm \infty$ in real limits because they have enough utility to warrant the effort. There are a great many circumstances where it is very helpful to have an artificial maximum/minimum on the real line, but we always need to keep an eye on it, to make sure it doesn't start multiplying itself to $0$, or something like that.
Dealing with $\pm \infty$ is kept only to specific circumstances and controlled situations, and real limits is one of those situations. They're also sometimes used in other areas, like the theory of convex functions, again where it becomes worth the hassle to introduce them into the system. It's not worth doing it with matrices.
What you can say safely is that the diagonal entries of $X_\alpha^{-1}$ tend to $\pm \infty$, as $\alpha \to 0^{\pm}$. It's better, again, to bring it back to limits on the real line. That is a useful observation to make about these matrices, and one we can make precise within the well-understood conventions of real limits. It says the same thing as what you're trying to convey when you claim the limit is $$\begin{pmatrix} \pm\infty & 0 \\ 0 & \pm\infty\end{pmatrix},$$ but it's precise, and it's correct.