One of my professor's lecture notes on Vector Spaces start by the following lines:-
We have seen that if $det(A)$ = 0, then system $AX=O$ has infinite number of solutions. We shall now see that in this case, the set of solutions has a structure called vector space.
My doubt is in what sense do the set of an infinite number of the solution of equation $AX = O$ (given |A|=0) is actually a structure of Vector Space? How does the term Vector Space come into picture?
What your professor means is that the set of all $X$'s that satisfy $AX=0$ satisfies the properties of a vector space. You have to check all the ten axioms of a vector space. Here is only a few:
It has an identity under vector addition. Namely, $X=(0,0,\cdots,0)^t$.
It's closed under addition because $A(X+Y)=AX+AY=0$ and if $X,Y$ are in the set, so is $X+Y$
It's closed under scalar multiplication because $A(\lambda X)=\lambda AX=0$.
And you can check all other axioms one by one. Another way to avoid checking many of the axioms, is that if you already have shown that $\mathbb{F}^n$ is a vector space for any field $F$. Then, since our set is a subset of that, associativity and many other properties are inherited from the superset naturally. And we only need to show that $\alpha X + \beta Y$ is in the set which follows from what we proved about being closed under addition and scalar multiplication.