Sipser has a proof this theorem that goes like this:
$$D = \text{"On input } w$$
$$1. \text{Let } n \text{ be the length of } w$$
$$2. \text{Compute } f(n) \\ \text{using space constructibility and mark off this much tape. If later stages ever attempt to use more, } reject$$
$$3. \text{If } w \text{ is not of the form } <M>10^* \text{ for some TM } M, reject.$$
$$4. \text{Simulate } M \text{ on } w \text{ while counting the number of steps used in the simulate. If the count ever exceeds } 2^{f(n)}, reject\text{"}.$$
$$5. \text{If } M \text{ accepts,} reject. \text{If } M \text{ rejects}, accept.$$
So I see get the contradiction if we say that some Turing Machine $M$ decides the language $A$, the language that is decided by $D$, which runs in $o(f(n))$ space.
But what happens when you run $D$ on itself . Doesn't $5$ give you a contradiction? If $D$ accepts, reject, and if $D$ rejects, accept. So wouldn't you get a contradiction assuming that following TM $D$ is a decider of $A$?
The theorem that we're trying to prove is the Space-Time hierarchy. That is, there is a Language A that is decided by TM $M$ $\in$ space (O(f(n)) but not o(f(n)). see:https://en.wikipedia.org/wiki/Space_hierarchy_theorem
Trying to figure out which theorem you are referring to; in any case, the answer below is only based on what you wrote in your question, i.e. the outline of the proof of this theorem, without the corresponding statement, which I assume is the Space Hirerarchy Theorem (Theorem 2 in these notes).
If you reach Step 4, the input $w\in\{0,1\}^\ast$ is of the form $w=\langle M \rangle x$ for some valid Turing Machine $M$ and $x\in\in\{0,1\}^\ast$.
Step 4 then runs $M$ on $w$ within the bound of $t$ time steps. Your question is: what if $M=D$?. But this is still valid: then you simulate the execution of $M=D$ on input $w$. Your issue is "what if we reach Step 5": but that will not happen.
In Step 4, you get to simulate $D$ on $w$, which costs some number of steps. Then in the simulation, you also starts to simulate (within the simulation) another execution of $D$ on $w$, etc. At each such "subsimulation," whose starting costs at least a few time steps in the original run of $D$, you start a new simulation in step 4.
In other terms, Step 4 causes (the very first execution of) $D$ to loop. So Step 4 of the "original simulation" will run out of time (reach the bound $t$) and reject. Nothing ever reaches Step 5.