Here is the theorem that defines definition by recursion in the book of Dirk van Dalen.
Then, the author gives an example.
From my point of view, this example is not transparent. The reason is that for the case of $Sub(\phi_1\square\phi_2)$, we cannot explicitly use $\phi_1$ and $\phi_2$, and can only operate with $Sub(\phi_1)$ and $Sub(\phi_2)$. It seems it would be correct to define it in this way, for example: $$ Sub(\phi_1\square\phi_2) = Sub(\phi_1)\cup Sub(\phi_2)\cup\{\mathtt{argmax_{\psi_1\in Sub(\phi_1)}\mathtt{rank}(\psi_1)}\square\mathtt{argmax_{\psi_2\in Sub(\phi_2)}\mathtt{rank}(\psi_2)}\} $$ We have the same problem defining $Sub(\neg\phi)$.
Am I right? If yes, then it is bizarre that the author hasn't emphasized that point. I wouldn't say it isn't worth stressing, especially for beginners.