Please see attached pages from Nonlinear Systems by Hassan K Khalil,
It is claimed that the loop in the upper H1 can "nullify" the transformations in the down H2. Does it mean that the transfer function of the system remains the same? I try to derive but I can hardly tell that the transfer functions of the subfigures are the same. Specifically, I can tell Fig 6.13 (a) = Fig 6.13 (b), But the rests are not the same.
They are the same in the sense that the input (and output) signals for $H_1$ and $H_2$ are the same. Thus, they are the same in the sense of stability and boundedness (convergence) of the circulating signals. They are not necessarily the same in the sense of input-output transformation, but it is up to the choice of the output.
The idea is that what we want to study is the stability of the system. What does it mean? It means that (in the absence of the reference input, the most left arrow on Fig. (a)) the circulating signals are bounded, or converge to zero, it is up to your definition of stability. What are the circulating signals? From Fig. (a) we see that these signals are the outputs of $H_1$, namely $y_1$, and the output of $H_2$, namely $y_2$. Then the input of $H_1$ is $u_1=-y_2$, and the input of $H_2$ is $u_2=y_1$. So the claim is as follows: if the signals $y_1$ and $y_2$ are bounded (converge), then the system is stable.
Now, the figures (a) and (d) are equivalent in the sense that the input of $H_2$ in (d) is still equal to the output of $H_1$, that is $y_1$, and the input of $H_1$ is still equal to the negative output of $H_2$, that is $-y_2$. It also means that if you simulate the systems (a) and (d) with the same initial conditions, then the signals $y_1$ and $y_2$ will be the same. Thus, if the system (d) is stable, then the system (a) is stable as well.