Let there be two functions $f(x)=\sqrt{x}$ and $g(x)=\sqrt{2-x}$.
So, $g(f(x))=\sqrt{2-\sqrt{x}}$.
As evident the domain of $g(x)$ is $(-\infty, 2]$ and the domain of $g(f(x))$ is $[0,4]$. But from what I understand about composite functions, the domain of $g(f(x))$ must be a subset of domain of $g(x)$. So according to that, the domain of $g(f(x))$ should be $[0,2]$.
Which one is the correct domain of $g(f(x))$?
The domain of the function $g(f(x))$ is not always a subset of the domain of $g(x)$. It depends on the specific functions $g(x)$ and $f(x)$. Instead, the domain of $g(f(x))$ must be a subset of the domain of $f(x)$, because before we check if $f(x)$ is in the domain of $g(x)$, we must first check that $x$ is in the domain of $f(x)$.