Let's say $f(x)$, and $g(x)$ are two completely defined functions.
My understanding of composition's existence is that the domain of $g(x)$ must be a subset of the domain of $f(g(x))$, is that correct?
If yes, what two functions I can think of that will result in a composition that does not exist?
Thank you.
The complete description of a function $f$ contains its domain $A = \operatorname{dom} f$ and its codomain $B = \operatorname{codom} f$ which are two sets. Therefore we should always write $f : A \to B$ instead of $f(x)$. The latter is inadequate because it does not make precise for which $x$ the function is defined and in which set its values $f(x)$ are supposed to lie.
In some cases it may be acceptable to write simply $f(x)$, for example if we exclusively deal with functions having $\mathbb R$ as domain and codomain. In such cases it is automatically clear what $\operatorname{dom} f$ and $\operatorname{codom} f$ are.
Note that we must not confuse $\operatorname{codom} f$ with its image $\operatorname{im} f = f(A) = \{ f(a) \mid a \in A \} \subset B$; we may have $\operatorname{im} f \subsetneqq B$.
The codomain of a function is frequently also called its range.
The standard definition of the composition of functions $f : A \to B$ and $g: B '\to C$ requires $B = B'$. Then we get $$g \circ f : A \to C, (g \circ f)(x) = g(f(x)) . \tag{1}$$
We can obviously extend this definition to the case that $B \subset B'$ or, even more generally, to the case that $$\operatorname{im} f \subset B' \tag{2} .$$
No. First of all we do not know the domain of $g(f(x))$ before we have a definition of the composite function. However, it is clear that the domain of $g(f(x))$, whatever its definition may be, must be subset of the domain of $f$; otherwise $f(x)$ would not be defined. But even if we would know the domain $A' \subset A$ of $g(f(x))$, this set has nothing to do with the domain $B'$ of $g$. Look at $(1)$ and $(2)$.
We can also define the composite function for any two functions $f : A \to B$ and $g: B '\to C$ via $$g \circ f : f^{-1}(B \cap B') \to C, (g \circ f)(x) = g(f(x)) . \tag{3}$$ This works because for $x \in f^{-1}(B \cap B')$ we have $f(x) \in B \cap B' \subset B'$ so that we can use $f(x)$ as input for $g$.
This definition may lead to $\operatorname{dom} (g \circ f) = \emptyset$ which is completely uninteresting.