How do I find the amount of possible ordered partition of $n$, given set of positive integer $S$? Here's an example,
With $n = 4$ and $S = \{1, 3, 4\}$, we should have $4$, as $(1,1,1,1)$, $(1, 3)$, $(3, 1)$, and $(4)$ are the solutions.
I've seen this page but didn't have a clue how he transformed the equation to functions of $x$. Neither do I know the name of the transformation technique nor what is $x$.
In your example, you're looking for the coefficient of $x^4$ in the generating function $1/(1-x-x^3-x^4) $. The general case is similar. In particular, in this way you can obtain asymptotics as well as an explicit formula for the number of representations of $n$ as an ordered partition with parts $1,3,4$. The idea is to decompose the generating function into "partial fractions".
I believe that all roots of $1-x-x^3-x^4$ are distinct, and so we can write $$\frac{1}{1-x-x^3-x^4}=\sum_{i=1}^{4} \frac{b_i}{1-a_ix}, $$ from which we can read the formula $\sum_{i=1}^4 b_ia_i^n$ for the number of partitions of $n$.