Show that each function\begin{align}f(x)=\dfrac{ax+b}{cx+d}, g(x)=\dfrac{ex+f}{jx+h}\end{align} can be represented by matrices such that the matrix representation of the composition $g(f(x))$ is the product of the matrices of $g$ and $f$.
I tried expressing $\frac{ax+b}{cx+d}$ to $\frac{a}{c}+\frac{bc-da}{c(cx+d)}=ac^2x+bc+cda-da$. Which made me think might $x$ be a constant, vector, etc...? I would like to as for hints on this.
On
Hint: Try evaluating $g(f(x))$.
By the way, in general it is not true that $$\frac{a}{c}+\frac{bc-da}{c(cx+d)} = ac^2 x + bc + cda - da.$$
As hinted in the comments by Ethan, try searching for Möbius Transformation.
You need to know some group theory up to the first isomorphism theorem to understand the matrix representation.
The map from $\mathrm{SL}(2,\Bbb F)$ to the group of all Möbius transformations, sending $\begin{bmatrix}a & b\\ c & d\end{bmatrix}$ to the Möbius transformation $f$, where $f(x)=\frac{ax+b}{cx+d}$, is a surjective group homomorphism, with the kernel of this map being $\{I,-I\}$. So by the first isomorphism theorem, $\mathrm{SL}(2,\Bbb F)/\{I,-I\}\cong$ Möbius transformation group. The induced isomorphism tells us that $\begin{bmatrix}a & b\\ c & d\end{bmatrix}\{I,-I\}$ represents the function $f$. But we may drop $\{I,-I\}$ and simply say that $\begin{bmatrix}a & b\\ c & d\end{bmatrix}$ represents $f$.
For details about the above claims with proofs, consult Alan Beardon's Algebra and Geometry, the chapter on Möbius transformations.