So $G=\{f(x)=ax+b|a,b \in\mathbb R,a\neq0\}$, I want to prove that it is a subgroup of the group Sym$_\mathbb R$ consisting of all bijective maps from $\mathbb R$ to $\mathbb R$.
I am aware the subgroup test states the following:
For a non empty subset $H ⊆ G$ the following are equivalent:
(i)$ H $ is a subgroup of $G$,
$(ii)$ $hh' ∈ H$ and $h^{-1} ∈ H$ for all $h, h' ∈ H$,
$(iii)$ $hh^{−1} ∈ H$ for all $h, h^{-1} ∈ H$
So now when I try to apply it, from my understanding $G$ is a subgroup of Sym$_\mathbb R$ if $(i)$ and $(ii)$ are true.
However, doesn't that mean I need to show the following is true for the second condition:
$(ax_1+b)(ax_2+b)$ must create a bijective function.
$\Rightarrow$ $a^2(x_1x_2)+ab(x_1+x_2)+b^2= K$
But how can this be true, if quadratic equations dont have an inverse thus cannot be bijective?
First, you say correctly the conditions are equivalent. Thus if one is true all are true. You need to show (i) is true. But to do this you can show (ii) is true. You can also show (iii) is true, if you prefer. (You can do this instead. To do it in addition is redundant.) Note though that there is an error in (iii). It should be $h h'^{-1} \in H$ for all $h,h' \in H$.
Then, you use the wrong operation. The term $hh'$ means the "product" of the two elements $h,h'$ according to the group law in question, which is composition of maps.
Your $h$ is a maps that takes $x $ to $ax+b$. Your $h'$ is a map that takes $x$ to $a'x+b'$.
So $h h' $ is a map that takes $x$ to $a(h'(x))+b = a(a'x+b')+b$. Some further simplification will show it is of the desired form.
Likewise $h^{-1}$ is the inverse as a function. To determine it consider $y = ax+b$ and 'solve' for $x$ to find that the inverse function is the map that takes $y$ to $(y-b)/a = a^{-1}y + a^{-1}b$. So is indeed of the desired form.
To be precise, you might want to start of your argument with asserting that $x$ to $ax+b$ indeed is a bijective function. That would correspond to asserting that $H$ even is a subset of $G$.