Ok so I think I know why this is incorrect, because of the following:
$$\det\frac{1}{ad-bc}\begin{bmatrix} d & -b\\ -c & a \end{bmatrix}\neq \frac{ad-bc}{ad-bc}$$
However, by adding a det in the middle it would be correct: $$\det(\frac{1}{ad-bc}det(\begin{bmatrix} d & -b\\ -c & a \end{bmatrix}))= \frac{ad-bc}{ad-bc}$$ And thus: $$\det\frac{1}{ad-bc}\begin{bmatrix} d & -b\\ -c & a \end{bmatrix}=\begin{bmatrix} \dfrac{d}{(ad-bc)} & \frac{-b}{(ad-bc)}\\ \frac{-c}{(ad-bc)} & \frac{a}{(ad-bc)} \end{bmatrix}$$
However the answer is:
I do not understand the divides twice by $ad-bc$ bit, is there a rule I've misinterpreted somewhere?
Answer, I just want to be clear for others having trouble:
$$\det\frac{1}{ad-bc}\begin{bmatrix} d & -b\\ -c & a \end{bmatrix}=\left ( \frac{1}{ad-bc} \right )^{2}det\begin{bmatrix} d & -b\\ -c & a \end{bmatrix}=\left ( \frac{1}{(ad-bc)^{2}} \right )(ad-bc)=\frac{ad-bc}{(ad-bc)^{2}}=\frac{1}{ad-bc}$$
If you multiply a constant $c$ to a $n \times n$ matrix $A$, the determinant is multiplied by $c^n$, i.e.
$$ \det(cA) = c^n \det(A) $$
Now $c = \frac{1}{ad - bc}$ and $n = 2$. The result follows directly.