A friend and me tried to solve this problem and we took two different approachs:
while both considered the probability as a quotient between areas, he used an auxiliary function $$g(a)=\frac{a^2}{8}$$ and computed the following area: Approach 1, getting probability as shadowed area over rectangle area $P\approx 0.801$.
On the other hand I took the two intervals $a^2\equiv x\in [1.2^2,5^2]$ and $8b\equiv y\in 8\cdot[-0.8,1.5]$ obtaining the following area Approach 2 and computating the probability as area below $y=x$ over rectangle area $P\approx 0.87$.
So, I have been trying to understand which one of the approachs is wrong and why but I have failed so I am here.
EDIT: also, is there any point $(x\equiv a^2,y\equiv 8b)$ in the region where $x>y$ of the approach 2 that return a quadratic polynomial without two real roots?
The issue with Approach 2 is the non-linearity of your variable transformation. The problem this creates is that the area of the triangle in the upper-left corner of the graph, which represents the complementary probability, is made disproportionately small , so a simple subtraction of that triangular area leads to an over-estimate of the probability sought.
Let's carry out the integration for "Approach 1" first in order to compare it against what must be done for the other Approach. We will compute the area "above" the parabola $ \ b \ = \ \frac{a^2}{8} \ \ , $ in order to determine the complement to the probability for two real zeroes. The total area of the "population space" rectangle is $ \ \mathbf{A} \ = \ ( \ 5 - 1.2 \ )·( \ 1.5 - [-0.8] \ ) \ = \ 8.74 \ \ . $ The area above the parabolic arc is found from the double integral $$ \mathcal{A} \ \ = \ \ \int_{1.2}^{\sqrt{12}} \int_{a^2/8}^{1.5} \ \ db \ \ da \ \ = \ \ \int_{1.2}^{\sqrt{12}} \int_{a^2/8}^{1.5} \ \ db \ \ da \ \ = \ \ \int_{1.2}^{\sqrt{12}} \ 1.5 \ - \ \frac{a^2}{8} \ \ \ da $$ $$ = \ \ \left. \left( \ 1.5·a \ - \ \frac{a^3}{24} \ \right)\right|_{1.2}^{\sqrt{12}} \ \ \approx \ \ 1.736 \ \ . $$ [The upper limit on the integral in $ \ a \ $ is found from $ \ \frac{a^2}{8} \ = \ 1.5 \ \ . $ ] We thereby calculate the probability that the given quadratic polynomial has two real zeroes in the specified "coefficient region" as $$ 1 \ - \ \frac{\mathcal{A}}{\mathbf{A}} \ \ \approx \ \ 1 \ - \ \frac{1.736}{8.74} \ \ = \ \ 0.801 \ \ . $$
For the variable substitution $ \ u \ = \ a^2 \ \ , \ \ v \ = \ 8b \ \ , $ we obtain the rectangle above; the parabolic arc becomes $ \ b \ = \ \frac{a^2}{8} \ \rightarrow \ v \ = \ u \ \ . $ We will want to compute the complementary probability by finding the area of the triangle above that line and within the rectangle. This requires us to transform the differentials in the integral as well, which we achieve by the use of the Jacobian $$ \mathfrak{J} \ \ = \ \ \left[\begin{array}{cc} \frac{\partial u}{\partial a} & \frac{\partial u}{\partial b} \\ \frac{\partial v}{\partial a} & \frac{\partial v}{\partial b} \end{array} \right] \ \ = \ \ \left[\begin{array}{cc} 2a & 0 \\ 0 & 8 \end{array} \right] \ \ \Rightarrow \ \ | \ \mathfrak{J} \ | \ \ = \ \ 16a \ \rightarrow \ 16·\sqrt{u} \ \ . $$
Under this transformation, even the area of the rectangle must be integrated and is not simply the product of its length and width. We now calculate $$ \mathbf{A'} \ \ = \ \ \int_{1.2^2}^{5^2} \int_{8·(-0.8)}^{8·1.5} \ \ \frac{1}{| \ \mathfrak{J} \ |} \ \ dv \ \ du \ \ = \ \ \int_{1.44}^{25} \int_{-6.4}^{12} \ \ \frac{1}{16·\sqrt{u}} \ \ dv \ \ du $$ $$ = \ \ \frac{1}{16} \ \int_{1.44}^{25} \ \left. \frac{v}{\sqrt{u}} \right|_{-6.4}^{12} \ \ \ du \ \ = \ \ \frac{18.4}{16} \ \int_{1.44}^{25} \ \frac{1}{\sqrt{u}} \ \ \ du \ \ = \ \ \frac{18.4}{16} \ · \ 2 · \sqrt{u}|_{1.44}^{25} \ \ = \ \ 8.74 \ \ . $$ So we find the same area for the "transformed" rectangle as we did with the original coordinates. The area of the transformed complementary region is
$$ \mathcal{A'} \ \ = \ \ \int_{1.2^2}^{\sqrt{12}^2} \int_{u}^{8·1.5} \ \ \frac{1}{16·\sqrt{u}} \ \ dv \ \ du \ = \ \ \frac{1}{16} \ \int_{1.44}^{12} \ \left. \frac{v}{\sqrt{u}} \right|_{u}^{12} \ \ \ du $$ $$ = \ \ \frac{1}{16} \ \int_{1.44}^{12} \ \left( \ \frac{12}{\sqrt{u}} \ - \ \sqrt{u} \ \right) \ \ du \ \ = \ \ \frac{1}{16} \ \left. \left( \ 12·2·\sqrt{u} \ - \ \frac{2}{3}·u^{3/2} \ \right)\right|_{1.44}^{12} \ \ \approx \ \ 1.736 \ \ . $$
We find the area of the transformed region to be the same as that of the original one, giving us the probability we wished to determine as $$ 1 \ - \ \frac{\mathcal{A'}}{\mathbf{A'}} \ \ = \ \ 1 \ - \ \frac{\mathcal{A}}{\mathbf{A}} \ \ \approx \ \ 0.801 \ \ . $$
The discrepancy in Approach 2 is due to the omission of the non-linear effect of the chosen variable transformation. (Leaving out the Jacobian is a not uncommon error in such calculations.)