The ellipse $6x^2+4xy+5y^2+8x+8y+1=0$ is neither expressed in terms of $x$; like $y=\pm\sqrt{a^2-x^2}$, nor in terms of $y$; like $x=\pm\sqrt{a^2-y^2}$.
Separation of $x$ (or $y%$) may be impossible.
I was thinking to let $x=r\cos(\theta)$, and $y=r\sin(\theta)$, and then integrating, but finding the limits of integration [with respect to $\theta$] is difficult, or may be impossible.
How to determine the area of the ellipse shown below?
On
The simplest way to compute the area is that of using the formulas given here to find semi-axes $a$ and $b$ and then area $=\pi ab$.
If you insist on using an integral, you can of course solve for $y$ to find two solutions $$ y_\text{up/down}={1\over5}\left( -2x-4\pm\sqrt{-26x^2-24x+3} \right) $$ and then compute the area as $$ \int_{x_\text{left}}^{x_\text{right}}(y_{up}-y_{down})\,dx, $$ where $x_\text{left/right}$ are the values of $x$ giving a vanishing square root.
On
Write the equation as: $$\begin{pmatrix}x & y & 1\end{pmatrix} \left(\begin{array}{ccc|c} \\&Q& &P \\ \\ \hline &P^T& &R \end{array}\right) \begin{pmatrix}x \\ y \\ 1\end{pmatrix} =\begin{pmatrix}x & y & 1\end{pmatrix} \left(\begin{array}{cc|c}6 & 2 & 4 \\ 2 & 5 & 4 \\ \hline 4&4&1 \end{array}\right) \begin{pmatrix}x \\ y \\ 1\end{pmatrix} = 0$$
Set the gradient equal to zero to find the translation (same as solving $Qx=-P$): $$\nabla(6x^2+4xy+5y^2+8x+8y+1)=0 \Rightarrow \begin{cases}12x+4y+8=0\\4x+10y+8=0\end{cases} \Rightarrow x=-\frac 6{13}, y=-\frac 8{13}$$ So the translation vector $t=-\frac 1{13}\begin{pmatrix}6\\8\end{pmatrix}$.
Divide the equation by $-(P\cdot t + R)$ to normalize it. The resulting $Q'$ is: $$Q'=-\frac{1}{P\cdot t + R}Q$$ Its determinant is the product of the eigenvalues, which are $\frac 1{a^2}$ and $\frac 1{b^2}$. So: $$\det Q' = \frac{1}{a^2b^2}$$ The area of an ellipse is $\pi a b$ so that: $$\text{Area} = \pi a b = \frac{\pi}{\sqrt{\det Q'}} = \frac{\pi\cdot |P\cdot t+R|}{\sqrt{|\det Q|}} =\frac{\pi\cdot |-\frac 1{13}(4\cdot 6 + 4\cdot 8)+1|}{\sqrt{|6\cdot 5-2\cdot 2|}} =\frac{43\pi}{13\sqrt{26}}$$
The general formula is: $$\text{Area}=\frac{\pi|P^TQ^{-1}P-R|}{\sqrt{|\det Q|}}$$
Consider the ellipse $Ax^2 + 2Hxy + By^2 + 2Gx + 2Fy + C = 0$. It is well known that (1) this conic is an ellipse iff $AB - H^2 > 0$, so that we can assume $A > 0$; and (2) assuming $A > 0,$ the ellipse has real points iff $L > 0$, where $$L = AF^2 + BG^2 + CH^2 - ABC - 2FGH.$$ To find area of the ellipse, one can apply an affine transformation that doesn't change the area. Such a transformation is $x = \lambda y + z$ where $\lambda$ is any constant. This gives $$(A\lambda^2 + 2H\lambda + B)y^2 + 2(A\lambda + H)yz + Az^2 +2(G\lambda + F)y + 2Gz + C = 0.$$ Get rid of $yz$ by choosing $\lambda = -H/A.$ Then $$(AB - H^2)y^2 + 2(FA - GH)y + A^2z^2 + 2GAz + AC = 0,$$ that is $$(AB - H^2)\left(y - {{(AF - GH)}\over{AB - H^2}}\right)^2 + (Az + G)^2 - {{(AF - GH)^2}\over{AB - H^2}} - G^2 + AC = 0.$$ Moving the centre of the ellipse to the origin, this can be put as $$(AB - H^2){y_1}^2 + A^2{z_1}^2 - K = 0,$$ where $$K = {1\over{(AB - H^2)}}\left[ (AF - GH)^2 + (AB - H^2)(G^2 - AC)\right],$$ which reduces to $K = LA/(AB - H^2)$ (with $L$ as above). Hence the semi-axes of the transformed ellipse are $a = \sqrt{K/(AB - H^2)}$ and $b = \sqrt{K}/A,$ and the required area is $\pi ab = \pi L/(AB - H^2)^{3/2}.$
In particular, the ellipse in the question has area $86\pi/26^{3/2}.$