Let $A \in M_{2\times 2}(\mathbb{C})$ and $\det(A)=1\DeclareMathOperator{\tr}{tr}$
Prove that $\det(A^2+A-I_2)+\det(A^2+I_2) = 5$
using Cayley-Hamilton Theorem $A^2-\tr(A)A+\det(A)I_2=0$
$\det\big(\tr(A)A-\det(A)I_2-I_2\big) + \det\big(A(A+A^{-1})\big)=5$
$\det\big(\tr(A)A-I_2(\det(A)+1)\big)+\det(A+A^{-1})=5$
using https://math.stackexchange.com/q/1937052 $\det(A+B)=\det A+\det B+\det A⋅\tr(A^{-1}B) $
$\tr(A)\det(A)-\big(\det(A)+1\big)\det(I_2)+\tr(A)\big(-\det(A)-1\big)^{-1}\tr(I^{-1}A)+\det(A)+\det(A^{-1})+\tr(A^2)=5 $
$\tr(A)-2-\tr(A)^20.5+1+1+\tr(A^2)=5$
$\tr(A)-\tr(A)^20.5+\tr\big(\tr(A)A-\det(A)I_2\big)=5$
$\tr(A)-\tr(A)^20.5+\tr(A)^2-\tr(I_2)=5$
$\tr(A)+0.5\tr(A)^2-2=5$
This is where I am stuck. And also I don't know how to prove the identity which I cited.
You're on the good track.
Let $t=\operatorname{tr}(A)$. Then, from $\det(A)=1$, we know by Cayley-Hamilton that $$ A^2-tA+I_2=0 $$ Thus $$ A^2+A-I_2=tA-I_2+A-I_2=(t+1)A-2I_2 $$ and $$ A^2+I_2=tA $$ Thus $\det(A^2+I_2)=t^2\det(A)=t^2$.
The formula you cite tells you that $$ \det(X+Y)=\det(X)+\det(Y)+\det(X)\operatorname{tr}(X^{-1}Y) $$ and we can apply it to $X=-2I_2$, $Y=(t+1)A$. Thus $$ \det(X)=4,\qquad \det(Y)=(t+1)^2,\qquad X^{-1}Y=-\frac{1}{2}(t+1)A, \qquad \operatorname{tr}(X^{-1}Y)=-\frac{1}{2}t(t+1) $$ Thus the expression on the left-hand side becomes $$ 4+(t+1)^2-2t(t+1)+t^2=5 $$