For reference: In triangle ABC: $AC=BC=a, AB=b$ and $\angle C= 40^\circ$.
Demonstrating identity: $3ba^2 = a^3\sqrt3+b^3$
My progress:
Draw $CD \perp AB, D\in AB$ and $AF \perp CD,(F \in CD)$
$\triangle ABF \sim \triangle CBD \implies: \frac{AF}{CD}=\frac{BF}{BD}=\frac{AB}{BC}\rightarrow \frac{BF}{\frac{b}{2}} = \frac{b}{a}\\ \therefore BF = \frac{b^2}{2a}\\ \triangle AGD \sim ABF \implies\\ \frac{DG}{BF}=\frac{AG}{b}=\frac{\frac{b}{2}}{AF}\rightarrow \frac{DG}{BF}=\frac{AG}{b}=\frac{b}{2AF}\implies \frac{DG}{AG} = \frac{b}{2a}(I)\\ \triangle AGD \sim \triangle CAD \implies\\ \frac{AD}{DC}=\frac{AG}{AC}=\frac{DG}{AD}\rightarrow \frac{b}{2DC}=\frac{AG}{a}=\frac{2DG}{b} \implies \frac{DG}{AG} = \frac{b}{2a}\\ \therefore (I): \frac{b}{2a} = \frac{BF}{b} \implies BF = \frac{b^2}{2a}\\ \triangle ADC: DC^2 = a^2-(\frac{b}{2})^2\implies DC = \frac{\sqrt{4a^2-b^2}}{2} $
But I am not able to equate...
Draw line segment $\small AD$ as in the figure.
$\small \triangle ACB\sim\triangle BAD\implies\dfrac{BD}b=\dfrac ba\implies BD=\dfrac{b^2}a$
Therefore, $\small CD=a-\dfrac{b^2}a$.
Now applying cosine rule to $\small \triangle ACD$ with the angle $\small 30^\circ$ gives the desired formula.
If you are not interested in using cosine rule directly,
Drop perp from $\small D$ to $\small AC$ and mark the foot $\small E$. Now using the fact that $\small \triangle ADE$ is a half of an equilateral triangle, we see $\small DE=b/2$ and $\small AE=\sqrt3b/2$. Now find $\small EC$ and already found $\small CD$, apply Pythagorean theorem to $\small \triangle CED$, from where you can derive the same expression.