I am working on a 3D modelling software.
I have a square anti-prism. The base and top squares have a side length of A. The height of the anti-prism is B.
There are 8 triangles on the sides. I am trying to calculate the angles between those triangles in 3D space using a formula.
The angle in question is the angle between 2 triangle that share the marked black line in the photo.

How can I calculate that angle according to A and B values.

Consider coordinate axes as in the following figure :
We have
$$\vec{EC}=\pmatrix{\frac{A}{2}(\sqrt{2}-1)\\ \frac{A}{2}\\-B}, \vec{EF}=\pmatrix{0\\A\\0} \implies N_{1}=\vec{EC} \times \vec{EF}=\pmatrix{AB\\0\\ \frac{A^2}{2}(\sqrt{2}-1)}$$
$$ \vec{CD}=\pmatrix{-\frac{A}{2}\sqrt{2}\\ \frac{A}{2}\sqrt{2}\\0}, \vec{CF}=\pmatrix{\frac{A}{2}(1-\sqrt{2})\\ \frac{A}{2}\\B} \implies N_{2}=\vec{CD} \times \vec{CF}=\pmatrix{\frac{\sqrt{2}}{2}AB\\ \frac{\sqrt{2}}{2}AB\\ \frac{A^2}{2}(1-\sqrt{2})},$$
where $N_1$, resp. $N_2$ is the external normal vector to face $CEF$, resp. $CDF$.
It remains to express :
$$\cos \alpha = \frac{1}{\|N_1\|\|N_2\|}\underbrace{N_1 . N_2}_{\text{dot product}}$$
$$\cos \alpha = \frac{1}{A^2B^2+\frac{A^4}{4}(3-2 \sqrt{2})}\left(\frac{\sqrt{2}}{2}A^2B^2-\frac{A^4}{4}(3-2 \sqrt{2})\right)$$
$$\cos \alpha =\frac{2\sqrt{2} B^2-A^2(3-2 \sqrt{2})}{4B^2+A^2(3-2 \sqrt{2})}\tag{1}$$
Sanity check :
$$\cos \alpha \approx \frac{\sqrt{2}}{2} = \cos \frac{\pi}{4}$$
$\alpha \approx \pi/4$ doesn't come as a surprise ; indeed, the mid-cross-section of the anti-prism is an octagon with, in this case, normal vectors almost in the plane of this mid-cross-section.