I'm unfamiliar with these kinds of problems.
I looked up some formulas and it says for an $(n,3)$ family of star polygons, $\theta = \frac{(1-\frac{6}{n}}{180}$
How do I get these formulas and what does $(n,3)$ family star polygon mean? how is it different from $(n,2)$ or $(n,4)$?


What here is mentioned as $(n, 3)$ can be rewritten in Schläfli symbol notation as $\{n/3\}$, meaning drawing a regular $n$-pointed star without leaving the pen from the paper, which winds thrice around the center. The general notion then is the $\{n/d\}$.
Just as the vertex angle of a convex regular polygon can be derived to be $\varphi = 180° (1-\frac 2n)$ (by considering the according centri triangle and that the angle sum of a triangle equates to $180°$), you'd alike would derive for the star-shaped regular polygrams $\{n/d\}$ that their vertex angle quite similarily is given by $\varphi = 180° (1-\frac {2d}n)$.
--- rk