From point B(0,b) and B'(0,-b) two lines perpendicular to Y-axis are drawn, as BN and B'N' respectively - towards the right. The multiplication of the lengths of those lines is $4a^2$. Find the set of points where B'N and BN' meet.
Although I think I have the data, in this exercise I can't find in which original formula to plug it into. Either I did something wrong, or it's not the distance formula. Would appreciate help.
Answer is $b^2x^2+a^2y^2=a^2b^2$
If I understand correctly, the error in your solution is where you assume the $x$ co-ordinate of both $N,N'$ as $t$. Why should they have equal $x$ co-ordinates?
Let the co-ordinates of $N,N'$ be $(x_1,b),(x_2,-b)$ respectively. The length of the segments $BN, B'N'$ is $x_1$ and $x_2$ respectively. Thus,$$x_1x_2=4a^2$$
The equation of the line $BN'$ is$$\displaystyle\frac{y-b}{x}=\frac{-2b}{x_2}$$ and the equation of line $B'N$ is$$\displaystyle\frac{y+b}{x}=\frac{2b}{x_1}$$
Their point of intersection is given by $\displaystyle\Big(\frac{x_1x_2}{x_1+x_2},b\Big[\frac{x_2-x_1}{x_1+x_2}\Big]\Big)=(p,q)$.
Now, notice that $$\displaystyle p=\frac{x_1x_2}{x_1+x_2}=\frac{4a^2}{x_1+x_2}\\\displaystyle\Big(\frac qb\Big)^2=\frac{x_1^2+x_2^2-2x_1x_2}{(x_1+x_2)^2}=\frac{(x_1+x_2)^2-4x_1x_2}{(x_1+x_2)^2}=1-16a^2\cdot\Big(\frac p{4a^2}\Big)^2=1-\frac{p^2}{a^2}$$
This gives the required locus as $\displaystyle\frac{p^2}{a^2}+\frac{q^2}{b^2}=1$ or $\displaystyle\frac{x^2}{a^2}+\frac{y^2}{b^2}=1$.