I was wondering about the equation of line I can write which can help me finding the coordinates of Point $P'$ in relation with coordinates of points on the sphere that is $P$.
Let the $P'(X,Y)$, then how can I find these following coordinates -
I tried equating the slope of $NP'$ with that of the slope of $PP'$. but could only write $\frac{y}{x} = \frac{Y-y}{X-x}$.
I donot know how it is getting these coordinates - $(X,Y) = (\frac{2ax}{a-z},\frac{2ay}{a-z})$
and $(x,y,z) = (\frac{2aX}{X^2+Y^2+a^2},\frac{2aY}{X^2+Y^2+a^2},\frac{X^2+Y^2-a^2}{X^2+Y^2+a^2})$?
Here we are following this notation that - points lying on the sphere are represented by $(x,yz)$ and points lying outside the sphere by $X,Y$
Let $S$ be the point of tangency of the sphere and the plane, and let $Q$ lie on line $NS$ such that angle $\angle NQP$ is a right angle.
Observe that triangles $\triangle NQP$ and $\triangle NSP'$ are similar, with $NQ = a - z$ and $NS = 2a.$ Therefore $$\frac{P'S}{PQ} = \frac{2a}{a-z}. \tag1$$
But we also have $P = (x,y,z)$ while $Q = (0,0,z),$ and $P' = (X,Y,-a)$ (in three dimensions) while $S = (0,0,-a).$ By proportions, $$\frac Xx = \frac Yy = \frac{P'S}{PQ},$$ and therefore (using equation $(1)$ to substitute for $\frac{P'S}{PQ}$), $$ X = \frac{2a}{a-z}x \qquad\text{and}\qquad Y = \frac{2a}{a-z} y. $$
To transform coordinates in the other direction, observe that triangles $\triangle NPS$ and $\triangle NSP'$ are similar, with $$ \frac{PN}{NS} = \frac{NS}{P'N}. $$ Then $$ \frac xX = \frac{PQ}{P'S} = \frac{PN}{P'N} = \frac{(NS)^2}{(P'N)^2}, $$ so $$ x = \frac{(NS)^2}{(P'N)^2}X. $$ Seeing that $P'N$ is the hypotenuse of a right triangle with legs $\sqrt{X^2 + Y^2}$ and $2a,$ so $$(P'N)^2 = X^2 + Y^2 + 4a^2,$$ and recalling that $NS = 2a,$ we have $$ x = \frac{4a^2}{X^2 + Y^2 + 4a^2}X. \tag2 $$ Similarly, $$ y = \frac{4a^2}{X^2 + Y^2 + 4a^2}Y. \tag3 $$ Using the fact that $z^2 = a^2 - x^2 - y^2,$ using equations $(2)$ and $(3)$ to substitute for $x$ and $y,$ and performing some algebraic manipulations, we find that $$ z = \frac{a(X^2 + Y^2 - 4a^2)}{X^2 + Y^2 + 4a^2}. $$
The formulas you were trying to justify for $(x,y,z)$ are then seen not to be correct. Those formulas, multiplied by $a,$ would give $(x,y,z)$ in terms of the coordinates of $P$ projected onto the plane $z = 0$ (not $z = -a$); that may explain where those formulas came from (though this does not explain the missing factor of $a$). If you multiply each of the formulas $\frac{2aX}{X^2+Y^2+a^2},$ $\frac{2aY}{X^2+Y^2+a^2},$ and $\frac{X^2+Y^2-a^2}{X^2+Y^2+a^2}$ by $a,$ substitute $\frac X2$ for $X,$ and substitute $\frac Y2$ for $Y,$ you will have correct formulas for the reverse projection from the plane $z = -a.$