I have two vectors $(2,5,7)$ and $(-1,x,3)$. I need to select an $x$ in a way that the second vector will be perpendicular to first. The angle of two vectors is described as $cos(\phi)=\frac{a \cdot b}{|a|\cdot|b|}$, so $cos90°=0$.
$$0=\frac{5x-19}{\sqrt {78} \cdot \sqrt {10+x^2}} = 5x \cdot \sqrt {780+78x^2}-19\cdot \sqrt {780+78x^2}=\sqrt {780+78x^2}\cdot(5x-19);$$
So whether $\sqrt {780+78x^2}=0$ or $(5x-19)=0$. Since $\sqrt {780+78x^2} \neq 0$, $x = \frac{19}{2} = 3\frac{4}{5}$.
But if I try to use it in the vector $(-1,x,3)$ it's gonna produce an angle around $29°$ between the two vectors.
So, any suggestions to solve it?
As you stated above two vectors are perpendicular if $$\cos(\phi) = \frac{\langle a,b\rangle}{|a||b|}=0,$$ where $\langle\cdot,\cdot\rangle$ denotes the inner product on $\mathbb{R}^3$. Note that, if $|a|,|b|\neq 0$, this is equivalent to $\langle a, b\rangle=0$. In your example you get $$\langle a,b \rangle = 2*(-1)+5*x+3*7.$$ Inserting this in the above condition and solving for $x$ will provide the solution $x=-\frac{19}{5}$.
I think you just miscalculated in the calculation of the scalar product and when you solved $5x+19=0$.