So I need to find $\vec x$ in $$[\vec a, \vec x]=\vec b,$$ where $\vec a\ne \vec0$.
Set the basis $$\vec e_1=\frac{\vec a}{|\vec a|}, \vec e_2=\frac{\vec b}{|\vec b|}, \vec e_3=\frac{[\vec a, \vec b]}{|[\vec a, \vec b]|}$$ By definition of vector product $\vec a$ should be perpendicular to $\vec b$, therefore $\vec a, \vec b$ and $[\vec a, \vec b]$ is orthogonal basis, which means $\vec e_1, \vec e_2, \vec e_3$ is orthogonal too.
Then $\vec a=\{|\vec a|, 0, 0\},\ \vec b=\{0, |\vec b|, 0\},\ \vec x=\{\alpha,0,\gamma\}\ \bigl(\vec x \in (\vec a, [\vec a, \vec b])$ because $\vec x \bot \vec b \bigr)$
$$[\vec a, \vec x]= \begin{vmatrix} \vec e_1 & \vec e_2 & \vec e_3 \\ |\vec a| & 0 & 0 \\ \alpha & 0 & \gamma \end{vmatrix},$$ whence $$\vec b=\vec e_2(-\gamma |\vec a|),\ \gamma=-\frac{|\vec b|}{|\vec a|}$$
And here I'm got stuck. How to find $\alpha$ coordinate?
Your approach is correct. Indeed, $\gamma=-\dfrac{|\vec b|}{|\vec a|}$, and $\alpha$ can take on any value. Note that $$[\vec a, \alpha \vec e_1] = \vec 0.$$ Consequently, $$[\vec a, \alpha \vec e_1 +\gamma \vec e_3] = [\vec a, \alpha \vec e_1]+[\vec a ,\gamma \vec e_3]=\vec 0 + [\vec a ,\gamma \vec e_3] = \vec b.$$