I am trying to solve the following equation for the matrix of 16 components ${\dfrac{\partial{x^\alpha}}{\partial{x^{\mu '}}}}$ with $\alpha, \mu '$ $\in$ {${0,1,2,3}$}:
$\displaystyle\frac{\partial{x^{\alpha}}}{\partial{x^{\mu '}}}\displaystyle\frac{\partial{x^{\beta}}}{\partial{x^{\nu '}}} g_{\alpha \beta} - \delta_{\mu '}^{\alpha}\delta_{\nu '}^{\beta} g_{\alpha \beta} =0$
Where $g_{\alpha \beta }$ is a given specific tensor (actually the metric) which is completely known and unique. Also, I have used Einstein summation convention. The unprimed and primed indices represent components of the tensorial quantities in coordinate set-ups $O$ and $O'$ respectively.
I know that if the above-stated equality had been an identity then I would have cancelled out the metric component terms. But here, since this is not an identity but an equation, I can't do that. (Notice that Einstein summation is implied.) In other words, we don't know a priori that this equality is valid under general coordinate transformation but rather we have to find the transformations under which this equality is valid. So, how do I approach this equation?
Since you're treating $P^\alpha_{\mu'} = \dfrac{\partial x^\alpha}{\partial x^{\mu'}}$ as the unknown, this is just the linear algebra problem
$$ P^T g P = g. $$
Since $g$ is symmetric and non-degenerate, there is some invertible matrix $Q$ such that $g = Q^T \eta Q$ where $\eta$ is a diagonal matrix with entries $\pm 1$. Substituting this and rearranging we get
$$ Q^{-T}P^TQ^T\ \eta\ QPQ^{-1} = (QPQ^{-1})^T\ \eta\ QPQ^{-1}=\eta.$$
This is true exactly when $QPQ^{-1}$ lies in the indefinite orthogonal group associated to $\eta$; e.g. $O(3,1)$ when $g$ is Lorentzian. Thus you could write the solution space as $P \in O(g) = Q^{-1} O(\eta) Q$.