I want to find the function $g(X)=a(X)+b(X)$.
Suppose that $\alpha$ is a solution to $\tan\theta x=\tan\phi(x-X)$[*], then I know that $a=\alpha\tan\theta$, $c=\alpha-X$, and $b=\sqrt{f^2(x')-c^2}$. I then get that $g(X)=\alpha\tan\theta+\sqrt{f^2(x')-(\alpha-X)^2}$.
For a given $X$, $x'=\sqrt{\alpha^2+(\alpha\tan\theta)^2}=\sqrt{\alpha^2(1+\tan^2\theta)}=|\alpha\sec\theta|$.
To simplify a bit, I can recursively define $\alpha=X+f(x')\sin\theta$, which leads me to $g(X)=\alpha\tan\theta+|f(x')\cos\theta|$.
If I substitute with the value of $x'$, I get $g(X)=\alpha\tan\theta+|f(|\alpha\sec\theta|)\cos\theta|$.
But in order to define $\alpha$ without recursion, I need to find $\phi$.
$$\alpha=\frac{\tan\phi}{\tan\phi-\tan\theta}X,\,[*]$$
Any ideas? Thank you for your time!
Your givens, $x', f(x')$ are in the primed coordinate system. It is easier to do this calculation if you consider this "the" coordinate system. So the $x$-axis of the original coordinate system is a line slanting down by an angle of $\theta$. It is the set of points $$\{\vec p\mid \hat n \cdot \vec p = 0\}$$ where $\hat n$ is the unit normal to the line, given by $\hat n = (\sin \theta, \cos \theta)$ (in primed coordinates).
$g(X)$ is the (directed) distance from the point $P = (x', f(x'))$ to this down-slanting line, which is given by $P\cdot \hat n$. The projection $Q$ of $P$ onto the line is then $P - (P\cdot \hat n)\hat n$. $X$ is the distance from $Q$ to the origin$. Putting it all together:
$$\hat n = (\sin \theta, \cos \theta)\\P = (x', f(x'))\\g(X) = P \cdot \hat n = x'\sin\theta + f(x')\cos \theta\\Q = \big(x' - (x'\sin\theta + f(x')\cos \theta)\sin \theta, f(x') - (x'\sin\theta + f(x')\cos \theta)\cos\theta\big)\\=(x'\cos\theta - f(x')\sin\theta)\big(\cos\theta, \sin\theta\big)\\X = x'\cos\theta - f(x')\sin\theta$$
In short: $$X = x'\cos\theta - f(x')\sin\theta\\g(X) = x'\sin\theta + f(x')\cos \theta$$