We know that if $y=f(x)$ and $f'(x)$ exists at $x=x_1$ then the equation of the line tangent at the point $(x_1,f(x_1)$ is given by
$$y-f(x_1)=f'(x_1)(x-x_1).$$
My question is this.
What if $x=g(y)$ and $g'(y)$ exists at $y=y_1$. What would be the equation of the tangent line at the point where $y=y_1$?
I am expecting an answer like
$$x-g(y_1)=g'(y_1)(y-y_1)$$
but got in trouble on how to explain it.
OP is right
The functions are curves but the converse may not be true. So for a curve like $x=g(y)$ one wouldn't ask for $g^{-1}(x)$ (it may not exist). Also, when $x=g(y)$ normally it is not easy to find $y_0$ for a given $x_0$, one has to solve the implicit equation.
Suppose the equation of the curve is $x=g(y) \implies dx=g'(y)dy$ and a point lying on that is knowm/given as $(x_0,y_0)$, then the equation of the thangent is $$y-y_0=\frac{1}{g'(y_0)}(x-x_0).....(1)$$ Once the point is found, (1) represents the equation of tangent.
Example of the curve: For $x=y^y$ the equation of tangent at (4,2) is $$ y-2=\frac{1}{4(1+\log 2)}(x-4)....(2)$$ because $g'(y)=y^y(1+\ln y)=x(1+\ln y)$ The point $(1,0)$ is a limit point, the Eq, of tangent on this is $y=0$. For $x=1$ there is another point (1,1). check the tangent at this point is $y=x.$
One can solve the implicit equation $y^y=2$ to get $y_0=1.55916...$ to write the tangent at $(2, 1.55961...)$ using (1).
See the curve $x=y^y$ and its tangents at points (4,2) and (1,1) in the fig. below