The method of characteristics for a partial differential equation such as
$$u_t + cu_x =0$$
Yields a solution $u(x,t) = f(x-ct)$.
On the other hand, separation of variables $u(x,t) = X(x)T(t)$ tells us that the solution would be
$u(x,t) = e^{\lambda(x-ct)}$
I am a bit confused as to why the first method yields a "more general" solution that the second one. I recognize that I have not specified any initial or boundary conditions and wonder if this may be a reason why I am confused. What is the relationship between both methods and how could the solution of the separation of variables method be extended so that it allows a solution with any function $f(x-ct)$ instead of just the exponential function?
The separation gives you one solution (and thus not the most general function). However, as the differential equation is linear, you can add different solutions to get another solution.
In your case, separation gives the solution $$u_\lambda(x,t) = e^{i \lambda (x-c t)};$$ here, I took the liberty to rename $\lambda \mapsto i \lambda$ with respect to the question.
Now, some very generic class of functions (e.g., $L^2$ which should be additionally sufficiently often differentiable) admit a Fourier representation $$ f(x) = \int\!d\lambda\, e^{i \lambda x} F(\lambda)\tag{1}$$ with $$ F(\lambda) = (2\pi)^{-1}\int\!dx \,e^{-i \lambda x} f(x).$$
So given a function $f(x)$, we can build the linear combination (of the solutions provided by the method of separation) $$ u(x,t) = \int\!d\lambda\, F(\lambda) u_\lambda(x,t)$$ which is again a solution of the differential equation. With property (1), we obtain $$ u(x,t) = f(x-c t),$$ that is the solution obtained by the method of characteristic.