Using $ f(x,y) = f_{X|Y} (Y|X) \ f_X (x)$ I got $\frac 1x e^{-x}$ and I know that
$$ \operatorname{corr}(X,Y) = \frac{E(XY) - E(X)E(Y)}{\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}}$$
but I'm struggling to calculate the expectations. My attempt for $E(XY)$ is
$$ E(XY) = \int_{x=1}^\infty\int_{y=0}^x ye^{-x}\ dy\ dx = (1-x)e^{-x} $$
and for $E(Y)$ I think it would be
$$ E(Y) = \int_{x=1}^{\infty}\int_{y=0}^x \frac{ye^{-x}}x\ dy\ dx $$
which would be zero and for $E(X)$ it looks like
$$ E(X) = \int_{x=1}^{\infty}\int_{y=0}^x e^{-x}\ dy\ dx $$
which would be zero, but this doesn't make much sense to me. Is there a flaw in my logic or is this just an odd problem?
First, your calculation of the expectation of $XY$ cannot possibly be correct, since if $x > 1$, the result is negative. The result must be a positive number for all admissible values of the parameter $x$. Second, the expectation of $Y$ is also wrong, since $Y$ can never be negative, and has a positive probability of having a positive outcome; thus it must have positive expectation. Third, the expectation of $X$ does not depend on $Y$.
I should also point out that your choice of notation is unfortunate, since I have assumed based on your calculations, that $x$ is a parameter for $X$, but if this is so, it is very confusing. Therefore, in my response, I will use $\lambda$ for the rate parameter of $X$; i.e., $$f_X(x) = \lambda e^{-\lambda x}, \quad x > 0, \quad \lambda > 0.$$
To rectify these issues, we begin with the last one. If $X \sim \operatorname{Exponential}(\lambda)$, we have $$\operatorname{E}[X] = \int_{x=0}^\infty x f_X(x) \, dx = \int_{x=0}^\infty \lambda x e^{-\lambda x} \, dx = \frac{1}{\lambda}.$$ This much should be familiar if you have studied the properties of the exponential distribution.
To compute the unconditional or marginal variance of $Y$, we observe by the law of total expectation $$\operatorname{E}[Y] = \operatorname{E}[\operatorname{E}[Y \mid X]] = \operatorname{E}[X/2] = \operatorname{E}[X]/2 = \frac{1}{2\lambda}.$$ The second equality in the chain follows from hte fact that the conditional distribution of $Y \mid X$ is uniform on $(0,X)$, thus the conditional expectation is $(0 + X)/2 = X/2$.
To compute the expectation of $XY$, we again use the law of total expectation: $$\operatorname{E}[XY] = \operatorname{E}[\operatorname{E}[XY \mid X]] = \operatorname{E}[X(X/2)] = \operatorname{E}[X^2/2] = \frac{\operatorname{E}[X^2]}{2} = \frac{\operatorname{Var}[X] + \operatorname{E}[X]^2}{2} = \frac{\operatorname{Var}[X] + \frac{1}{\lambda^2}}{2}.$$ We need to compute the variance of $X$, or recall that $$\operatorname{Var}[X] = \frac{1}{\lambda^2},$$ the proof of which is left as an exercise. Then we get $$\operatorname{E}[XY] = \frac{1}{\lambda^2}.$$ We have almost all of the pieces; the only remaining issue is to compute $\operatorname{Var}[Y]$. This is obtained via the law of total variance: $$\operatorname{Var}[Y] = \operatorname{E}[\operatorname{Var}[Y \mid X]] + \operatorname{Var}[\operatorname{E}[Y \mid X]].$$ We recall that the variance of a uniform distribution on $(a,b)$ is simply $(b-a)^2/12$. Therefore, $$\operatorname{Var}[Y \mid X] = \frac{(X - 0)^2}{12} = \frac{X^2}{12},$$ hence $$\operatorname{Var}[Y] = \frac{\operatorname{E}[X^2]}{12} + \operatorname{Var}[X/2] = \frac{2/\lambda^2}{12} + \frac{\operatorname{Var}[X]}{4} = \frac{1}{6 \lambda^2} + \frac{1}{4\lambda^2} = \frac{5}{12\lambda^2}.$$ Then the correlation is $$\rho_{X,Y} = \frac{(1/\lambda^2) - (1/\lambda)(1/(2\lambda))}{\sqrt{(1/\lambda^2)(5/(12\lambda^2))}} = \sqrt{\frac{3}{5}}.$$ No double integrals are needed, and note that the result is independent of $\lambda$.