Let $X_{1},\cdots,X_{n}$ be random samples which has normal distribution $N(\mu,\sigma^{2})$.
When $\mu$ and $\sigma^{2}$ are unknown, I want to find UMVU estimator for $\frac{\mu}{\sigma}$.
I know that $(\sum_{i=1}^{n}X_{i},\sum_{i=1}^{n}X_{i}^{2})$ is complete sufficient statistic for $\left(\mu,\sigma^{2}\right)$.
Let $\bar{X}=\frac{1}{n}\sum_{i=1}^{n}X_{i}$, $S^{2}=\frac{1}{n}\sum_{i=1}^{n}X_{i}^{2}$. Then $E(\bar{X})=\mu$ and $E(S^{2})=\sigma^{2}+\mu^{2}$.
So for $\frac{\mu}{\sigma}$, I guess as a estimator $$ Y^{2}=\frac{\bar{X}^{2}}{\frac{1}{n}\sum_{i=1}^{n}X_{i}^{2}-\bar{X}}. $$ But how can I compute $E(Y^{2})?$
Comment. It seems to me you may not be on a useful track. Maybe this will help.
Let $\bar X$ be the sample mean, $S^2$ be the sample variance, and $\tau = \mu/\sigma.$ Then by the method of moments it might be reasonable to look at $\hat \tau = \bar X/S$ as an estimate of $\tau$ based on sufficient statistics. One would not necessarily expect it to be an unbiased estimator (perhaps asymptotically unbiased), but might check to see how biased it is. Because $S$ is known to be slightly negatively biased for $\sigma$ it is not surprising that $\hat \tau$ is slightly positively biased for $\tau.$
I do not want to claim this is the path you should take, but I think it may make more sense than what you have suggested. (Your denominator does not seem feasible because your $S^2$ and $\bar X$ have different dimensionalities.)
With a quick simulation of a million samples of size $n = 10$ from $Norm(\mu = 100, \sigma=20)$ (so that $\tau = 5$), in R statistical software, we can investigate this idea. At the end of my simulation I have shown a modification of your estimator that may have promise.