A lemma states:
Let $R$ be a UFD and $F=\operatorname{Frac}(R)$. Let $d\in R$, then equation $a^2=d$ has a root in $R$ iff it has a root in $F$.
So I want to ask, is there a counterexample for this if $R$ is not a UFD? I only know some non-UFD integral domains like ring of integers for some values.
On
You already have a very good example, but since you mentioned ring of integers, and to put this in a more general context:
A (full) ring of algebraic integers (a maximal order) can never work as a counter-example. The point is that those are (basically by definition) integrally closed.
This means that every element of the fraction field of $R$ that is a root of a monic polynomial over $R$ is already in $R$. Since your equation corresponds to a particular type of monic polynomial having a root, the assertion holds for this one in particular.
However, if you take subrings of rings of algebraic integers (non-maximal orders) you can get examples. For example, in $\mathbb{Z}[2\sqrt{2}]$ the equation $X^2 = 2$ has not solution. But $\sqrt{2}$ is of course in the quotient field.
Let me end with an abstract argument for UFDs having the property you recalled: A UFD is integrally closed and thus it is also two-root closed (this is a somewhat common name for the property you give).
Put differently, when looking for counter-examples you need to avoid integrally closed domains, since they all still have the property mentioned in your lemma.
Let $k$ be a field and $R=k[x^2,x^3]$, the subring of $k[x]$ consisting of polynomials with no linear term. Taking $d=x^2$, the equation $a^2=d$ has no root in $R$ since $x\not\in R$. But $x=\frac{x^3}{x^2}$ is an element of the field of fractions $F$, so there does exist a root in $F$.
(To see directly that $R$ is not a UFD, note that $x^2$ and $x^3$ are both irreducible in $R$, but $(x^2)^3=(x^3)^2$, giving two distinct factorizations of $x^6$.)