I obtained this question from my school provided material. At first, it's quite easy to see through simple substitution that for any natural value of $n$, we get the answer as $\dfrac{1}{8}$, but I couldn't figure out how to prove it. I tried using binomial expansion below
$\{\dfrac{3^{2n}}{8}\} = \{\dfrac{(2+1)^{2n}}{2^3}\}$
On expanding:
$[\dfrac{1}{2^3}$ ${2n\choose{0}}2^{2n}$ +${2n\choose{1}}2^{2n-1}$ + ${2n\choose{2}}2^{2n-2}$ + ${2n\choose{3}}2^{2n-3} ... + {2n\choose{2n}}2^{0}]$
Now how do I make it so that all terms except the last one get canceled by $2^3$, so that I get $\dfrac{1}{8}$ as the answer?
Asking for $\{\frac{3^{2n}}{8}\}$ is the same as asking for the remainder when $3^{2n}$ is divided by $8$. If that remainder is $r$, then the fractional part is $\tfrac{r}{8}$.
The key observation is that $3^{2n} = 9^n$, and since $9$ divided by $8$ leaves remainder $1$, $9^{n}$ divided by $8$ also leaves remainder $1$. You can prove that with the binomial theorem, by writing $9^n = (8+1)^n$ and expanding. But it's simpler than that - the product of any two remainder $1$ numbers is another remainder $1$ number, since
$$(8q+1)(8p+1) = 8(8pq+p+q)+1.$$
More generally, knowing about modular arithmetic makes this question almost trivial. So you might be interested in reading about it.