Let the potential energy of a system of two atoms whose equilibrium separation is $R_0$ be given by $$U=U_0((\frac{R_0}{r})^{12} -2( \frac{R_0}{r})^{6})$$ where r is the separation at any instant.
The problem is to calculate the restoring force for this function in the form $ F=-kx$
I used the definition of force $F=-dU/dr$.but it didn't simplify in the form required. Please help me in this regard, thanks.
The potential energy function of a restoring force is of the form $U(r)= \frac 12kr^2$ (quadratic), however this is the Lennard-Jones potential which is clearly not quadratic. See a plot here: https://en.wikipedia.org/wiki/Lennard-Jones_potential
The idea however is that close to the stable equilibrium point, the function looks quadratic. We can therefore take a Taylor series expansion of the potential function about the equilibrium point up to the quadratic term which approximates the oscillations in this potential well as a harmonic oscillator (which gives you your required restoring force).
First we must find the equilibrium point which can be done by setting the derivative to zero: $$\frac {dU}{dr}=U_0 \Big( \frac{-12R_0^{12}}{r^{13}}+\frac{12R_0^6}{r^7} \Big)=0$$
$$\implies\frac{12R_0^{12}}{r^{13}}=\frac{12R_0^{6}}{r^{7}}$$ $$\implies r^6=R_0^{6}$$ $$\implies r=R_0$$ And you can show that this is a stable equilibrium by showing that $U''(R_0)>0$. We can now take a Taylor series expansion about $r=R_0$ up to the quadratic term: $$U(r)\approx U(R_0) + U'(R_0)(r-R_0)+\frac 12U''(R_0)(r-R_0)^2$$ where $$U(R_0)=-U_0$$ $$U'(R_0)=0, \hspace{2mm} \text{(since this is an equlibrium point)}$$ $$U''(R_0)=\frac{72U_0}{R_0^2} \hspace{2mm} \text{(I'll leave this for you to show.)}$$ and so $$U(r) \approx -U_0+ \frac 12 \frac{72U_0}{R_0^2}(r-R_0)^2$$ Which finally looks like the restoring potential $U(r)=\frac 12 kr^2$ that we want. Remember still that this is all an approximation technique, the potential only looks parabolic for points close to $r=R_0$ so when enough energy is put into the system this approximation falls apart. We are now finally ready to find the restoring force, simply take the negative gradient of our potential close to the equlibrium: $$F(r)= -\frac{dU}{dr}\approx-\frac{72U_0}{R_0^2}(r-R_0)$$ A restoring force with equilibrium $r=R_0$ and Hooke's constant $k=\frac{72U_0}{R_0^2}$.