RK4 is:
$$\begin{aligned}k_1&=hf(x_n, y_n)\\ k_2&=hf(x_n+h/2. y_n+k_1/2)\\ k_3&=hf(x_n+h/2, y_n+k_2/2)\\ k_4&=hf(x_n+h, y_n+k_3)\\ y_{n+1}&=y_n+(k_1+2k_2+2k_3+k_4)/6\end{aligned}$$
I want to know the reason of the coefficients $1/2, 1/6, 1/3$.
Is this the solution to some optimization problem? What is the general rule?
Short answer: you can think of $v:=\frac{1}{6}\left(k_1+2 k_2+2 k_3+k_4\right)$ as a weighted average slope, so $$ y_{n+1}=y_n+h v $$ is nothing but a kind of Euler method but with an improved slope. Instead of taking the slope at $(x_n,y_n)$ you take several measures ($k_i$) in the interval $(x_n,x_{n+1})$. Why these weights exactly? This requires a long answer.
Long answer: see this great answer. To summarize, the exact value of $y_{n+1}$ is $y_n+\int_{x_n}^{x_{n+1}}f(x,y)dx$ so you should approximate this integral numerically (because you don't know $y(x)$). When you use Simpson's rule you obtain the formula you have presented.