We are trying to approximate the value of at $x=nh$ and we know that $u_{n+1}=u(x+h)$, $u_{n}=u(x)$ and $u_{n-1}=u(x-h)$. The question is, for what values of $a, b$ and $C$ is the following an approximation?
$C(5u_{n+1}+au_{n-1} - bu_{n})\approx u^{\prime\prime}$
I am not too sure what this question is asking. I am familiar with the traditional central difference approximation for the second derivative, but does there have to be some kind of relationship between the constants above for the approximation to be true?
As a special case, consider $u_n=c_0+c_1n+c_2n^2$. Then one can show that \begin{align} 5u_{n+1}+au_{n-1}-b u_n &=(5+a-b)c_0+[5-a+(5+a-b)n]c_1\\&\;\;\;\;+[5 + a + 2n(5 - a) + (5+a-b) n^2]c_2. \end{align}
But the second derivative of $u(x)=u_n$ should be independent of $c_0,c_1$, so we need the above coefficients of $c_0,c_1$ to vanish identically. From the former, we have $b=a+5$; from the latter, we have $5-a+(5+a-b)n=5-a=0.$ Hence $a=5,b=10,$ in which case the proposed approximation for $u''(x)$ is $$C(5u_{n+1}-10u_n + 5u_{n-1})=10Cc_2.$$
But $u(x)=u_n=c_0+c_1(x/h)+c_2(x/h)^2$ since $x=nh$, so $u''(x)=2c_2/h^2$. So for the above to work, we need $10Cc_2 = 2c_2/h^2\implies C=1/5h^2.$ Hence the approximation must be of the form
$$\frac{1}{h^2}(u_{n+1}-2u_n+u_{n-1})\approx u''(x)$$ which if written in terms of $u(x)$ is just the familiar central difference formula for $u''(x)$.