Assume that we have your dynamical system:
$$x(k+1) = Ax(k)+Bu(k)$$
When we simulate it inside a computer, it looks like straight equation:
$$y(k) = kx(k) + m = Cx(k) + Du(k)$$
But our real dynamical system is the sin wave equation when we run the simulation in real life.
$$y(k) = \sin(x(k)) = f(x(k),u(k))$$
Kalman filter
$$\hat x(k+1) = A \hat x(k) - KC \hat x(k) + Bu(k) + Ky(k)$$
Question:
If we using a linear Kalman filter $K$ and basic state feedback. Can our Kalman filter then estimate the states of the sine wave?
$$\sin(x(k)) \approx \sin(\hat x(k))$$
I want to estimate states for a rotary arm, and see if a linear Kalman filter can do the job if I have state feedback.
I suppose that you mean $y_k=\sin(Cx_k)$, where $Cx$ is a scalar. Otherwise, either your $x$ is a scalar (and $C=1$), or you should specify the $\sin(x)$ function.
Ok, so let us define the estimation error signal as $\tilde{x}:=\hat{x}-x$. Suppose that $|\tilde{x}|$ is small. Then the following approximation holds $$\sin(Cx)\approx \sin(C\hat{x}) - \cos(C\hat{x})C\tilde{x},$$ or equivalently $\sin(C\hat{x})\approx \sin(Cx) + \cos(Cx)C\tilde{x}$.
Now for the observer $$\hat{x}_{k+1} = A\hat{x}_k + Bu_k + K\left(y-\sin(C\hat{x}_k)\right)$$ the error dynamics approximation is given by $$\tilde{x}_{k+1} = \left(A - \cos(C\hat{x}_k)KC\right)\tilde{x}_k.$$ Or you can alternatively approximate it as $$\tilde{x}_{k+1} = \left(A - \cos(Cx_k)KC\right)\tilde{x}_k.$$ If you have some ideas about your trajectory, e.g., $x$ remains close to the origin and $\cos({Cx})\approx 1$, then you have a linear system. If $Cx$ remains in a small sector close to zero then you can find a constant $K$ ensuring stability, e.g., via linear matrix inequalities (LMIs). If $Cx$ significantly rotates and $\cos(Cx)$ changes its sign then you need a parameter-varying observer, where $K=K(y,\hat{x})$.
Note that this analysis is based on the idea that $\tilde{x}_0$ is small enough and $\tilde{x}_k$ remains small enough.