I have a doubt about a possible alternative to EKF that i have applied successfully but that seems not to be theoretically correct. Suppose to have a system \begin{equation} x_{k+1} = Ax_k + w_k\\ y_k = h(x_k)+Cx_k + v_k \end{equation} where $w,v$ are i.i.d guassian vectors and the measurement equation can be divided into a linear term plus a nonlinear function $h(x)$.
The standard EKF formulation needs to compute the correction term based on the measurement equation jacobian H, which results in this case \begin{equation} H = \frac{\partial h(\hat{x}_{k+1|k})}{\partial x}+C \end{equation} The corrected estimate is thus obtained by \begin{equation} \hat{x}_{k+1|k+1} = \hat{x}_{k+1|k} + PH^T(HPH^T+R)^{-1}(y_{k+1}-h(\hat{x}_{k+1|k})-C\hat{x}_{k+1|k}) \end{equation}
What i tried was to create a fictitious measurement defined as \begin{equation} \tilde{y}_{k+1} = y_{k+1}-h(\hat{x}_{k+1|k}) \end{equation} In this way, the new measurement equation becomes \begin{equation} \tilde{y}_{k+1} = Cx_{k+1} \end{equation} and it is possible to apply the Linear Kalman Filter using the modified measurement in the correction term obtaining \begin{equation} \hat{x}_{k+1|k+1} = \hat{x}_{k+1|k} + PC^T(CPC^T+R)^{-1}(\tilde{y}_{k+1}-C\hat{x}_{k+1|k}) \end{equation} without resorting to any linearization.
I would like to know if it can be a reasonable approximation in order to avoid linearization, maybe in systems such that $h(x)$ is highly non-linear.
Thank you everyone
Unfortunately this won't work, and might in some cases actually become unstable. In order to show this by using a first order Taylor approximation of $h(\hat{x}_{k+1|k})$
$$ h(\hat{x}_{k+1|k}) = h(x_{k+1}) + \left.\frac{\partial\,h(x)}{\partial x}\right|_{x=x_{k+1}} (\hat{x}_{k+1|k} - x_{k+1}) + O(\|x_{k+1} - \hat{x}_{k+1|k}\|^2). \tag{1} $$
Using $(1)$ in the measurement error yields
\begin{align} y_{k+1} - \hat{y}_{k+1} &= h(x_{k+1}) + C\,x_{k+1} - h(\hat{x}_{k+1|k}) - C\,\hat{x}_{k+1|k}, \tag{2a} \\ &\approx \left(C + \left.\frac{\partial\,h(x)}{\partial x}\right|_{x=x_{k+1}}\right) (x_{k+1} - \hat{x}_{k+1|k}). \tag{2b} \end{align}
So the partial derivative is still indirectly there, but now you are using and incorrect $H$ to update the covariance, from which you calculate the correction gain.
Another way of seeing that your approach can't work reliable is the extreme case of redefine the problem with a new nonlinear function $h'(x_k) = h(x_k) + C\,x_k$ and $C' = 0$, which would make the Kalman gain always zero.
An example of where this can become unstable would be
$$ A = \begin{bmatrix} \sqrt{3}/2 & 1/2 \\ -1/2 & \sqrt{3}/2 \end{bmatrix}, \quad C = \begin{bmatrix} 1 & 0 \end{bmatrix}, \quad h(x_k) = -2\tanh(\begin{bmatrix} 0 & 1 \end{bmatrix} x_k), $$
where the Kalman gain is obtained from the stationary solution using $R=1$ and $Q = I$. Simulation this even without noise and an initial state estimate close to the true state yields the following plots, where the blue lines are the true states and the red lines are the estimated states.