I am interested in a process such that
\begin{align*} &x_{t+1}=\rho_{t} x_t+\varepsilon_t, \varepsilon_t\sim \mathcal{N}(0, \sigma_{\varepsilon}^2)\\ &\rho_t=\lambda \rho_{t-1}+(1-\lambda)u_t, u_t\sim U[0, 1] \end{align*}
where $x_t$ for all $t$ are observable but not $\rho_t$. All parameters are also given (including the variance of $\varepsilon_t$ and $\lambda$) Suppose at time $t$, we want to know $E_tx_{t+1}$, then we must know $E_t\rho_t$ so we have to derive the conditional expectation of $\rho_{t-1}$. Can this problem be solve by the recursive Kalman filtering or any other reference could anyone suggest? I hardly found any useful reference online.
This problem can be accounted by using an Extended Kalman Filter (EKF), which can be employed when the equations of the dynamics are nonlinear wrt the state variables.
In your case, we can define the state of the system as follows:
$$z_{t} = \begin{bmatrix} x_t\\ \rho_t \end{bmatrix}.$$
The dynamics of the system is described by the following equations:
\begin{align*} &x_{t+1}=\rho_{t} x_t+\varepsilon_t \\ &\rho_{t+1}=\lambda \rho_{t}+(1-\lambda)u_{t} \end{align*}
where $\varepsilon_t$ and $u_t$ are random processes. The nonlinearity is due to the product $\rho_t x_t$ in the first equation.
The idea behind the EKF is to linearize using the 1st-order Taylor expansion of system dynamics. That is, the equation for the covariance update is:
$$P_{t+1|t} = F_t P_{t|t} F_t^\top + Q_t,$$
where $Q_t$ is the covariance matrix of the disturbances $\varepsilon_t$ and $u_t$ and $F_t$ is the jacobian matrix of the dynamics:
$$F_t = \begin{bmatrix} \frac{\partial x_{t+1}}{\partial x_{t}} & \frac{\partial x_{t+1}}{\partial \rho_{t}}\\ \frac{\partial \rho_{t+1}}{\partial x_{t}} & \frac{\partial \rho_{t+1}}{\partial \rho_{t}} \end{bmatrix} = \begin{bmatrix} \rho_t & x_t \\ 0 & \lambda \end{bmatrix}. $$
For the correction part, you still use the linear version of the standard KF. Specifically, the measurement matrix is:
$$H_t = \begin{bmatrix}1 & 0\end{bmatrix},$$
since you measure just $x_t$.
I'm quite sure that the scheme of EKF can be also applied recursively. You just need to use $F_t$ wherever it is needed.