Let $M_t$ be the compensated process associate the Poisson Process $N_t$ with intensity $\lambda_t.$ Then consider a smooth function $F(t,X_t)$ where $dX_t = \alpha_t dt + \beta_t dN_t.$
By Ito's formula, we have, $$F(t,X_t) = F(0,X_0) + \int_0^t \left[\partial_s F(s,X_s) + \partial_x F(s,X_s) \alpha_s\right] ds + \sum_{s\leq t} F(s,X_s) -F(s,X_{s-})$$ which can be re-written as $$F(t,X_t) = F(0,X_0) + \int_0^t \left[\partial_s F(s,X_s) + \partial_x F(s,X_s) \alpha_s\right] ds + \int_0^t \left[F(s,X_{s-}+\beta_s) -F(s,X_{s-})\right]dN_s.$$
Since $dM_s=dN_s - \lambda_s ds$ I can further re-write the above expression as follows, $$F(t,X_t) = F(0,X_0) + \int_0^t \left[\partial_s F(s,X_s) + \partial_x F(s,X_s) \alpha_s+\left(F(s,X_{s-}+\beta_s) -F(s,X_{s-})\right)\lambda_s\right] ds + \int_0^t \left[F(s,X_{s-}+\beta_s) -F(s,X_{s-})\right]dM_s.$$
Is this final expression correct and also can I get rid of $X_{s-}$ in the $ds$ integral and write $$F(t,X_t) = F(0,X_0) + \int_0^t \left[\partial_s F(s,X_s) + \partial_x F(s,X_s) \alpha_s+\left(F(s,X_{s}+\beta_s) -F(s,X_{s})\right)\lambda_s\right] ds + \int_0^t \left[F(s,X_{s-}+\beta_s) -F(s,X_{s-})\right]dM_s?$$
I have read in these lecture notes, Proposition 1.3.2 for instance that this can be done since the set of jump times forms a countable set. But is there a rigorous justification for this fact, if it is indeed true?
You got it, you can replace Xs− with Xs in the ds integral. The reason is:
The set of jump times {s : Ns ≠ Ns−} is at most countable (as it is a subset of the jump times of a Poisson process) For any fixed s, Xs = Xs− outside of this countable set Therefore, the difference F(s, Xs) − F(s, Xs−) is zero for almost every s, and the integral is unchanged if you replace Xs− with Xs
So your final expression is correct. This is a standard trick when working with Itô integrals - you can replace the "pre-jump" value of the process with the post-jump value in the diffusion term, as long as you are careful to keep the "correct" value in the jump term.