$$\mathbf{X}'(t) = \mathbf{A}(t) \mathbf{X}(t),\ \mathbf{A}(t) = \begin{pmatrix} 2-2\sin t & \sin t\\ 2-4\sin t & 1 + 2\sin t \end{pmatrix}$$
We are asked to find the solution that satisfies that: $$\mathbf{X}_0 = \begin{pmatrix} 1 \\ 4 \end{pmatrix}$$
As a hint we are told to use: $$\mathbf{X}(t) = \mathbf{P}(t) \mathbf{Y}(t), \mathbf{P}(t) = \begin{pmatrix} x_p & 0\\ y_p & 1 \end{pmatrix} $$
I would normally change the system using:
$$\mathbf{Y}'(t) = \mathbf{P}^{-1}(t)[\mathbf{A}(t)\mathbf{P}(t)-\mathbf{P}'(t)] \mathbf{Y}(t)$$
But I don't know how to deal with the $x_p$ and $y_p$ in the matrix .Is there another way to tackle this problem?
First, by staring very hard at the system (or using a previous edit), we find a particular solution to the system, given by \begin{equation} \begin{pmatrix} 1 \\ 2 \end{pmatrix} e^{2 t}. \end{equation} Now, the notation 'subscript $p$' in the hint with $x_p$, $y_p$ is somewhat confusing, since we're clearly dealing with a periodic system here, with a particular solution. Turns out, the best thing to do is use the latter interpretation for '$p$'. In other words, when you use \begin{equation} x_p(t) = e^{2t},\quad y_p(t) = 2 e^{2t}, \end{equation} the approach you proposed immediately leads to an explicitly solvable system.