In the book Model categories on the page 14,
do they assume that
$U$
in
3.
(so that $(F,U,\phi)$ is a Quillen adjunction)
is a RIGHT QUILLEN FUNCTOR or just
Right adjoint WITHOUT the preservation of fibrations and trivial fibrations?
As is commented, Lemma 1.3.4. answers your question: though this definition of Quillen adjunction is not a priori assuming $U$ to preserve fibrations and trivial fibrations, it still holds as a consequence.
We use that the cofibrations can be described as those maps which have the left lifting property for trivial fibrations, and a similar description for trivial cofibrations.
Now, let $f:A\to B\, \in\mathcal C$ and $p:D\to E\, \in \mathcal D$ be arbitrary arrows.
Then the following commutative squares correspond to each other, using the adjunction: $$\matrix{A&\longrightarrow & UD\\ f\downarrow\phantom{f} && \phantom{Up}\downarrow Up \\ B & \longrightarrow & UE} \quad\quad \leftrightarrow\quad\quad \matrix{FA&\longrightarrow & D\\ Ff\downarrow\phantom{Ff} && \phantom{p}\downarrow p\\ FB & \longrightarrow & E} $$ Also, any diagonal fill-in arrow $B\to UD$ in the left square corresponds to a diagonal fill-in $FB\to D$ in the right square.
Can you finish from here?