Let me show my work before presenting the problem itself.
Let $M=\{(x,y,z) \in \mathbb{R}^3 : x+y=5 x+z=cos^2y\}$.
We can easily see that $M$ is a submanifold of $\mathbb{R}^3$ of dimension $1$.
We can also see that we can construct a global atlas for $M$, say $\{(M,\varphi)\}$, where $\varphi:M\rightarrow\mathbb{R}$ is given by $(x,y,z) \mapsto y$.
If $P=(p_1,p_2,p_3) \in M$, then $T_PM=\langle (-1,1,1-2sin(p_2)cos(p_2)) \rangle$. In particular, for $P=(5,0,-4) \in M$, $T_{(5,0,-4)}M = \langle (-1,1,1) \rangle$.
Consider the morphism (smooth map) $F:M \rightarrow S^1$ given by $(x,y,z) \mapsto (\frac{x}{\sqrt{x^2+y^2}},\frac{y}{\sqrt{x^2+y^2}})$.
Remember that $S^1=\{(x,y) \in \mathbb{R}^2 : x^2+y^2=1\}$, so the given morphism is well defined.
Expressing $F$ in the atlas of $M$, we get $$F(x,y,z)=F(\varphi^{-1}(y))=(\frac{5-y}{\sqrt{2y^2-10y+25}},\frac{y}{\sqrt{2y^2-10y+25}})$$
We can see that $F(5,0,-4)=(1,0)$.
Now we choose a coordinate chart $\psi$ of $S^1$ in a neighborhood of $(1,0)$. For example, $\psi:U=\{(x,y) \in S^1 : x > 0\} \longrightarrow ]-1,1[$ given by $(x,y) \mapsto y$.
I now want to compute in the chosen charts of $M$ and $S^1$ the expression of $DF_{(5,0,-4)}$.
How can I proceed? Some help would be appreciated. Thanks in advance.
At this point you have expressed $F : \mathbb{R} \to S^1$ as $$F(\varphi^{-1}(y))=(\frac{5-y}{\sqrt{2y^2-10y+25}},\frac{y}{\sqrt{2y^2-10y+25}})$$ You also have the mapping $\psi : U \to ]-1, 1[$ given by $$\psi(x, y) = y$$ and have demonstrated you would like the differential at the point $0$ in the domain of $F \circ \varphi^{-1}$.
So we can compose $\psi$ with $F$ to obtain: $$\psi(F(\varphi^{-1}(y))) = \frac{y}{\sqrt{2y^2-10y+25}}$$
Now $$D(\psi(F(\varphi^{-1}(y))))\big|_{y = 0} = \frac{d}{dy} \left(\frac{y}{\sqrt{2y^2-10y+25}}\right)\bigg|_{y = 0}$$
Evaluate the RHS of that and you will be done.