If $F$ is free and $R$ is normal in $F$, then $F/R'$ is torsion-free, where $R'=[R,R]$.
This is Exercise 11.50 in Rotman's An Introduction to the Theory of Groups with the following hint attributed to Rosset:
First reduce to the case $F/R$ cyclic of prime order $p$. Let $x \in F$ satisfy $x^p \in R'$; if $x \in R$, its coset has finite order in $R/R'$; if $x \notin R$, then $x \notin F'$ (since $F' \leq R$), and $x^p \notin F'$, hence $x^p \notin R'$.
I am unable to parse what the hint seems to be indicating in terms of direction, and I am just interested in proving the theorem in general for myself, regardless of whether or not it uses this hint. There are a few immediate questions: First, how the case $F/R$ cyclic of prime order $p$ proves the general case. Second, why $x \notin F'$ implies $x^p \notin F'$. Other than that the argument so far seems to be: if the coset of $x$ in $F/R'$ has order $p$, then $x^p \in R'$ and according to the hint $x \in R$, so the order of its coset in $R/R'$ is finite, which means it must be the identity since $R/R'$ is free abelian, so $x \in R'$, and its coset in $F/R'$ was actually the identity, so $F/R'$ has no elements of order $p$ - but I'm not sure about elements of order coprime to $p$.
For your first concern, I think the following argument works: Suppose, for a contradiction, that $F/R'$ has nontrivial torsion. Then, there exists some element $xR' \in F/R'$ with finite order $p$, for some prime $p$. In other words, there exists some $x \in F$ such that $x \notin R'$, but $x^p \in R'$.
If $x \in R$, then $xR'$ is an element of $R/R'$ of order $p$, contradicting the fact that $R/R'$ is a free abelian group. So, we must have $x \notin R$.
Now, consider the subgroup $$G = \langle x^{j}r \mid 0 \leq j < p, r \in R\rangle.$$ This subgroup contains $R$, and the quotient $G/R$ is cyclic of order $p$ (because $x \notin R$ but $x^{p} \in R' \subset R$).
Then, if we assume the result for cyclic prime order quotients, it follows that $G/R'$ must be torsion-free. But it is clear that $xR'$ is a torsion element of $G/R'$, contradiction. So $F/R'$ cannot have nontrivial torsion.
This shows that the general case follows from the prime order quotient case.
For your second concern, we want to show that if $x \notin R$ and thus $x \notin F'$, then we have $x^{p} \notin F'$ and thus $x^{p} \notin R'$.
It suffices to show that if $x^{p} \in F'$, we must have $x \in F'$. Suppose $x^{p} \in F'$. Then, the coset corresponding to $x$ in the quotient $F/F'$ has order dividing $p.$ But we know that $F/F'$ is the abelianization of the free group $F$, hence is a free abelian group, which is torsion-free. So, the coset corresponding to $x$ in the quotient $F/F'$ must be the identity, hence $x \in F'$.