Let $R(n)$ denote the sum of all positive rational numbers whose numerators and denominators are less than or equal to $n$ and have no common factors. I have estimated this sum to be $$ \begin{align*} R(n)=\sum_{\substack{a,b\leq n\\ (a,b)=1}}\frac{a}{b}&=\sum_{m\leq n}\sum_{\substack{k\leq m\\ (m,k)=1}}\frac{m}{k}+\sum_{m\leq n}\sum_{\substack{k\leq m\\ (m,k)=1}}\frac{k}{m}-1\\ &=\frac{1}{2}n^2\left(\frac{6}{\pi^2}(\log n+\gamma)+A\right)+O(n), \end{align*} $$ using this technique and some well-known facts from analytic number theory. Here $\gamma$ is the Euler-Mascheroni constant and $A=-\sum_{n=1}^\infty\frac{\mu(n)}{n^2}\log n=0.3465...$.
Is there any literature on this sum (or something similar)? And if so, is there an estimate with an error term smaller than $n$?
I am not sure if there is anything in the literature, but it can be rephrased in terms of the average of the reciprocals of the Farey sequence. See this related MSE thread for a similar series which yields a nearly identical main term. I include a short proof of your stated result at the bottom - it is better to keep the sum together rather than splitting it up.
Let $F_{k}$ denote the Farey sequence of order $k$, so that when $k=6$ we have $$F_{6}=\left\{ \frac{0}{1},\ \frac{1}{6},\ \frac{1}{5},\ \frac{1}{4},\ \frac{1}{3},\ \frac{2}{5},\ \frac{1}{2},\ \frac{3}{5},\ \frac{2}{3},\ \frac{3}{4},\ \frac{4}{5},\ \frac{5}{6},\ \frac{1}{1}\right\}.$$ In particular, $|F_{k}|=1+\sum_{n\leq k}\phi(n)$. Then your above identity is equivalent to the fact that
$$\mathbb{E}_{y\in F_{N}}\frac{1}{y}=\log N+\gamma-\frac{\zeta'(2)}{\zeta(2)}-\frac{1}{2}+O\left(\frac{\log^{2}N}{N}\right)$$
To see why, notice that $$\sum_{\begin{array}{c} a,b\leq N\\ (a,b)=1 \end{array}}\frac{b}{a}=-1+\sum_{y\in F_{N}}y+\sum_{y\in F_{N}}\frac{1}{y}.$$ As the Farey sequence is symmetry, we can see that $$\sum_{y\in F_{N}}y=\frac{1}{2}|F_{n}|,$$
and since $|F_{N}|=1+\sum_{n\leq N}\phi(n)=\frac{3}{\pi^{2}}N^{2}+O\left(N\log N\right)$, it follows that \begin{eqnarray*} \mathbb{E}_{y\in F_{N}}\frac{1}{y} & = & \frac{1}{|F_{N}|}\sum_{y\in F_{N}}\frac{1}{y}\\ & = & \log N+\gamma-\frac{\zeta'(2)}{\zeta(2)}-\frac{1}{2}+O\left(\frac{\log^{2}N}{N}\right), \end{eqnarray*}