Show that $(x,y)=(4,5), (5,4)$ are the only positive integer solutions to $x+y=3n, x^2+y^2-xy=7n.$
I'm not very certain how to proceed on this problem. I know $x^2+y^2=(x+y)^2-2xy,$ so we essentially have $x+y=3n, (x+y)^2-3xy=7n$ for positive integers $x, y, n.$ However, this doesn't really help. I've also tried writing it as a fraction and doing some algebraic manipulations, but I haven't gotten anywhere either. May I have some help? Thanks in advance.
We are given
$$x + y = 3n, \; x^2 + y^2 - xy = 7n \tag{1}\label{eq1A}$$
As you showed,
$$\begin{equation}\begin{aligned} (x + y)^2 - 3xy & = 7n \\ 9n^2 - 3xy & = 7n \\ 3xy & = 9n^2 - 7n \\ xy & = 3n^2 - \frac{7n}{3} \end{aligned}\end{equation}\tag{2}\label{eq2A}$$
From this, plus the first part of \eqref{eq1A}, then using Vieta's formula's gives that $x$ and $y$ are the roots of the quadratic equation
$$z^2 - (3n)z + \left(3n^2 - \frac{7n}{3}\right) = 0 \tag{3}\label{eq3A}$$
The quadratic formula gives
$$\begin{equation}\begin{aligned} z & = \frac{3n \pm \sqrt{9n^2 - 4\left(3n^2 - \frac{7n}{3}\right)}}{2} \\ & = \frac{3n \pm \sqrt{-3n^2 + \frac{28n}{3}}}{2} \end{aligned}\end{equation}\tag{4}\label{eq4A}$$
Using that $n$ is positive, then factoring the part in the square root above (i.e., the discriminant), and requiring it to be non-negative, gives
$$-3n^2 + \frac{28n}{3} = n\left(-3n + \frac{28}{3}\right) \implies -3n + \frac{28}{3} \ge 0 \implies n \le \frac{28}{9} = 3 + \frac{1}{9} \tag{5}\label{eq5A}$$
We have \eqref{eq1A} indicating $n$ must be either an integer or a rational value of the form $\frac{m}{3}$ for some positive integer $m$ where $3 \not\mid m$. For the latter case, though, the second part of \eqref{eq1A} would not be an integer. Thus, $n$ is an integer, with \eqref{eq2A} showing it must be a multiple of $3$.
Next, the upper limit in \eqref{eq5A} shows $n = 3$ is the only possible solution, with \eqref{eq4A} giving $z = \frac{9 \pm 1}{2}$, i.e., $z = 4$ or $z = 5$. Thus, we get $(x,y) = (4,5)$ or $(5,4)$.