I'm working on the function $\psi_{(\lambda,\xi)}$, defined as follows: $$\psi_{(\lambda,\xi)} = \frac{(\lambda^2+\xi^2)^2}{(\lambda^2-\xi^2-2\lambda\xi)^2}$$(See the EDIT below), where $\lambda$ and $\xi$ are positive integers ($\lambda \in N$ and $\xi \in N$) with $\lambda > \xi$. I'm intersted only in the pairs of values $(\lambda,\xi)$ such that the ratio $\psi_{(\lambda,\xi)}$ is an integer $(\psi_{(\lambda,\xi)} \in N)$ and for which: $$\lambda^2-\xi^2-2\lambda\xi > 0$$
Here the first integral values of $(\lambda,\xi,\psi_{(\lambda,\xi)})$ for $3\leq \lambda \leq 75$:
I've tried a lot of different ways, for example trying to sum and subtract $1$ from both sides, obtaining: $$\psi_{(\lambda,\xi)} = \frac{4\lambda\xi(\lambda^2+\xi^2)}{(\lambda^2-\xi^2-2\lambda\xi)^2}+1$$ but it seems not very useful. Any idea?
EDIT:
As suggested by @Steven Stadnicki, we can only study the function: $$\psi_{(\lambda,\xi)}^{'} = \frac{\lambda^2+\xi^2}{\lambda^2-\xi^2-2\lambda\xi}$$ because both numerator and denominator are square.
As you stated, you are looking for cases where
$$\psi_{(\lambda,\xi)}^{'} = \frac{\lambda^2+\xi^2}{\lambda^2-\xi^2-2\lambda\xi} \tag{1}\label{eq1A}$$
is an integer. You also have the constraint of
$$\lambda^2-\xi^2-2\lambda\xi > 0 \tag{2}\label{eq2A}$$
First, note that since the sum of the powers of $\lambda$ and $\xi$ is $2$ in each term in the numerator and denominator in \eqref{eq1A}, this means that $\psi_{(\lambda,\xi)}^{'}$ is a Homogenous function. This helps to simplify the handling of the results. In particular, all solutions are based on a common multiple of those where $\gcd(\lambda,\xi) = 1$. To see this, let
$$\gcd(\lambda, \xi) = d, \; \lambda = da, \; \xi = db, \; \gcd(a,b) = 1 \tag{3}\label{eq3A}$$
Now, if $(\lambda,\xi)$ is a solution to \eqref{eq1A}, you get
$$\begin{equation}\begin{aligned} \frac{\lambda^2+\xi^2}{\lambda^2-\xi^2-2\lambda\xi} & = \frac{(da)^2+(db)^2}{(da)^2-(db)^2-2(da)(db)} \\ & = \frac{d^2(a^2+ b^2)}{d^2(a^2- b^2 - 2ab)} \\ & = \frac{a^2+ b^2}{a^2- b^2 - 2ab} \end{aligned}\end{equation}\tag{4}\label{eq4A}$$
Thus, $(a,b)$ would also be a solution. Conversely, for any $(\lambda,\xi)$ being a solution, then for any positive integer $m$, then $(m\lambda,m\xi)$ is also a solution (e.g., your results show that $(3,1)$, $(6,2)$ and $(9,3)$ are solutions with a result of $25$). As such, we only need to study cases of solutions $(\lambda,\xi)$ where $\gcd(\lambda,\xi) = 1$ to then determine all other solutions as being multiples of these solutions, so consider this to be the case for the rest of this answer.
In \eqref{eq1A}, let
$$\lambda^2-\xi^2-2\lambda\xi = c \tag{5}\label{eq5A}$$
Thus, you have that $c \mid \lambda^2+\xi^2$, so both the numerator and denominator must be congruent to $0$ modulo $c$. In particular, for the denominator, you have
$$\begin{equation}\begin{aligned} & \lambda^2 - \xi^2 - 2\lambda\xi \equiv 0 \pmod c \\ & \lambda^2 \equiv \xi^2 + 2\lambda\xi \pmod c \end{aligned}\end{equation}\tag{6}\label{eq6A}$$
Using this, for the numerator you get
$$\begin{equation}\begin{aligned} & \lambda^2 + \xi^2 \equiv 0 \pmod c \\ & (\xi^2 + 2\lambda\xi) + \xi^2 \equiv 0 \pmod c \\ & 2\xi^2 + 2\lambda\xi \equiv 0 \pmod c \\ & 2\xi(\xi + \lambda) \equiv 0 \pmod c \end{aligned}\end{equation}\tag{7}\label{eq7A}$$
Consider any prime factor $p$ of $c$. One possibility, as shown by \eqref{eq7A}, is that $p = 2$. Next, consider $p$ to be an odd prime factor instead. Thus, $p \mid \xi$ and/or $p \mid \xi + \lambda$. In the first case, since $p \mid \lambda^2 + \xi^2$, then $p \mid \xi$, but that's not possible since $\lambda$ and $\xi$ are relatively prime. For the second case, you have $\lambda \equiv -\xi \pmod p$, so $\lambda^2 + \xi^2 \equiv (-\xi)^2 + \xi^2 \equiv 2\xi^2 \equiv 0 \pmod p$. Since $p$ is odd, this means $p \mid \xi$. However this is the first case, which was just shown to not be possible.
This means that $p$ can only possibly be $2$, i.e., $c$ is a non-negative power of $2$. Consider that $c$ is a positive power of $2$. Thus, $2 \mid \lambda^2 + \xi^2$ means that both $\lambda$ and $\xi$ must be odd, which also gives that $\lambda^2 + \xi^2 \equiv 1 + 1 \equiv 2 \pmod 4$, so the numerator has only one factor of $2$. Similarly, $4 \mid \lambda^2 - \xi^2$, but $2\lambda\xi$ has only one factor of $2$, so the denominator has only one factor of $2$.
This shows the denominator can only be $1$ or $2$. Note also that $$\begin{equation}\begin{aligned} \lambda^2 - \xi^2 - 2\lambda\xi & = \lambda^2 - 2\lambda\xi + \xi^2 - 2\xi^2 \\ & = (\lambda - \xi)^2 - 2\xi^2 \\ & = 1 \text{ or } 2 \end{aligned}\end{equation}\tag{8}\label{eq8A}$$
It being $1$ is a case of Pell's equation. The Wikipedia page explains how to solve these equations. Also, the Transformations section shows how to convert the case where \eqref{eq8A} is $2$ to the case where it's $1$.
There's also a lot of other information about Pell's equations available, including on this site, with Are Pell solutions "unique"?? being a fairly good one which has some basic explanations about how to deal with & solve these equations.