Find a number greater than $3$ such that, divided by $8$, its diminished square of $1$ yields a prime for quotient.
I set up this equation: $\frac{(k^2 - 1)} {8} = p$, which would fall into a diophantine equation, at the end being $k = 7$ and $p = 6$. But that's not it.
Using the equation that you found
$$\frac{k^2 - 1}{8} = p$$
multiply both sides by $8$ to form
$$k^2 - 1=8p$$
then consider the context of the question. You need to find a $k>3$ such that $k^2-1=8p$ for some prime number $p$. Testing $k=4$ would produce
$$15=8p \implies p=\frac{15}{8}$$
where $p=\frac{15}{8}$ isn't a prime number. Testing $k=5$ forms
$$24=8p \implies p=3$$
where $p=3$ is a prime number. As the question asked to find the smallest prime $p$ such that
$$k^2 - 1=8p$$
it is clear that testing higher values of $k$ would increase $p$. Therefore, the correct answer is $p=3$ and $k=5$.