Here's a list of the $x$ in the fundamental solution of $x^2-dy^2=1$. It seems when $d=n^2-3$ for an integer $n$ the $x$ is often quite large. Is this a coincidence or is there an explanation?
For $n>5$ the continued fraction of $\sqrt {n^2-3}$ starts with $$\sqrt {n^2-3} = \left[n-1;1,n-2-\lfloor n/3\rfloor, a_n...\right]$$ Where $a_n=3$ if $n\equiv 2\pmod 3$ and $1$ otherwise. Maybe this is related.
When $n$ is a multiple of 3, the solution is small because $(6n^2-1)^2-(9n^2-3)(2n)^2=1$.
In this file the $n$th line is the continued fraction of $\sqrt{(3n+1)^2-3}$. There are some interesting patterns in it, for example in the 12th column there seems to be a pattern repeating mod 9: $13,1,1,1,1,1,2,3,5$. Here's the same thing for $\sqrt{(3n+2)^2-3}$.

Here the blue line is the log of the geometric average of the x in the minimal solution of the pell equation and the orange line is that only for numbers of the form $n^2-3$. Eyeballing that last plot $$\ln\left(\left(x_1x_6x_{13}\dots x_{9998241}\right)^{1/3162}\right)\approx\ln\left(\left(x_1x_2x_3\dots x_{10^7}\right)^{10^{-7}}\right) + 150$$ $$(x_1x_6x_{13}\dots x_{9998241})^{1/3162}\approx e^{150}(x_1x_2x_3\dots x_{10^7})^{10^{-7}}$$ Where $(x_n, y_n)$ is the fundamental solution of $x^2-ny^2=1$, when $n$ is a square I set $x_n=1$. Also, the geometric average of the subsequence is higher than the average of the whole sequence for all $61\leq n < 10^7$.
Exluding the cases where n is a multiple of 3 makes the effect even larger:

Method described by Prof. Lubin at Continued fraction of $\sqrt{67} - 4$
Here is $$ D = 8^2 - 3 = 61. $$
$$ \sqrt { 61} = 7 + \frac{ \sqrt {61} - 7 }{ 1 } $$ $$ \frac{ 1 }{ \sqrt {61} - 7 } = \frac{ \sqrt {61} + 7 }{12 } = 1 + \frac{ \sqrt {61} - 5 }{12 } $$ $$ \frac{ 12 }{ \sqrt {61} - 5 } = \frac{ \sqrt {61} + 5 }{3 } = 4 + \frac{ \sqrt {61} - 7 }{3 } $$ $$ \frac{ 3 }{ \sqrt {61} - 7 } = \frac{ \sqrt {61} + 7 }{4 } = 3 + \frac{ \sqrt {61} - 5 }{4 } $$ $$ \frac{ 4 }{ \sqrt {61} - 5 } = \frac{ \sqrt {61} + 5 }{9 } = 1 + \frac{ \sqrt {61} - 4 }{9 } $$ $$ \frac{ 9 }{ \sqrt {61} - 4 } = \frac{ \sqrt {61} + 4 }{5 } = 2 + \frac{ \sqrt {61} - 6 }{5 } $$ $$ \frac{ 5 }{ \sqrt {61} - 6 } = \frac{ \sqrt {61} + 6 }{5 } = 2 + \frac{ \sqrt {61} - 4 }{5 } $$ $$ \frac{ 5 }{ \sqrt {61} - 4 } = \frac{ \sqrt {61} + 4 }{9 } = 1 + \frac{ \sqrt {61} - 5 }{9 } $$ $$ \frac{ 9 }{ \sqrt {61} - 5 } = \frac{ \sqrt {61} + 5 }{4 } = 3 + \frac{ \sqrt {61} - 7 }{4 } $$ $$ \frac{ 4 }{ \sqrt {61} - 7 } = \frac{ \sqrt {61} + 7 }{3 } = 4 + \frac{ \sqrt {61} - 5 }{3 } $$ $$ \frac{ 3 }{ \sqrt {61} - 5 } = \frac{ \sqrt {61} + 5 }{12 } = 1 + \frac{ \sqrt {61} - 7 }{12 } $$ $$ \frac{ 12 }{ \sqrt {61} - 7 } = \frac{ \sqrt {61} + 7 }{1 } = 14 + \frac{ \sqrt {61} - 7 }{1 } $$
Simple continued fraction tableau:
$$ \begin{array}{cccccccccccccccccccccccccccccccccccccccccccccccc} & & 7 & & 1 & & 4 & & 3 & & 1 & & 2 & & 2 & & 1 & & 3 & & 4 & & 1 & & 14 & & 1 & & 4 & & 3 & & 1 & & 2 & & 2 & & 1 & & 3 & & 4 & & 1 & & 14 & \\ \\ \frac{ 0 }{ 1 } & \frac{ 1 }{ 0 } & & \frac{ 7 }{ 1 } & & \frac{ 8 }{ 1 } & & \frac{ 39 }{ 5 } & & \frac{ 125 }{ 16 } & & \frac{ 164 }{ 21 } & & \frac{ 453 }{ 58 } & & \frac{ 1070 }{ 137 } & & \frac{ 1523 }{ 195 } & & \frac{ 5639 }{ 722 } & & \frac{ 24079 }{ 3083 } & & \frac{ 29718 }{ 3805 } & & \frac{ 440131 }{ 56353 } & & \frac{ 469849 }{ 60158 } & & \frac{ 2319527 }{ 296985 } & & \frac{ 7428430 }{ 951113 } & & \frac{ 9747957 }{ 1248098 } & & \frac{ 26924344 }{ 3447309 } & & \frac{ 63596645 }{ 8142716 } & & \frac{ 90520989 }{ 11590025 } & & \frac{ 335159612 }{ 42912791 } & & \frac{ 1431159437 }{ 183241189 } & & \frac{ 1766319049 }{ 226153980 } \\ \\ & 1 & & -12 & & 3 & & -4 & & 9 & & -5 & & 5 & & -9 & & 4 & & -3 & & 12 & & -1 & & 12 & & -3 & & 4 & & -9 & & 5 & & -5 & & 9 & & -4 & & 3 & & -12 & & 1 \end{array} $$
$$ \tiny \begin{array}{cccccccccccccccccccccccccccccccccccccccccccccccc} & & 7 & & 1 & & 4 & & 3 & & 1 & & 2 & & 2 & & 1 & & 3 & & 4 & & 1 & & 14 & & 1 & & 4 & & 3 & & 1 & & 2 & & 2 & & 1 & & 3 & & 4 & & 1 & & 14 & \\ \\ \frac{ 0 }{ 1 } & \frac{ 1 }{ 0 } & & \frac{ 7 }{ 1 } & & \frac{ 8 }{ 1 } & & \frac{ 39 }{ 5 } & & \frac{ 125 }{ 16 } & & \frac{ 164 }{ 21 } & & \frac{ 453 }{ 58 } & & \frac{ 1070 }{ 137 } & & \frac{ 1523 }{ 195 } & & \frac{ 5639 }{ 722 } & & \frac{ 24079 }{ 3083 } & & \frac{ 29718 }{ 3805 } & & \frac{ 440131 }{ 56353 } & & \frac{ 469849 }{ 60158 } & & \frac{ 2319527 }{ 296985 } & & \frac{ 7428430 }{ 951113 } & & \frac{ 9747957 }{ 1248098 } & & \frac{ 26924344 }{ 3447309 } & & \frac{ 63596645 }{ 8142716 } & & \frac{ 90520989 }{ 11590025 } & & \frac{ 335159612 }{ 42912791 } & & \frac{ 1431159437 }{ 183241189 } & & \frac{ 1766319049 }{ 226153980 } \\ \\ & 1 & & -12 & & 3 & & -4 & & 9 & & -5 & & 5 & & -9 & & 4 & & -3 & & 12 & & -1 & & 12 & & -3 & & 4 & & -9 & & 5 & & -5 & & 9 & & -4 & & 3 & & -12 & & 1 \end{array} $$
$$ \begin{array}{cccc} \frac{ 1 }{ 0 } & 1^2 - 61 \cdot 0^2 = 1 & \mbox{digit} & 7 \\ \frac{ 7 }{ 1 } & 7^2 - 61 \cdot 1^2 = -12 & \mbox{digit} & 1 \\ \frac{ 8 }{ 1 } & 8^2 - 61 \cdot 1^2 = 3 & \mbox{digit} & 4 \\ \frac{ 39 }{ 5 } & 39^2 - 61 \cdot 5^2 = -4 & \mbox{digit} & 3 \\ \frac{ 125 }{ 16 } & 125^2 - 61 \cdot 16^2 = 9 & \mbox{digit} & 1 \\ \frac{ 164 }{ 21 } & 164^2 - 61 \cdot 21^2 = -5 & \mbox{digit} & 2 \\ \frac{ 453 }{ 58 } & 453^2 - 61 \cdot 58^2 = 5 & \mbox{digit} & 2 \\ \frac{ 1070 }{ 137 } & 1070^2 - 61 \cdot 137^2 = -9 & \mbox{digit} & 1 \\ \frac{ 1523 }{ 195 } & 1523^2 - 61 \cdot 195^2 = 4 & \mbox{digit} & 3 \\ \frac{ 5639 }{ 722 } & 5639^2 - 61 \cdot 722^2 = -3 & \mbox{digit} & 4 \\ \frac{ 24079 }{ 3083 } & 24079^2 - 61 \cdot 3083^2 = 12 & \mbox{digit} & 1 \\ \frac{ 29718 }{ 3805 } & 29718^2 - 61 \cdot 3805^2 = -1 & \mbox{digit} & 14 \\ \frac{ 440131 }{ 56353 } & 440131^2 - 61 \cdot 56353^2 = 12 & \mbox{digit} & 1 \\ \frac{ 469849 }{ 60158 } & 469849^2 - 61 \cdot 60158^2 = -3 & \mbox{digit} & 4 \\ \frac{ 2319527 }{ 296985 } & 2319527^2 - 61 \cdot 296985^2 = 4 & \mbox{digit} & 3 \\ \frac{ 7428430 }{ 951113 } & 7428430^2 - 61 \cdot 951113^2 = -9 & \mbox{digit} & 1 \\ \frac{ 9747957 }{ 1248098 } & 9747957^2 - 61 \cdot 1248098^2 = 5 & \mbox{digit} & 2 \\ \frac{ 26924344 }{ 3447309 } & 26924344^2 - 61 \cdot 3447309^2 = -5 & \mbox{digit} & 2 \\ \frac{ 63596645 }{ 8142716 } & 63596645^2 - 61 \cdot 8142716^2 = 9 & \mbox{digit} & 1 \\ \frac{ 90520989 }{ 11590025 } & 90520989^2 - 61 \cdot 11590025^2 = -4 & \mbox{digit} & 3 \\ \frac{ 335159612 }{ 42912791 } & 335159612^2 - 61 \cdot 42912791^2 = 3 & \mbox{digit} & 4 \\ \frac{ 1431159437 }{ 183241189 } & 1431159437^2 - 61 \cdot 183241189^2 = -12 & \mbox{digit} & 1 \\ \frac{ 1766319049 }{ 226153980 } & 1766319049^2 - 61 \cdot 226153980^2 = 1 & \mbox{digit} & 14 \\ \end{array} $$