The numbers reading across and down in these squares are square: $\begin{array}{ccc} 1 & 4 & 4\\ 4 &8&4\\ 4&4&1 \end{array}$
$\begin{array}{ccc} 5&2&9\\ 2&5&6\\ 9&6&1 \end{array}$
$\begin{array}{cccc} 2&1&1&6\\ 1&2&2&5\\ 1&2&9&6\\ 6&5&6&1 \end{array}$
Are there any such square squares where the diagonals are also squares? If not in base 10, is it possible in other bases?
Those matrices are certainly hard to find, I tried it a lot and found that there are no $4\times4$ or $3\times3$ matrices that do what you want in base $10$ or below. However watch this one in base $11$:
$$\begin{array}{ccc} 1 & 9 & 5\\ 9 & 6 & 1\\ 5 & 1 & 9 \end{array}$$
You have
$169_{11}=196=14^2$
$195_{11}=225=15^2$
$519_{11}=625=25^2$
$565_{11}=676=26^2$
$961_{11}=1156=34^2$
All colums, rows, diagonals when read in any way (colums and rows only down and right) are squares, woha ;-)