Show that $p\gt 3, 2p+1, 4p+1$ cannot be both prime
I know that for testing if an integer is odd, need check if its square is of the form $\exists k\in \mathbb{Z}, 8k+1$.
If $p\gt 3, 2\nmid p$, so $p$ is odd.
So, one way can be to show that the congruence class $8k+1$ cannot be split into $(2p+1)\cdot(4p+1) = 8p^2+6p+1 \ne \exists k'\in \mathbb{Z},\, 8k'+1$.
I am asking this for addtl. reason that the solution provided by a book states that :
A prime can only take the form $3k-1$, or $3k+1$.
If $p=3k+1$, then
$$2p+1=2(3k+1)+1=3(2k+1),$$
so that $2p+1$ is composite. If $p=3k-1,$ then$$4p+1=4(3k-1)+1=3(4k-1),$$
so that $4p+1$ is composite.
I believe the solution approach is faulty, for the reason that it is not necessary for the integers of the form $3k-1, 3k+1$ to be prime.
For your initial attempt:
if $x$ is an odd number, then $x^2 \equiv 1 \pmod{8}$.
However, if $x$ is odd and $y$ is odd, it is possible that $xy \not\equiv 1 \pmod{8} $. For example $$9 \cdot 7 = 63 \equiv 7 \pmod{8}$$
For the book attempt:
Edit (for book's approach)
Case $1$: $p=3k$, since $p>3$, $k>1$, but this mean $p$ is not a prime, hence it is a contradiction. Case $1$ doesn't happen.
Case $2$: $p=3k-1$, then $4p+1=3(4k-1)$, hence $4p+1$ is not a prime.
Case $3$: $p=3k+1$, then $2p+1=3(2k+1)$ is not a prime.