The textbook I'm reading says this for this problem:
$$x^2y''+2xy' − 2y=0.$$
Since differentiating a power pushes down the exponent by one unit, the form of this equation suggests that we look for possible solutions of the type $y=x''$. On substituting this in the differential equation and dividing by the common factor $x''$, we obtain the quadratic equation $n(n − 1)+2n − 2=0$
What do they mean by this substitution, taking it like a literal substitution gives this nonsense which I obviously can't divide out the common factor $x''$
Literal substiution gives $$x^2(x'')'' + 2x(x'')' -2(x'') = 0$$
I can't divide everything by $x''$ because not everything is being multiplied by $x''$, for example, $(x'')''$ is the second derivative of $x''$ so I can't factor out $x''$ from that. What does the author mean by substituting $y=x''$, how did he end up with a quadratic?

That is a typographical error in the book. Both occurrences of $x''$ should be $x^n$.
$x$ is the independent variable, so $y = x''$ is the zero function, i.e. the trivial solution of the differential equation.
With $y = x^n$ everything falls into place: $y' = n x^{n-1}$, $y = n(n-1) x^{n-2}$, and substitution into the differential equation gives $$ n(n − 1)x^n +2nx^n − 2x^n=0 $$ so that the common factor $x^n$ can be divided out: $$ n(n − 1)+2n − 2=0 \, . $$
Remark: From a Google search is seems that the quote is from Differential Equations with Applications and Historical Notes, Third Edition, by George F. Simmons, which really has that typographical error.