I am currently trying to prove that the ordinary least squares estimate doesn't have a bias with a given dataset 
with the bias given as
Why does this identity hold in the following calculation $$(X^TX)^{-1}(X^TX)\theta = \theta ?$$
The matrix $X$ is assumed to be fixed in this case.
$n$ is the number of samples and $d$ is the number of features.
$n >> d$ does not ensure the invertibility of $X^TX$.
If $X$ has $d$ linearly independent columns, then $X^TX$ is invertible. If $X$ follows certain distribution, say $X$ are samples drawn from contious uniform distribution or normal distribution, then the probability that it is invertible is $1$.
Hence we can write the notation $(X^TX)^{-1}$, being the inverse of $X^TX$, multiplying them both would give us the identity matrix.