From Wikipedia:
Not all numbers of a given length are equally hard to factor. The hardest instances of these problems (for currently known techniques) are semiprimes, the product of two prime numbers.
There are no references to this claim.
Why is a prime factorization harder than a not-prime factorization?
When the best factoring algorithms are analyzed, their runtimes turn out to be a function of the size of the smallest prime factor of the number being factored. We define "harder to factor" as requiring longer runtimes for factorization algorithms. So to maximize the runtime of the algorithm, you need to maximize the size of the smallest prime factor. This hapopens when you have a semiprime with 2 factors, both of which are of similar magnitudes.