I know that $\int \frac{1}{x}~dx = \ln|x| + C$ and I know the antiderivative method works for all powers of $x$ except $-1$. But why is that the case? I am still in high school and teachers aren't really helpful with these questions.
Edit: I do realize dividing by zero is meaningless. That's not the question. Just because dividing by zero is meaningless doesn't mean mathematicians just choose to ignore it and work just make up a new answer. There has to be an explanation, and that's what I am asking for. The explanation.
I think it is important to understand how the formula $$\int x^{n}\,dx = \frac{x^{n + 1}}{n + 1} + C\tag{1}$$ is coming into picture. Well, it is a result of inverting the differentiation formula $$\frac{d}{dx}x^{n + 1} = (n + 1)x^{n}\tag{2}$$ If $n + 1\neq 0$ then we can divide the formula $(2)$ by $(n + 1)$ to get $$\frac{d}{dx}\frac{x^{n + 1}}{n + 1} = x^{n}\tag{3}$$ If we put $n = -1$ then the formula $(2)$ remains valid (check!) but the division by $(n + 1)$ is not allowed and hence the formula $(3)$ becomes meaningless and therefore it can not be inverted to get the integral formula $(1)$.
So it is not possible to invert differentiation formula like $(2), (3)$ to obtain $(1)$ in case $n = -1$. The natural question to ask now is:
However before answering the above question, it makes sense to ask one further question:
This is where the Fundamental Theorem of Calculus (FTC) comes into picture which provides us with the following:
And the above is true even if $[a, b]$ is replaced by $(a, b)$.
The function $f(x) = x^{-1}$ under consideration is continuous on intervals $(-\infty, 0)$ and $(0, \infty)$ and hence on each of these intervals it does possess an anti-derivative.
The FTC also gives us an expression for the anti-derivative in terms of what is usually called a definite integral (or more technically a Riemann integral). Thus if $x > 0$ then the following integral $$F(x) = \int_{1}^{x}\frac{dt}{t}$$ is an anti-derivative of $f(x) = 1/x$ in interval $(0, \infty)$.
In case the presentation of calculus is such that anti-derivatives are studied before Riemann integral (this is the more common approach used in introductory calculus courses), then it is sufficient to assume (via the use of corollary of FTC mentioned above) that there exists an anti-derivative $F(x)$ of $f(x) = 1/x$ for $x > 0$ with the following properties $$F(1) = 0, F(xy) = F(x) + F(y), F(x/y) = F(x) - F(y), F'(x) = 1/x\tag{4}$$ for all positive $x, y$. The function $F$ introduced above with some nice / curious properties is traditionally denoted by $\log x$ (or by $\ln x$) and is famously known as the logarithm function. It is now easy to prove that $\log (-x)$ is anti-derivative of $1/x$ for $x < 0$ and thus we have the formula $$\int \frac{dx}{x} = \log |x| + C\tag{5}$$
Looking at the formulas $(1)$ and $(5)$ an inquiring mind may ask if there is any connection between these formulas. There is a connection between the formulas $(1)$ and $(5)$ but it is visible only when we switch to definite integrals and the connection has been amply described in the answer from user "Simply Beautiful Art". But unless one has studied definite integrals it is best to think of $(1)$ and $(5)$ as separate formulas.