According to Terence Tao's book Analysis I, the axiom of substitution is one of the four axioms of equality for objects (which are basically eligible set elements). In particular, it says that for any two objects $x$ and $y$ of the same type, if $x$ = $y$, then $f(x)=f(y)$ for all functions and operations $f$, and that furthermore, $P(x)$ and $P(y)$ are logically equivalent propositions (i.e. they imply each other) for all propositions $P$ whose truth-value depends on $x$ and $y$.
Sometimes, the axiom of substitution makes sense (i.e. is correct), such as when considering sets to be the object in question (they are indeed valid objects according to the first set-theory axiom presented in the book), and considering the operation $f(X) = X \cup C$, where $C$ is some arbitrary set. Then indeed, one may verify that if two sets $A$ and $B$ are equal to each other (i.e. $A=B$), then $f(A)=f(B)$ for the operation $f$ defined above.
But there are times when it starts acting up. Still considering sets to be the object in question, let $P(X)$ be the proposition that the elements of $X$ are ordered from least to greatest (this proposition's truth value clearly depends on what the set $X$ is that you "feed" it so to speak). Evidently however, as Tao himself acknowledges, even if two sets $A$ and $B$ were equal to each other, because sets are unordered collections of elements, there are plenty of counterexamples to give that would thoroughly disprove the statement: $P(A)$ iff $P(B)$. So my question then is: does the axiom of substitution have limits to its applicability, and if so, can someone very explicitly clarify what they are? Thanks.
In short, it is not a problem of the axiom of equality but a problem that your "property" $P$ is not a property.
First of all, since you are "considering the object of sets", you cannot in general talk about "least" or "greatest". For instance, it is meaningless to talk about what element is larger for the set $\{\text{cat, banana}\}$.
Given a finite subset $S$ of integers, "whether the elements of $S$ are ordered from least to greatest" is not a property of $S$. This is similar to the scenario that $$ f(\frac{a}{b})=a+b\qquad a\in\mathbb{Z},\ b\in\mathbb{Z}\setminus\{1\}. $$ is not a well-defined function on the set of rational numbers. The value of the "function" depends on the concrete representations of the object of rational numbers. For instance, $\frac{1}{2}$ and $\frac{2}{4}$ are the same element in the set of rational numbers, but $1+2\ne 2+4$.
Returning to your case, $\{1,2\}$ and $\{2,1\}$ are two different "representations" (whatever that means) of the same object of sets. But your "property" depends on different "representations" and is thus not a property on a finite subset of integers.