Hierarchy theory is the theory obtained by adding a one place function $V$ to the first order language of set theory.
Define $``ordinal"$ along Von Neumann's, also $``<"$ is defined in the usual manner over von Neumann ordinals only.
Add axioms of: Extensionality; Separation, stipulated in the usual manner.
Stages: $\forall \ ordinal \ \alpha: V_\alpha = \{x \subseteq V_\beta | \beta < \alpha\}$
Foundation: $\forall x \exists \ ordinal \ \alpha \ ( x \in V_\alpha )$
Infinity: $\exists \alpha [\alpha \neq \emptyset \land \forall \beta < \alpha \exists \gamma (\beta < \gamma < \alpha)]$
Height: $\forall x [x \text{ is well ordered } \to \exists \ ordinal \ \alpha: \alpha= ord(x)] $
Where $ord(x)$ means the order type of $x$.
/ Theory definition finished.
I think the above axioms are the ones that truly capture the notion of an infinite hierarchy of sets that is built from below, the last axiom [to me] makes full sense and it does belong to the notion of a hierarchy itself, since it reflects the build up from below notion, i.e. we can add up heights to the hierarchy as long as those are formed at earlier stages!
Now of course we can add up further ordinal lengths like: fixed points, inaccessible, Mahlo, etc.. and thereby obtain higher hierarchies. On the other hand we can thin up stages by adding axioms stating that all sets are well ordered, or that all of them are constructible, thereby filtering out any discourse about non-well ordered sets, or about non-constructible sets.
However those additions of lengths to the hierarchy and those thinnings of its stages, I don't see any of them really stemming from the very basic notion of hierarchy itself, they can be conceived as manipulations on that notion rather than stemming essentially from it. However these manipulations seem to preserve the notion of a hierarchy, and so can be seen as constituting different versions of it.
So for example $\sf ZFC$ can be seen as a special case of a hierarchy, namely when the hierarchy has inaccessible height and thinned out to have only well ordered sets.
So the overall notion of a hierarchy with various lengthening and\or thinning of it, is a bigger notion than the axioms of $\sf ZFC$, and it is the foundational notion about sets and mathematics thereof. And I see the above theory being the basic form of it, not $\sf ZFC$ which only captures a special manipulation of its height and width.
What mathematical and foundational arguments can be held against this position?
There are a few problems.
Some mathematical: e.g. every real is a subset of $\omega$ and thus can be well-ordered, but if the reals cannot be well-ordered, you just said you can omit them, so in particular you won't have a von Neumann hierarchy for more than $\omega+1$ steps.
Others linguistic, who said that ZFC is the foundational theory? It's a popular one, yes, but there are others, and most mathematicians are not particularly picky about their foundations anyway.
Finally, philosophical. We know that ZFC is far stronger than "bare necessity" in most of mathematics. It's nothing new that Replacement, in some sense, is not needed for foundations. Replacement has its set theoretic merits (Reflection, for example) and a few foundational merits (implementation insensitivity which means it doesn't matter how you encode structures into set theory). But as a whole, it is not necessary for most of mathematics.
You seem to keep some of the consequences of Replacement (most notably the von Neumann hierarchy, and "enough ordinals"), but not all. Your theory is already "too strong" for most of mathematics, and it is not strong enough for set theoretic research, where Reflection plays a key role.
So what should we do with this theory? Yes, it's cute, it's interesting to ask what things it can't do, but ultimately, if you're interested in foundations for foundational sake, ditch Replacement altogether, and any Separation beyond $\Delta_0$ is unnecessary either. If you're really pressing it, kill power set and preserve, maybe, the power set of $\omega$, or go "full Pocket" and treat uncountable sets as classes.
But ultimately the question on "the foundation" is sociological. To quote a prominent mathematician replying to my question about AC when I was starting my M.Sc. studies, a good foundational theory is one that you don't notice when you do mathematics. If your foundations interferes with your mathematics, it's not the best foundations you can find. (Note, an inconsistent foundations does interfere with your mathematics, proving that $0=1$ is quite disruptive to mathematics.)