Notation: $M_2^c$ are continuous square integrables martingales, $\langle X\rangle_t$ is the quadratic variation and $V_t^{(2)}(\Pi)$ the sum of square of increments of $X$ over the partition $\Pi$.
In the book Brownian motion and stochastic calculus by Karatzas and Shreves, 3 results are used to motivate the study of quadratic variation.
Theorem 5.8. Let $X$ be in $M_2^c$. For partition $\Pi$ of $[0,t]$, we have $\lim_{\|\Pi\|\to0} V_t^{(2)}(\Pi)=\langle X\rangle_t$ (in probability); i.e., for every $\varepsilon>0$,$\eta>0$ there exists $\delta>0$ such that $\|\Pi\|<\delta$ implies $$P[|V_t^{(2)}(\Pi)-\langle X\rangle_t|>\varepsilon]<\eta.$$
5.11 Problem. Let $\{X_t,\mathcal{F}_t;0\le t<\infty\}$ be a continuous process with the property that for each fixed $t>0$ and for some $p>0$, $$\lim\limits_{\|\Pi\|\to0}V_t^{(p)}(\Pi)=L_t\;\;\;\text{(in probability),}$$ where $L_t$ is a random variable taking values in $[0,\infty)$ a.s. Show that for $q>p$, $\lim_{\|\Pi\|\to0}V_t^{(q)}(\Pi)=0$ (in probability), and for $0<q<p, \lim_{\|\Pi\|\to0}V_t^{(p)}(\Pi)=\infty$ (in probability) on the event $\{L_t>0\}$.
5.12 Problem. Let $X$ be in $M_2^c$, and $T$ be a stopping time of $\{\mathcal{F}_t\}$. If $\langle X\rangle_T=0$, a.s. $P$, then we have $P[X_{T\wedge t}=0;\forall 0\le t<\infty]=1$.
The conclusion to be drawn from Theorem 5.8 and Problems 5.11 and 5.12 is that for continuous, square-integrable martingales, quadratic variation is the "right" variation to study. All variations of higher order are zero, and, except in trivial cases where the martingale is a.s. constant on an initial interval, all variations of lower order are infinite with positive probability. Thus, the sample paths of continuous, square-integrable martingales are quite different from "ordinary" continuous functions. Being of unbounded first variation, they cannot be differentiable, nor is it possible to define integrals of the form Ito $\int_0^t Y_s(\omega) dX_s(\omega)$ with respect to $X\in M_2^c$, in a pathwise (i.e., for every or P-almost every $\omega\in\Omega$), Lebesgue-Stieltjes sense.
First, I don't understand how problem 5.12 comes into play and how it is used to argue that the quadratic variation is "right" variation to study (see last quoted paragraph).
Second, I don't understand how they can deduce from this discussion that we cannot define a pathwise integral with respect to martingales of unbounded variations.
What this is saying is that a martingale is either constant or it has non-vanishing (and locally finite) quadratic variation. This highlights the fact that the quadratic order is the right order at which to study the variation of martingales.
The Lebesgue-Stieltjes theory allows you to define integrals where your integrators have bounded variation, as functions of bounded variation are the difference of two increasing functions (which in turn can be associated with measures). If you wish to define a stochastic integral with a sufficiently rich class of integrands, a pathwise integral will fail for the following reason:
Theorem: Suppose $G:[0,T] \to \mathbb{R}$ is a function for which the Riemann-Stieltjes integral $\int_0^T f dG$ exists for all continuous functions integrands $f$. Then $G$ is of bounded variation.
Proof. This is a consequence of the Banach-Steinhaus theorem. $\blacksquare$
A pathwise stochastic calculus does exist (e.g. see here for some references); it certainly is more elementary than Itô calculus, but the class of integrands is too restrictive for some applications.