The Szemeredi regularity lemma for graphs uses the definition of $\epsilon$-pseudo-randomness:
Definition: For $\epsilon>0$, a pair of vertex sets $X$ and $Y$ is called $\epsilon$-pseudo-random, if for all subsets $A\subseteq X,~B\subseteq Y$ satisfying $|A|\geq\epsilon |X|$, $|B| \geq \epsilon |Y|$, we have that $|d(X,Y) - d(A,B)| \leq \epsilon$.
Is there an equivalent notion for graphons? Can we simply replace the cardinalities of the vertex sets with the equivalent notion of measures of sets in the graphon setting? Does an equivalent statement of the standard Regularity Lemma hold for graphons?
Regularity Lemma: For every $\epsilon >0$ and positive integer $m$ there exists integer $M$ such that if $G$ is a graph with at least $M$ vertices, there exists integer $k$ in the range $m\leq k \leq M$ and an $\epsilon$-regular partition of the vertex set of $G$ into $k$ sets.
Though I have not read Chapter 9 of Lovász's Large Networks and Graph Limits as thoroughly as I'd like, I can say that section 9.2 will do a good job answering your question. So-called "weak regularity lemmas" for graphs and graphons get better bounds on the number of steps, at the cost of weakening the type of approximation by a Szemerédi partition: rather than $\epsilon$-pseudo-random (or -regular), a weak regularity lemma approximates the graph(on) in small cut-norm ($\epsilon$-homogeneity, it seems to be called often) rather than $L_1$-norm (also known as edit distance). Back on topic, we might as well discuss the strong regularity lemmas in both contexts.
Strong Regularity Lemma for Graphs Let any sequence $\epsilon = (\epsilon_0;\epsilon_1,\epsilon_2,\dots)$ of positive numbers be specified. Then there exists a positive integer $S(\epsilon)$ such that any graph $G = (V,E)$ has an equitable partition $\mathcal{P}$ and an equitable partition $\mathcal{P}$ of $V$ into some $k\leq S(\epsilon)$ number of parts and into some $k\leq S(\epsilon)$ number of parts and some $G' = (V,E')$ such that $$d_1(G,G')\leq\epsilon_0\text{ and }d_\square(G',G'_{\mathcal P})\leq\epsilon_k$$ where $G_{\mathcal P}'$ is the graph on the $k$ blocks of the partition with edges weighted by the respective edge densities.
Here, $d_1$ and $d_\square$ are the normalized edit distance and normalized cut distance, respectively and you would need to muse over what $d_\square$ means here since $G'$ and $G_{\mathcal P}'$ don't have the same number of vertices (think distance of blowups on the same number of vertices). Note also that $d_1$ is less forgiving than $d_\square$: $d_\square\leq d_1$ always and a decrease in $d_\square$ does not imply a decrease in $d_1$.
Strong Regularity Lemma for Kernels (e.g., graphons) Let any sequence $\epsilon = (\epsilon_0;\epsilon_1,\epsilon_2,\dots)$ of positive numbers be specified. Then there exists a positive integer $S(\epsilon)$ such that any graphon $W$ has another graphon $W'$ and a stepfunction $U$ (also a graphon) with some $k\leq S(\epsilon)$ number of parts such that $$d_1(W,W')\leq\epsilon_0\text{ and }d_\square(W',U)\leq\epsilon_k. $$