My professor said that converting a language to a CFG is more of an art than anything else. I looked at this problem and didn't even know how to get started, or how I would reason my way to the solution (which I understand how it is correct).
Are there any useful patterns to deal with these conversions?
On
If we rewrite $R$ as $\{\, S \to \epsilon, S \to SaSbS, S \to SbSaS \,\}$, it's obvious that every word produced by the CFG is in the language, because no rule causes an imbalance in the number of $a$'s and $b$'s.
In the other direction, one can use induction to show how to derive any balanced word $w$ (one with equal numbers of $a$'s and $b$'s) from $S$.
If the length $|w|$ of $w$ is less than $2$, it's obviously possible: There are no words of length $1$ in the language, and $\epsilon$ is produced by $S \to \epsilon$.
Otherwise we consider two cases, based on the first letter of $w$. Suppose $w$ starts with $a$. (The other case is symmetric.) Then
$$ S \to SaSbS \to aSbS \enspace. $$
We now have a word $u$ such that $w = au$ and $u$ has one more $b$ than it has $a$'s. Hence it can be written as $s_1bs_2$ in such a way that each $s_i$ has the same number of $a$'s and $b$'s (and is possibly empty). Both $s_1$ and $s_2$ are strictly shorter than $w$; hence the induction hypothesis applies to each $s_i$ and we are done.
There's more than one way to split $u$ into $s_1$, $b$, and $s_2$ so that $s_1$ and $s_2$ are balanced. A simple one is to let $s_1$ be the longest balanced prefix of $u$. The next letter of $u$ then necessarily exists and is a $b$; $s_2$ is whatever is left.
If we look back at the proof, we realize that the following set of rules will do:
$$\begin{align} S &\to \epsilon \\ S &\to aSbS \\ S &\to bSaS \enspace. \end{align}$$
This illustrates the frequent usefulness of proofs beyond the obvious primary goal of establishing correctness.
On
One common mistake is :
$S\rightarrow \epsilon$, $S\rightarrow SabS$, $S\rightarrow SbaS$.
is also an answer. Because any nonempty string with same numbers of $a$ and $b$ must has one $a$ right next to one $b$ somewhere.
That answer is wrong, because the two parts got from the ab or ba splitting may be not $S$ either.
For similar reason, the answer from Fabio Somenzi is also wrong:
$S\rightarrow \epsilon$, $S\rightarrow aSbS$, $S\rightarrow bSaS$.
One counterexample is "aaaabbbb", it can not be generated by these rules.
For the language given, you need the number of $a$'s to match up with the number of $b$'s. Notice that the strategy used to find a CFG for the language is to make sure that whenever we introduce an $a$, that we also introduce a $b$ at the same time. By doing this, we make sure that the number of $a$'s and $b$'s are equal. Also, whenever we introduce an $a$ or a $b$, we also want the ability to put substrings on either side of them or inbetween them, hence the $S \to SASBS$ and $S \to SBSAS$ production rules. Notice that since the empty string also has the same number of $a$'s and $b$'s, then we need the $S \to \varepsilon$ production rule.
From this point, it is clear that everything that is in the grammar is also in the language. What's not so clear is that everything in the language is produced by the grammar; however, it turns out that these production rules suffice for generating the entire language (this isn't too difficult to prove).