I want to find the minimum size of edge cut that separates $K_{3n}$ into three components. I am thinking for the three components after separation, they will all be complete, with orders $x,y$ and $z$. The relations are $x+y+z=3n$, and finding the $\max ((x-1)x/2+(y-1)y/2+(z-1)z/2)$ will lead to the answer. But I am stuck with finding the maximum.
What should I do to continue? Or am I in a wrong direction.
If you have done the two-component version, then you have seen that the minimum edge cut there separates a $1$-vertex component from an $n-1$ vertex component using $n-1$ edges. So it makes sense to conjecture that the optimal solution here will have two $1$-vertex components and an $(n-2)$-vertex component.
This has $2n-3$ edges, which is pretty low. Can we do better?
Well, it takes $k(n-k)$ edges to separate a component of order $k$ from the rest of the graph. This is already more than $2n-3$, unless $k \in \{1, 2, n-2, n-1\}$. (You can check this part with calculus, if you like.) So each of our components must have one of these orders.
The only way to get three components of these orders to add up to $n$ is to take $1$, $1$, and $n-2$, provided that $n > 6$.