I'd like to present a symbolic expression in mathematical form for any number of members from 3 to n ($m=3...n$).
The expression for $m=3$ is:
$$ \begin{split} M_3 &= R_1 R_2 R_3^2 \cos(\alpha_1 + \alpha_2 - 2\alpha_3) \\ &+ R_1 R_2^2 R_3\cos( \alpha_1 - 2 \alpha_2 + \alpha_3) \\ &+ R_1^2 R_2 R_3 \cos(\alpha_2 - 2 \alpha_1 + \alpha_3) \\ \end{split} $$
and for $m=4$, it looks like this:
$$ \begin{split} M_4 &= R_1 R_2 R_3^2 \cos(\alpha_1 + \alpha_2 - 2 \alpha_3) \\ &+ R_1 R_2^2 R_3 \cos(\alpha_1 - 2 \alpha_2 + \alpha_3) \\ &+ R_1^2 R_2 R_3 \cos(\alpha_2 - 2 \alpha_1 + \alpha_3) \\ &+ R_1 R_2 R_4^2 \cos(\alpha_1 + \alpha_2 - 2 \alpha_4) \\ &+ R_1 R_2^2 R_4 \cos(\alpha_1 - 2 \alpha_2 + \alpha_4) \\ &+ R_1^2 R_2 R_4 \cos(\alpha_2 - 2 \alpha_1 + \alpha_4) \\ &+ R_1 R_3 R_4^2 \cos(\alpha_1 + \alpha_3 - 2 \alpha_4) \\ &+ R_1 R_3^2 R_4 \cos(\alpha_1 - 2 \alpha_3 + \alpha_4) \\ &+ R_1^2 R_3 R_4 \cos(\alpha_3 - 2 \alpha_1 + \alpha_4) \\ &+ R_2 R_3 R_4^2 \cos(\alpha_2 + \alpha_3 - 2 \alpha_4) \\ &+ R_2 R_3^2 R_4 \cos(\alpha_2 - 2 \alpha_3 + \alpha_4) \\ &+ R_2^2 R_3 R_4 \cos(\alpha_3 - 2 \alpha_2 + \alpha_4) \\ \end{split}$$
It follows a pattern, but I do not now how to present it in mathematical form. Any help is appreciated.
How about something like this:
Define:
$$ \begin{align*} R_0 &= 1 \\ \alpha_0 &=0 \\ \Xi (i, j;N) &\equiv ({R_i\over R_j}{R_{1}R_{2}\dots R_i \dots R_N}) \cos({-\alpha_j-3\alpha_i+\sum_{k=1}^{N}{\alpha_k}}) \\ &\equiv ({R_i\over R_j}{\prod_{k}^{N}{R_k}})\cos(-\alpha_j-3\alpha_i+\sum_{k=1}^{N}{\alpha_k}) \end{align*} $$
The reason for $\equiv$ rather than $=$ is to establish it as an identity so even if $R_j = 0$ the identity holds.
So,
$$ \begin{align*} \Xi(1,0;3) &\equiv {{R_1}\over{R_0}}{R_1R_2R_3}\cos(-\alpha_0 - 3\alpha_1 + \sum_{k=1}^{3}{\alpha_k}) &= R_1^2R_2R_3 \cos(-2\alpha_1 +\alpha_2+\alpha_3) \\ \Xi(2,0;3) &\equiv {{R_2}\over{R_0}}{R_1R_2R_3}\cos(-\alpha_0 - 3\alpha_2 + \sum_{k=1}^{3}{\alpha_k}) &= R_1R_2^2R_3 \cos(\alpha_1-2\alpha_2+\alpha_3) \\ \Xi(3,0;3) &\equiv {{R_3}\over{R_0}}{R_1R_2R_3}\cos(-\alpha_0 - 3\alpha_3 + \sum_{k=1}^{3}{\alpha_k}) &= R_1R_2R_3^2 \cos(\alpha_1-2\alpha_2+\alpha_3) \\ \quad &\vdots \\ \Xi(2,3;4) &\equiv {{R_2}\over{R_3}}{R_1R_2R_3R_4}\cos(-\alpha_3 - 3\alpha_2 + \sum_{k=1}^{4}{\alpha_k}) &= R_1R_2^2R_4\cos(\alpha_1-2\alpha_2+\alpha_4) \end{align*}$$
Now, let's map your $M$s:
$$ \begin{align*} M_3 = \Xi(1,0;3) + \Xi(2,0;3) + \Xi(3,0;3) = \sum_m{\Xi(m, 0; 3)} \end{align*} $$ $$ \begin{align*} M_4 = \quad &\Xi(1,4;4) + \Xi(2,4;4) + \Xi(3,4;4) &(= R_1^2R_2R_3\cos(-2\alpha_1+\alpha_2+\alpha_3)+\dots) \\ +&\Xi(1,3;4) + \Xi(2,3;4) + \Xi(4,3;4) \\ +&\Xi(4,2;4) + \Xi(1,2;4) + \Xi(3,2;4) \\ +&\Xi(4,1;4) + \Xi(3,1;4) + \Xi(2,1;4) \\ \end{align*} $$
Since I don't have the $M_5$ example, I cannot really be definitive. However there's a pattern emerging for $\Xi(\_,\_;4)$ that can be converted to a more compact form.
Rearranging the terms above a little: $$ \begin{align*} M_4 = \quad &&+\Xi(2,1;4) &+ \Xi(3,1;4) &+ \Xi(4,1;4) \\ &+\Xi(1,2;4) &&+ \Xi(3,2;4) &+ \Xi(4,2;4) \\ &+\Xi(1,3;4) &+ \Xi(2,3;4) &&+ \Xi(4,3;4) \\ &+\Xi(1,4;4) &+ \Xi(2,4;4) &+ \Xi(3,4;4) \end{align*} $$
Now obviously the terms that are missing are of the form $\Xi(A,A;N)$ so you can simply define $\Xi(A,A;N) \equiv 0$
Which means the definition above becomes something like:
$$ \Xi (i, j;N) \equiv \begin{cases} ({R_i\over R_j}{\prod_{k}^{N}{R_k}})\cos(-\alpha_j-3\alpha_i+\sum_{k=1}^{N}{\alpha_k}) & \forall {i\neq j}\\ 0 \end{cases} \\ R_0 = 1; \alpha_0 = 0; i=1,2,3\dots; j=1,2,3\dots; N\in\mathbb{Z_+} $$
Given this new definition we can:
$$ \begin{align*} M_4 = \quad&\Xi(1,1;4) &+& \Xi(2,1;4) &+& \Xi(3,1;4) &+& \Xi(4,1;4) \\ + &\Xi(1,2;4) &+& \Xi(2,2;4) &+& \Xi(3,2;4) &+& \Xi(4,2;4) \\ + &\Xi(1,3;4) &+& \Xi(2,3;4) &+& \Xi(3,3;4) &+& \Xi(4,3;4) \\ + &\Xi(1,4;4) &+& \Xi(2,4;4) &+& \Xi(3,4;4) &+& \Xi(4,4;4) \end{align*} $$
Now we can simplify the expression by defining $M_N$ as (order of summation is changed but works)
$$ M_N = \sum_{i=1}^{N}{\sum_{j=1}^{N}{\Xi(i,j;N)}} $$