How can one find a formula for $\big(1+\cos(x)+\cos(2x)+\cos(3x)+...+\cos((n-1)x)\big)^2$ in the form
$$ \Big(1+\cos(x)+\cos(2x)+\cos(3x)+...+\cos((n-1)x)\Big)^2=\sum _{i=0}^N A_i \cos (B_i \,x), $$
where $N$ depends only on $n,$ and the coefficients $A_i$ and $B_i$ depend only on $i$ and $n,$ but none of these depend on $x.$
On
Yes it is. It is helpful to note that for any $m,n$, $$ \cos(mx)\cos(nx) = \frac 12 \left[\cos((n+m)x) + \cos((n - m)x)\right] $$ From there, we have \begin{align} \left[\sum_{i=0}^N \cos(i)\right]^2 &= \sum_{i=0}^N\sum_{j=0}^N \cos(i) \cos(j) = \frac 12\sum_{i=0}^N\sum_{j=0}^N \left[\cos((i+j)x) + \cos((i-j)x)\right] \\ & = \sum_{j=0}^N\sum_{i \geq j} \left[\cos((i+j)x) + \cos((i-j)x)\right] \end{align}
On
Hint: The angles are in A.P., with a common difference of x.
Multiply and divide by $2\sin(x/2)$
As you multiply each term of the series by $2\sin(x/2)$ , make use of this trigonometric identity : $2\sin(A)\cos(B) = \sin(A+B) + \sin(A-B)$
The terms will cancel out, leaving you with a very simplified expression.
We'll show that
\begin{align}\boxed{\text{ } \\\quad\Big(\sum_{k=0}^{n-1} \cos(kx)\Big)^2= \frac{n+1}{2}+\sum_{k=1}^{n-1} \Big(n-\frac{k-1}{2} \Big)\cos(kx)+\sum_{k=n}^{2n-2} \Big(n-\frac{k+1}{2} \Big)\cos(kx).\quad\\} \end{align}
Note that this is in the form requested since the constant term is just the $\cos (0x)$ term.
Here's how to prove this:
\begin{align} \Big(\sum_{k=0}^{n-1} \cos(kx)\Big)^2&=\sum_{j=0}^{n-1}\sum_{k=0}^{n-1}\cos(jx)\cos(kx) \\&=\sum_{j=0}^{n-1}\sum_{k=0}^{n-1}\frac{\cos\big((j+k)x\big)+\cos\big((j-k)x\big)}{2} \\&=\frac12\Big(\sum_{j=0}^{n-1}\sum_{k=0}^{n-1} \cos\big((j+k)x\big)+\sum_{j=0}^{n-1}\sum_{k=0}^{n-1}\cos\big((j-k)x\big)\Big) \\&=\frac12\Big(\sum_{m=0}^{2n-2}\big(\text{# of ways of writing }m\text{ as a sum }j+k\text{ with }0\le j\le n-1\text{ and }0\le k\le n-1\big) \cos(mx) \\&\quad+\sum_{m=-(n-1)}^{n-1}\big(\text{# of ways of writing }m\text{ as }j-k\text{ with }0\le j\le n-1\text{ and }0\le k\le n-1\big) \cos(mx)\Big). \end{align}
If $\boxed{0\le m \le n-1},$ there are $\boxed{m+1\text{ ways of writing }m\text{ as }j+k},$ namely $0+m, 1+(m-1), 2+(m-2), \dots, m+0.$
If $\boxed{n \le m \le 2n-2},$ then $0 \le m-n \le n-2,$ and there are $\boxed{2n-m-1\text{ ways to write }m\text{ as }j+k},$ namely $(m-n+1)+(n-1), (m-n+2)+(n-2), \dots, (n-1)+(m-n+1).$
Turning to the differences now, if $\boxed{0\le m \le n-1},$ then there are $\boxed{n- m \text{ ways of writing }m\text{ as }j-k},$ namely $m-0, (m+1)-1, (m+2)-2, \dots, (n-1)-(n-m-1).$
Finally, if $\boxed{-(n-1)\le m \lt 0},$ then there are $\boxed{n+m \text{ ways of writing }m\text{ as }j-k},$ namely $0-\lvert m \rvert, 1-(1+\lvert m \rvert), 2-(2+\lvert m \rvert), \dots, (n+m-1)-(n-1).$
It follows that the quantity that we're computing, which is half of $$\sum_{m=0}^{2n-2}\big(\text{# of ways of writing }m\text{ as a sum }j+k\text{ with }0\le j\le n-1\text{ and }0\le k\le n-1\big) \cos(mx) \\\quad\quad+\sum_{m=-(n-1)}^{n-1}\big(\text{# of ways of writing }m\text{ as }j-k\text{ with }0\le j\le n-1\text{ and }0\le k\le n-1\big) \cos(mx),$$ is equal to half of
\begin{align} &\sum_{m=0}^{n-1} (m+1) \cos(mx) + \sum_{m=n}^{2n-2}(2n-m-1) \cos(mx) + \sum_{m=0}^{n-1}(n-m)\cos(mx) + \sum_{m=-(n-1)}^{-1} (n+m)\cos(mx) \\&= \sum_{m=0}^{n-1} (n+1) \cos(mx) + \sum_{m=n}^{2n-2}(2n-m-1) \cos(mx) + \sum_{m=-(n-1)}^{-1} (n+m)\cos(mx). \end{align}
So we find that \begin{align}{\quad\Big(\sum_{k=0}^{n-1} \cos(kx)\Big)^2= \sum_{m=-(n-1)}^{2n-2} \frac12 B_m \cos(mx)\quad} \end{align}
where
$${\quad B_m=\begin{cases} m+n, &\text{ if }-(n-1)\le m\lt 0,\quad \\n+1, &\text{ if }0 \le m \lt n,\quad \\2n-m-1,&\text{ if }n \le m \le 2n-2.\quad \end{cases}}$$
Finally, we can use the identity $\cos(-mx)=\cos(mx)$ to combine terms, and get
\begin{align}\boxed{\quad\Big(\sum_{k=0}^{n-1} \cos(kx)\Big)^2= \sum_{m=0}^{2n-2} A_m \cos(mx)\quad} \end{align}
where
$$\boxed{\quad A_m=\begin{cases} \frac{n+1}{2}, &\text{ if }m=0,\quad \\n-\frac{m-1}{2}, &\text{ if }1 \le m \lt n,\quad \\n-\frac{m+1}{2},&\text{ if }n \le m \le 2n-2.\quad \end{cases}}$$
And you can write this as
\begin{align}\boxed{\text{ } \\\quad\Big(\sum_{k=0}^{n-1} \cos(kx)\Big)^2= \frac{n+1}{2}+\sum_{k=1}^{n-1} \Big(n-\frac{k-1}{2} \Big)\cos(kx)+\sum_{k=n}^{2n-2} \Big(n-\frac{k+1}{2} \Big)\cos(kx).\quad\\} \end{align}