The question is:
Let $V$ be an $n$-dim space, find m vectors $\{\alpha_k\}_{k=1}^m$ in $V$ such that the value $\max\limits_{i\neq j}|\cos(\alpha_i,\alpha_j)|$ is minimal.
Here, one might assume that $m>n$ (it is trivial when $m\leq n$).
Below is what I tried.
- If $n=2$, one might choose m vectors to divide the plane into 2m equal parts.
- Besides, the question is equivalent to the following:
Find $P\in M_{m\times m}$ a semidefine matrix such that:
- $\operatorname{rank}(P)=n$
- $P_{ii}=1,\forall i=1,\cdots,n$
- $\max\{|P_{ij}|:i\neq j\}$ is minimal
This feels like a classical question, but hard to solve when $n\geq 3$.
Just some thought, assuming we are in $n$-dimensional Euclidean space:
The length of the vectors does not play a role because $\cos(v,w) = \cos(v,w/\|w\|$). This means we can project the vectors on the $n-1$ dimensional (unit) sphere. The task is then to maximize the distances of these points.
However, minimizing $|\cos(v,w)|$ means we have to maximize the distance between $v$ and $w$ and between $v$ and $-w$ simultaneously because $\cos(v,-w) = -\cos(v,w)$. This means we have to identify each point $P$ on the sphere with its antipodal $-P$, which effectively turns the sphere into projective space due to the identification $P=-P$ (and because the sphere is the double cover of projective space). This turns your problem into a sphere-packing optimization problem in $n-1$ dimensional projective space.
For example, in $\Bbb R^2$ we get the sphere $\Bbb S^1$, and as you already said it is easy to arrange $m$ points in such a way that their minimal distance gets maximized: Due to the identificaton with the antipode we have effectively $2m$ points, and an optimal distribution is to have $m$ lines at angles $180^\circ /m$ so that the end-points of a line on $\Bbb S^1$ has a distance of $\pi/m$ to the neighboring point.
In dimension 3 the problem is already much more complicated because we have to arrange $m$ lines through the origin of the 2-sphere. If $m=3$ we can take the corners of an octahedron projected on the 2-sphere which gives 3 pairs of antipodal points, and the lines that connect antipodal pairs intersect at $90^\circ$. This is just like a Cartesian right-angled coordinate system.
If $m=4$ we project the corners of a cube onto $\Bbb S^2$, and for $n=6$ we can take a regular icosahedron with its 12 vertices. Using the tetrahedron $T$ I am unsure what to make of it, but $T\cup-T$ should give the same four vertice-pairs like from the cube. Dodecahedron $D$ can maybe used as $D\cup-D$ in the case $m=20$, where the vertices are no more derivable from a Platonic solid, and it's not from some Archimedian solid either, because there is no Archimedian solid with 40 vertices. It's a body made from 60 congruent, isosceles triangles.
In general, the problem will be hard and you won't find explicit solutions of this best-sphere-packing-in-projective-space problem. And even if you find a solution that's optimal, you will have a hard time proving that it is actually optimal.