Let $p_i,q_j\in\mathbb{R}^3$ be sequences of points with $i,j\in\{1,2,3\}$. How can I solve the optimization problem
$$D:=\arg\min_{R,t}\left(\max_{i=1,2,3}\Vert p_i-(Rq_i+t) \Vert\right),$$
where $R$ is a Rotation matrix and $t\in\mathbb{R}^3$? This somewhat similar to the well-known "Orthogonal Procrustes problem,” but differs in what is subject to the optimization.
Does anyone know how to attack this problem or can someone send me a possible implementation for solving this problem?
I think this problem is not very tractable from a numeric point of view. The big issue here is the matrix $R$: since you want it to be a rotation matrix, you'll have to add the constraint that $R$ is orthogonal, and an equality constraint on the norm of optimization variables is nonconvex. You can try to tackle this problem with a generic nonlinear solver (such as IPOPT) but in this case you won't have the guarantee to find a solution, and even if you find it, it will be local. An alternative could be global optimization (see, e.g., COUENNE): in this case the solver will relax the nonconvex constraints with mixed integer ones, and it will solve the problem globally, with the guarantee to find a solution if one exists. The second is much slower, but your problem seems to have a reasonable size...