I am making a game with 2D objects that are grid based. These objects are made out of tiles that are actually the cells of the grid.
Each tile has a "mass" that is more than 0 and no upper limit. I want to calculate the tile which is the center of mass of the object.
Consider that the objects are made out of hundreds of blocks and some estimations are acceptable.
I am expecting that, once I find out the center of mass, and apply a force to the object that is tangential with that center of mass, the object will spin around its own axis, without moving in 2D space, just spinning.
Assuming that each tile has uniform surface density (mass per unit of area) then the centre of mass of any one tile is located at its geometrical centre. Then, it's easy to prove that the CM of an object made of multiple tiles is simply the weighted average of the centres of the constituent tiles, where the weights are the masses of the tiles.
In equations, if an object is made of $n$ tiles and the centre of tile $k$ (of mass $m_k$) is at the point $p_k$, then the CM of the object is located at the point $P_{cm}$ such that
$P_{cm} = \displaystyle\frac{\sum_{k=1}^{k=n} m_k\,p_k}{\sum_{k=1}^{k=n} m_k}$
Again, to make it crystal clear, $p_k$ is the geometrical centre of tile $k$, assuming that each tile has uniform density.
I am expecting that, once I find out the center of mass, and apply a force to the object that is tangential with that center of mass, the object will spin around its own axis, without moving in 2D space, just spinning.
First, notice that the CM of the object may not necessarily be part of the object (for example, the CM of a ring is not part of the ring) so you may not necessarily be able to apply forces at that point.
Secondly, if you apply a single force to the object along any line (passing or not by the CM of the object), the object will accelerate, according to Newton's second law, $\vec{F} = m\vec{a}$.
If you want the object to rotate but not accelerate, then you'll need to apply at least two forces of equal magnitude and opposite directions but with a non-zero total torque. In other words, you'll need to apply torques made of forces that cancel each other (the forces cancel each other, not the torques).
As an example, consider the object below, with 8 identical tiles. Because the tiles are identical, the object's CM is as indicated by the red dot.
In the top figure, a single force is applied to the object. The object will simply accelerate. In both of the bottom figures, two forces of the same magnitude but opposite directions are applied to the object. In both cases, the object will not accelerate but will rotate.