A uniform rectangular lamina $ABCD$, where $AB$ is of length $a$ and $BC$ of length $2a$, has a mass $10m$. Further point masses $m$, $2m$, $3m$ and $4m$ are fixed to the points $A$, $B$, $C$ and $D$ respectively.
Find the centre of mass of the system relative to $x$- and $y$-axes along $AB$ and $AD$ respectively
If the lamina is suspended from the point $A$ find the angle that the diagonal $AC$ makes with the vertical, and to what must the mass at point $D$ be altered if this diagonal is to hang vertically?
So, I found the centre of mass to be $(0.5a, 1.2a)$, by taking moments about A, but I am confused on how to approach the second part of the question. I assumed it would be $tan^{-1} \left(\frac{0.5}{1.2}\right)$, but I am told I failed to account for the point masses? Please help.
To show how i got the centre of mass: the $x$ component is $((0.5a*10m)+(3m*a)+(2m*a))/20m$, the $y$ component is $((10m*a)+(4m*2a)+(3m*2a))/20m$ ($20m$ is used because it is the sum of the rectangle's mass and the point masses at ABCD. )
I am certain the centre of mass calculation is correct, but am very confused on what do from there.
Your calculation of center of mass is correct, and you have perfectly accounted for the point masses. Whoever told you otherwise probably wasn't paying much attention to what you did and just assumed that was the problem.
Your actual mistake was your own assumption that the answer would just be $\tan^{-1}\frac 5{12}$. You paid no attention to exactly what angle they were asking for and just pulled a formula out without giving any thought about what it means.
When you hang an object from some point, it will orient itself so that the center of gravity (when gravity can be assumed as constant, this is the same as the center of mass) is directly below the point, so one side of the angle you are looking for is $\vec{AP}$, where $P$ is the center of mass. The other side is the diagonal ray $\vec{AC}$. So you want $\angle PAC$. What right triangle do you find with this angle whose ratio of opposite to adjacent sides is $5/12$?
I suggest treating $P$ and $C$ as vectors from $A$ and using the inner product formula instead: $$P \cdot C = \|P\|\|C\|\cos \theta$$ where $\theta$ is the angle between the vectors $P$ and $C$.