Is it an unwritten rule at the heart of mathematics to always look for an "elegant" solution and always look at brute force computation as sort of tarnishing the mathematical beauty?
Take for example there is some group theory question about a group of order $k$. And someone solves it theoretically while someone explicitly finds all groups of that order and then does case-by-case computation. Is the second method always frowned upon?
And in cases where there is already a brute force solution but no "elegant" solution, are mathematicians always secretly hoping for an elegant solution in spite of there already being a solution?
Yes, elegant solutions are preferred. The most compelling reason for this preference is that an elegant answer to a question provides insight into why it would be the answer, whereas a brute force solution gives us only the answer without the insight. Mathematicians typically want understanding over results. It's about the journey, not the destination ;)
But also, the understanding usually gives you the results that a brute force solution would find anyways, but more compactly. For example, say you want to find (up to isomorphism) all abelian groups of order $27$. A brute force approach would eventually find that there are three of them, and you could write down their operation tables and point to them. But if you know the elegant, more general solution, the classification theorem of finite abelian groups, you can (1) immediately say that there are three of them, (2) write them down more compactly as $\mathbf{Z}_{27}$, $\mathbf{Z}_{3}\times \mathbf{Z}_{9}$, and $\mathbf{Z}_{3} \times \mathbf{Z}_{3} \times \mathbf{Z}_{3}$, and (3) by writing them down more compactly you can see their structure better than you could from the operation table.
To address your other questions directly: