I want to solve to following two-part problem. I solved the first part but I am not sure how to start on part B.
A) How many ways are there to color the 6 vertices of a regular hexagon using 4 colors ?
My answer is 430, which I am pretty sure about.
B) The 4 colors are red, blue, green, yellow. You are only allowed to use (at max): red and blue twice, green and yellow once - How many ways can you color the hexagon now?
This should result in far less possible solutions, but I am not sure how I can solve this problem.
As mentioned in the comments you are using exactly $2$ red, $2$ blue, $1$ green, and $1$ yellow vertices.
To find the number of colorings including rotation and reflection we use Burnside's Lemma. We need to count how many colorings are fixed under the various symmetries of the Hexagon. Since we only have $1$ green vertex, any symmetry which fixes a coloring must also fix a vertex of the hexagon. This reduces the cases we must check.
For the colorings fixed by the identity: $6$ choices for green, $5$ remaining choices for yellow, and then $\binom{4}{2}$ choices for red, and blue is determined giving a total of $180.$
The only other symmetries which fix a vertex are the $3$ reflections over the a line which goes through two opposite vertices. For a coloring to be fixed under this reflection the two fixed vertices must be green and yellow and the two pairs of reflected vertices must be red and blue. This gives $2\times2=4$ colorings fixed under each of these $3$ reflections.
Now we add up and divide by the size of the symmetry group.
$\frac{180+3\times4}{12}=16$ colorings.