I am not sure what I'm not seeing when using these equations shouldn't the matrix be $6×2$, if using $AX=B$, I know it should be a $6×6$
$ 500- F_{BC}sin45 =0$
$ F_{BC}cos45 - F_{AB} = 0 $
$ -F_{CA} + F_{BC}sin45 = 0 $
$C_y -F_{BC}sin45 = 0 $
$ F_{AC}-A_x = 0 $
$ F_{BA} - A_y = 0 $
I know that $B=\begin{pmatrix}0\\0\\0\\0\\0\\0\end{pmatrix}$and $X = \begin{pmatrix}A_x\\A_y\\C_y\\F_{BA}\\F_{CA}\\ F_{BC} \end{pmatrix}$
Should I just be adding zeros in the matrix so that the dimensions match?
Short Answer-
Yes, you should be adding zeroes whereever the coefficient of a variable is $0$.
Full Answer-
No, while using the $AX=B$ matrix representation of a system of equations, the order of the matrix should be decided by the number of variables.
For example, on your question, you have $6$ variables ( and hence $6$ equations ), so $A$ would be a $6×6$ matrix
In your case $A$ would be-
$A=\begin{pmatrix} 0 & 0 & 0 & 0 & 0 & sin(45^°)\\ 0&0&0&(-1)&0&cos(45^°)\\ 0&0&0&0&(-1)&sin(45^°)\\ 0&0&1& 0&9&sin(45^°)\\ (-1)&0&0&0&1&0\\ (-1)&0&0&1&0&0 \end{pmatrix}$,
(Assuming $F_{CA}=F_{AC}$ and $F_{BA}=F_{AB}$)
Also, $B =\begin{pmatrix} 500\\ 0\\ 0\\ 0\\ 0\\ 0\\ \end{pmatrix}$
(You had incorrectly assumed the constant part of the first equation to be 0)