Canadian mortgage rates are different from US rates. Canadian rates are compounded semi-anually. So a rate of 6% apparently would be 6.09% in practice.
What is the formula for converting a US rate to Canadian rate? I came across a formula here but it doesn't help since I don't know how to get RM in the formula.
Rewording question - I need to find the formula for monthly mortgage payments. The normal formula finds US monthly payments. What would be the formula to find Canadian monthly payments?
Your question is how to convert a nominal annual rate, compounded semi-annually, to a nominal annual rate, compounded monthly. A nominal annual rate compounded $n$ times a year is usually designated $r^{(n)}.$ The equivalent effective rate $r$ is given by $$ \left(1+\frac{r^{(n)}}{n}\right)^n=1+r $$ The formula gets a bit complicated, so let me do an example. Suppose you are quoted a Canadian rate of $8\%$. The effective annual rate is given by $$\left(1+\frac{.08}{2}\right)^2=1+r\implies r =.0816$$
Now we have to find the equivalent monthly rate: $$ \left(1+\frac{r^{(12)}}{12}\right)^{12}=1.0816\implies r^{(12)}=12\left(\sqrt[12]{1.0816}-1\right)\approx .078698$$
The equivalent U.S. rate is about $7.87\%$
EDIT Just in case I haven't made myself clear, what I'm calling the "effective annual rate" is what the U.S. truth-in-lending law calls the "annual percentage rate" (APR).
EDIT To get the monthly payments, use the formula you linked, with $n=$ the total monthly payments. (For a $30$ year loan, $n$ is $360$). The $r$ to use in that formula should be the nominal monthly rate that I called $r^{(12)}$ divided by $12$. So in the example above, we would have $$r = \frac{.078698}{12}\approx .006558$$