Let $N$ be the subgroup of $A = \left\lbrace f : x \mapsto ax + b : a \in \Bbb R^*, b \in \Bbb R\right\rbrace$ defined as
$$N = \left\lbrace g : x \mapsto x +b : b \in \Bbb R \right\rbrace$$
What do the cosets of $N$ in $A$ look like? Are they simply $fN$ where $f$ is a transformation as given in $A$?
Then, for instance, if $f(x) = \alpha x + \beta$ the coset $fN$ looks like
$$fN = \left\lbrace f \circ g : x \mapsto \alpha x + \beta + b: \alpha \in \Bbb R^*, b, \beta \in \Bbb R\right\rbrace$$
and the quotient $A/N$ has operation $(f_1N)(f_2N) = (f_1 \circ f_2)N$.
Is this correct?
This idea is correct, but the expression for the elements of $fN$ isn't correct: Once we've fixed $f : x \mapsto \alpha x + \beta$, $\alpha$ and $\beta$ are fixed. So, for $f : x \mapsto \alpha x + \beta$, we have $$ \begin{align} fN &:= \{f \circ n : n \in N \} \\ &\phantom{:}= \{x \mapsto \alpha(x + b) + \beta : b \in \Bbb R\} \end{align} $$ Now, we can rewrite the generic function in $fN$ as $x \mapsto \alpha x + (\alpha b + \beta)$, and since $\alpha \neq 0$, for any $B \in \Bbb R$ we can choose $b$ so that the constant term $\alpha b + \beta$ is $B$. Thus, we may as well write the coset as $$\color{#bf0000}{\boxed{fN = \{x \mapsto \alpha x + B : B \in \Bbb R\}}} .$$ In other words, two functions are in the same (left) coset of the affine group $A$ modulo $N$ iff they have the same leading coefficient, and thus we can specify a coset $fN$ simply by giving that coefficient, namely, $\alpha$.
One can check that $N$ is normal is $A$, so $A / N$ inherits a group structure. You give the correct general formula, but it's instructive to write it the product of $f_1 N$ and $f_2 N$ explicitly in terms of the representative affine functions $f_1, f_2$. We know from the previous paragraph that we should be able to write this product in terms of the respective leading coefficients $\alpha_1, \alpha_2$.