My attempt:If the function is not monotonous then we can consider points $a < b$ and $b <c$ such that $f(a)<f(b)$ and $f(b)>f(c)$.My intuition behind this was to consider a subset $[a,c]$ of the domain where we can consider two intervals $[a,b]$ and $[b,c]$ such that in the first interval the function is montonically increasing and in the second interval it is monotonically decreasing . We can get such an element $b$ in the domain because the function is not monotone.
Since the function maps every open interval into an open interval then we can consider $(a,b)$ as the first open set which is being mapped to an open set $(f(a),f(b))$ and another open set $(b,c)$ such that $(f(b),f(c))$.[In the first case I am considering the open sets as the open interval]
case $1$: $f(a)<f(b)$ and $f(b)>f(c)$
subcase $i$ : $f(a)<f(c)$ then the interval $(a,c)$ will be mapped to the interval $(f(a),f(b)) \cup (f(b),f(c))$.Also , we see that since $f(c)$ is contained in the interval $(f(a),f(b))$ .Now, I am trying to arrive at a contradiction by concluding that since $b \in (a,c)$ so it is seen that the image of $f$ becomes $(f(a),f(c)) \cup (f(c),f(b)]$ where $f(b)$ is not an interior point of the image of $f((a,c))$ .
subcase $ii$:$f(c)<f(a)$ then the interval $(a,c)$ will be mapped to the interval $(f(c),f(a)) \cup (f(a),f(b)]$.So we can arrive at the same contradiction.
case$2$:We can assume that the function is continuous in the interval $[a,b]$ and $[b,c]$.
Now, we try to consider the case where $(f(a),f(b))$ is the union of countable disjoint intervals.
case $1$: We try to proceed with same approach that $f(a)<f(b)$ and $f(b)>f(c)$ .Then the $(f(a),f(b)) = (f(a),f(a_1)) \cup .........\cup (f(a_k),f(b)) $ .Also, $(f(b),f(c))= (f(b),f(b_1)) \cup ...\cup (f(b_k),f(c))$. $f(a)<f(b)$ and also $f(b)>f(c)$ .Then,
subcase $1$:$f(a)<f(c)$ then the interval $(a,c)$ will be mapped to the all the disjoint intervals between $(f(a),f(c)) \cup (f(c),f(b)]$ .Now, I can conclude by the same logic.
This is the summary of how I want to proceed with the sum but I needed some suggestions on where I am gong wrong and what more cases I am missing out in the proceeed.Hints will really be appreciated instead of complete answers.
Without further hypothesis, this is wrong. It exists real functions that map any open interval onto $\mathbb R$. Those maps are obviously not monotonic.
This is however true for continuous maps. The proof can be performed by contradiction as you did. It can be simplified using the extreme value theorem.
If $f$ is not constant on $(a,b)$, $f$ attains its extreme values on that interval. By changing $f$ into $-f$ if necessary, we can suppose that the maximum of $f$ is attained at $a\lt c \lt b$. But then $f[(a,b)]$ is an interval that is not open: it contains its upper value.