Let $(X,d)$ and $(Y,d')$ be metric spaces.
Let $f$ and $g$ be two functions with the following mapping: $X\rightarrow Y$
Let $f$, $g$ be continuous on $X$. Let $A\subseteq X$ with $A$ dense in $X$ such that $f(a)=g(a)\space\forall\space a\in A$
I want to show $f(x)=g(x)\space\forall\space x\in X$
Let $y\in X$
Since $f$ and $g$ are continuous on $X$, then $f-g$ is continuous on $X$
Suppose $f(y)>g(y)$ which is equivalent to saying $f(y)-g(y)>0$
By the definition of continuity, $\exists\space\delta_{0}>0$ such that:
If $|x-y|<\delta_{0}$, then $|(f(x)-g(x))-(f(y)-g(y))|<f(y)-g(y)$
This implies:
If $y-\delta_{0}<x<y+\delta_{0}$, then $f(x)>g(x)$
But $A$ is dense in $X$
So $\exists\space a\in (y-\delta_{0},y+\delta_{0})\cap A$
So $\exists x\in (y-\delta_{0},y+\delta_{0})$ such that $f(x)=g(x)$
We have contradiction. I can then go on to find a similar contradiction for the case of $f(y)<g(y)$ to conclude that $f(y)$ must equal $g(y)$.
Is my approach correct?
Your idea is not bad , if your functions where $f,g : (\mathbb{R},|.|) \to (\mathbb{R}, |.|)$ it would be correct ! But now since the space $(Y, d')$ is arbitrary you cant be sure whether you have algebraic operations in $Y$ to add and substract elemements inside $Y$.
But still you can modify your argument to prove the argument .
Suppose that there is a $y \in X$ such that $f(y) \neq g(y)$ (now you cant say $f(y) > g(y)$ or $ f(y) < g(y)$ since you dont know if there is any notion of comparison of two elements in $Y$ like we have in the well ordered set $\mathbb{R}$.)
But from $f(y) \neq g(y)$ you know that $\epsilon = d'(f(y),g(y)) >0$. (This is the same like you said $f(y)-g(y) >0$ but we cant substract $g(y) $ from $f(y)$ in the random space $Y$ ).
Now you can procceed in the same way and say :
Since $f$ is continuous at $y$ there is an $r_1>0$ such that for every $x \in X$ with $d(x,y)< r_1$ (thats the same with $|x-y| < r_1$) you have $d'(f(y),f(x)) < \frac{\epsilon}{2}.$
Again by the continuity of $g$ there is an $r_2 >0$ such that for every $x \in X$ with $d(x,y) < r_2$ you have $d'(g(x),g(y)) < \frac{\epsilon}{2}.$
Pick $r = min \{r_1,r_2\} >0$
Now since $A$ is dense there exist $\alpha \in A$ such that $ \alpha \in B(y,r)$ where $B(y,r) = \{ x \in X : d(x,y) <r\}$ , which means that $d(\alpha, y ) < r.$
Now since $d \in A$ by hypothesis $f(\alpha) = g(\alpha)$
So $\epsilon = d'(f(y),g(y)) \leq d'(f(y),f(\alpha)) + d'(g(y),f(\alpha)) =d'(f(y),f(\alpha)) + d'(g(y),g(\alpha)).$ (*)
Now remember that $d(y,\alpha) < r = min\{r_1,r_2\}$ so $d'(f(y),f(\alpha)) < \dfrac{\epsilon}{2}$ and $d'(g(y),g(\alpha))< \dfrac{\epsilon}{2}.$
So if you plug this in (*) you have $\epsilon < \epsilon$ which is a contradiction!
Hence $f(y)=g(y).$